Diff of the two buildlogs: -- --- b1/build.log 2021-08-25 09:28:44.180678703 +0000 +++ b2/build.log 2021-08-25 09:31:35.506253497 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Tue Aug 24 21:26:03 -12 2021 -I: pbuilder-time-stamp: 1629883563 +I: Current time: Wed Aug 25 23:28:56 +14 2021 +I: pbuilder-time-stamp: 1629883736 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration @@ -16,8 +16,8 @@ I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' -gpgv: keyblock resource '/tmp/dpkg-verify-sig.gIqeacWY/trustedkeys.kbx': General error -gpgv: Signature made Sat Jan 18 19:24:28 2020 -12 +gpgv: keyblock resource '/tmp/dpkg-verify-sig.JjUvhcv8/trustedkeys.kbx': General error +gpgv: Signature made Sun Jan 19 21:24:28 2020 +14 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: failed to verify signature on ./golang-github-hashicorp-raft_1.1.2-1.dsc @@ -32,135 +32,169 @@ dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/23540/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/7057/tmp/hooks/D01_modify_environment starting +debug: Running on virt32b. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +Removing 'diversion of /bin/sh to /bin/sh.distrib by dash' +Adding 'diversion of /bin/sh to /bin/sh.distrib by bash' +Removing 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash' +Adding 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by bash' +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/7057/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/7057/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='armhf' - DEBIAN_FRONTEND='noninteractive' - DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=3' - DISTRIBUTION='' - HOME='/root' - HOST_ARCH='armhf' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="1" [2]="4" [3]="1" [4]="release" [5]="arm-unknown-linux-gnueabihf") + BASH_VERSION='5.1.4(1)-release' + BUILDDIR=/build + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=armhf + DEBIAN_FRONTEND=noninteractive + DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=4' + DIRSTACK=() + DISTRIBUTION= + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/root + HOSTNAME=i-capture-the-hostname + HOSTTYPE=arm + HOST_ARCH=armhf IFS=' ' - INVOCATION_ID='6c7596e6fc9146c585771f224c323c99' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='23540' - PS1='# ' - PS2='> ' + INVOCATION_ID=3c69134a93044a789ab16a0186c8d167 + LANG=C + LANGUAGE=it_CH:it + LC_ALL=C + MACHTYPE=arm-unknown-linux-gnueabihf + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnueabihf + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=7057 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/pbuilderrc_UzKj --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' - SUDO_GID='113' - SUDO_UID='107' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - _='/usr/bin/systemd-run' - http_proxy='http://10.0.0.15:8000/' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/pbuilderrc_HV2Y --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.wjmIrRuTSj/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' + SUDO_GID=112 + SUDO_UID=106 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + _='I: set' + http_proxy=http://10.0.0.15:8000/ I: uname -a - Linux virt64b 5.10.0-8-arm64 #1 SMP Debian 5.10.46-4 (2021-08-03) aarch64 GNU/Linux + Linux i-capture-the-hostname 5.10.0-8-armmp-lpae #1 SMP Debian 5.10.46-4 (2021-08-03) armv7l GNU/Linux I: ls -l /bin total 3580 - -rwxr-xr-x 1 root root 816764 Aug 4 08:25 bash - -rwxr-xr-x 3 root root 26052 Jul 20 2020 bunzip2 - -rwxr-xr-x 3 root root 26052 Jul 20 2020 bzcat - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzcmp -> bzdiff - -rwxr-xr-x 1 root root 2225 Jul 20 2020 bzdiff - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzegrep -> bzgrep - -rwxr-xr-x 1 root root 4877 Sep 4 2019 bzexe - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzfgrep -> bzgrep - -rwxr-xr-x 1 root root 3775 Jul 20 2020 bzgrep - -rwxr-xr-x 3 root root 26052 Jul 20 2020 bzip2 - -rwxr-xr-x 1 root root 9636 Jul 20 2020 bzip2recover - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzless -> bzmore - -rwxr-xr-x 1 root root 1297 Jul 20 2020 bzmore - -rwxr-xr-x 1 root root 26668 Sep 22 2020 cat - -rwxr-xr-x 1 root root 43104 Sep 22 2020 chgrp - -rwxr-xr-x 1 root root 38984 Sep 22 2020 chmod - -rwxr-xr-x 1 root root 43112 Sep 22 2020 chown - -rwxr-xr-x 1 root root 92616 Sep 22 2020 cp - -rwxr-xr-x 1 root root 75524 Dec 10 2020 dash - -rwxr-xr-x 1 root root 75880 Sep 22 2020 date - -rwxr-xr-x 1 root root 55436 Sep 22 2020 dd - -rwxr-xr-x 1 root root 59912 Sep 22 2020 df - -rwxr-xr-x 1 root root 96764 Sep 22 2020 dir - -rwxr-xr-x 1 root root 55012 Jul 28 07:09 dmesg - lrwxrwxrwx 1 root root 8 Nov 6 2019 dnsdomainname -> hostname - lrwxrwxrwx 1 root root 8 Nov 6 2019 domainname -> hostname - -rwxr-xr-x 1 root root 22508 Sep 22 2020 echo - -rwxr-xr-x 1 root root 28 Nov 9 2020 egrep - -rwxr-xr-x 1 root root 22496 Sep 22 2020 false - -rwxr-xr-x 1 root root 28 Nov 9 2020 fgrep - -rwxr-xr-x 1 root root 47492 Jul 28 07:09 findmnt - -rwsr-xr-x 1 root root 26076 Feb 26 04:12 fusermount - -rwxr-xr-x 1 root root 124508 Nov 9 2020 grep - -rwxr-xr-x 2 root root 2346 Mar 2 11:30 gunzip - -rwxr-xr-x 1 root root 6376 Mar 2 11:30 gzexe - -rwxr-xr-x 1 root root 64212 Mar 2 11:30 gzip - -rwxr-xr-x 1 root root 13784 Nov 6 2019 hostname - -rwxr-xr-x 1 root root 43180 Sep 22 2020 ln - -rwxr-xr-x 1 root root 35068 Feb 7 2020 login - -rwxr-xr-x 1 root root 96764 Sep 22 2020 ls - -rwxr-xr-x 1 root root 99940 Jul 28 07:09 lsblk - -rwxr-xr-x 1 root root 51408 Sep 22 2020 mkdir - -rwxr-xr-x 1 root root 43184 Sep 22 2020 mknod - -rwxr-xr-x 1 root root 30780 Sep 22 2020 mktemp - -rwxr-xr-x 1 root root 34408 Jul 28 07:09 more - -rwsr-xr-x 1 root root 34400 Jul 28 07:09 mount - -rwxr-xr-x 1 root root 9824 Jul 28 07:09 mountpoint - -rwxr-xr-x 1 root root 88524 Sep 22 2020 mv - lrwxrwxrwx 1 root root 8 Nov 6 2019 nisdomainname -> hostname - lrwxrwxrwx 1 root root 14 Apr 18 03:38 pidof -> /sbin/killall5 - -rwxr-xr-x 1 root root 26652 Sep 22 2020 pwd - lrwxrwxrwx 1 root root 4 Aug 4 08:25 rbash -> bash - -rwxr-xr-x 1 root root 30740 Sep 22 2020 readlink - -rwxr-xr-x 1 root root 43104 Sep 22 2020 rm - -rwxr-xr-x 1 root root 30732 Sep 22 2020 rmdir - -rwxr-xr-x 1 root root 14144 Sep 27 2020 run-parts - -rwxr-xr-x 1 root root 76012 Dec 22 2018 sed - lrwxrwxrwx 1 root root 4 Aug 20 21:24 sh -> dash - -rwxr-xr-x 1 root root 22532 Sep 22 2020 sleep - -rwxr-xr-x 1 root root 55360 Sep 22 2020 stty - -rwsr-xr-x 1 root root 46704 Jul 28 07:09 su - -rwxr-xr-x 1 root root 22532 Sep 22 2020 sync - -rwxr-xr-x 1 root root 340872 Feb 16 2021 tar - -rwxr-xr-x 1 root root 9808 Sep 27 2020 tempfile - -rwxr-xr-x 1 root root 67696 Sep 22 2020 touch - -rwxr-xr-x 1 root root 22496 Sep 22 2020 true - -rwxr-xr-x 1 root root 9636 Feb 26 04:12 ulockmgr_server - -rwsr-xr-x 1 root root 22108 Jul 28 07:09 umount - -rwxr-xr-x 1 root root 22520 Sep 22 2020 uname - -rwxr-xr-x 2 root root 2346 Mar 2 11:30 uncompress - -rwxr-xr-x 1 root root 96764 Sep 22 2020 vdir - -rwxr-xr-x 1 root root 38512 Jul 28 07:09 wdctl - lrwxrwxrwx 1 root root 8 Nov 6 2019 ypdomainname -> hostname - -rwxr-xr-x 1 root root 1984 Mar 2 11:30 zcat - -rwxr-xr-x 1 root root 1678 Mar 2 11:30 zcmp - -rwxr-xr-x 1 root root 5880 Mar 2 11:30 zdiff - -rwxr-xr-x 1 root root 29 Mar 2 11:30 zegrep - -rwxr-xr-x 1 root root 29 Mar 2 11:30 zfgrep - -rwxr-xr-x 1 root root 2081 Mar 2 11:30 zforce - -rwxr-xr-x 1 root root 7585 Mar 2 11:30 zgrep - -rwxr-xr-x 1 root root 2206 Mar 2 11:30 zless - -rwxr-xr-x 1 root root 1842 Mar 2 11:30 zmore - -rwxr-xr-x 1 root root 4553 Mar 2 11:30 znew -I: user script /srv/workspace/pbuilder/23540/tmp/hooks/D02_print_environment finished + -rwxr-xr-x 1 root root 816764 Aug 5 10:25 bash + -rwxr-xr-x 3 root root 26052 Jul 21 2020 bunzip2 + -rwxr-xr-x 3 root root 26052 Jul 21 2020 bzcat + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzcmp -> bzdiff + -rwxr-xr-x 1 root root 2225 Jul 21 2020 bzdiff + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzegrep -> bzgrep + -rwxr-xr-x 1 root root 4877 Sep 5 2019 bzexe + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzfgrep -> bzgrep + -rwxr-xr-x 1 root root 3775 Jul 21 2020 bzgrep + -rwxr-xr-x 3 root root 26052 Jul 21 2020 bzip2 + -rwxr-xr-x 1 root root 9636 Jul 21 2020 bzip2recover + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzless -> bzmore + -rwxr-xr-x 1 root root 1297 Jul 21 2020 bzmore + -rwxr-xr-x 1 root root 26668 Sep 23 2020 cat + -rwxr-xr-x 1 root root 43104 Sep 23 2020 chgrp + -rwxr-xr-x 1 root root 38984 Sep 23 2020 chmod + -rwxr-xr-x 1 root root 43112 Sep 23 2020 chown + -rwxr-xr-x 1 root root 92616 Sep 23 2020 cp + -rwxr-xr-x 1 root root 75524 Dec 11 2020 dash + -rwxr-xr-x 1 root root 75880 Sep 23 2020 date + -rwxr-xr-x 1 root root 55436 Sep 23 2020 dd + -rwxr-xr-x 1 root root 59912 Sep 23 2020 df + -rwxr-xr-x 1 root root 96764 Sep 23 2020 dir + -rwxr-xr-x 1 root root 55012 Jul 29 09:09 dmesg + lrwxrwxrwx 1 root root 8 Nov 8 2019 dnsdomainname -> hostname + lrwxrwxrwx 1 root root 8 Nov 8 2019 domainname -> hostname + -rwxr-xr-x 1 root root 22508 Sep 23 2020 echo + -rwxr-xr-x 1 root root 28 Nov 10 2020 egrep + -rwxr-xr-x 1 root root 22496 Sep 23 2020 false + -rwxr-xr-x 1 root root 28 Nov 10 2020 fgrep + -rwxr-xr-x 1 root root 47492 Jul 29 09:09 findmnt + -rwsr-xr-x 1 root root 26076 Feb 27 06:12 fusermount + -rwxr-xr-x 1 root root 124508 Nov 10 2020 grep + -rwxr-xr-x 2 root root 2346 Mar 3 13:30 gunzip + -rwxr-xr-x 1 root root 6376 Mar 3 13:30 gzexe + -rwxr-xr-x 1 root root 64212 Mar 3 13:30 gzip + -rwxr-xr-x 1 root root 13784 Nov 8 2019 hostname + -rwxr-xr-x 1 root root 43180 Sep 23 2020 ln + -rwxr-xr-x 1 root root 35068 Feb 8 2020 login + -rwxr-xr-x 1 root root 96764 Sep 23 2020 ls + -rwxr-xr-x 1 root root 99940 Jul 29 09:09 lsblk + -rwxr-xr-x 1 root root 51408 Sep 23 2020 mkdir + -rwxr-xr-x 1 root root 43184 Sep 23 2020 mknod + -rwxr-xr-x 1 root root 30780 Sep 23 2020 mktemp + -rwxr-xr-x 1 root root 34408 Jul 29 09:09 more + -rwsr-xr-x 1 root root 34400 Jul 29 09:09 mount + -rwxr-xr-x 1 root root 9824 Jul 29 09:09 mountpoint + -rwxr-xr-x 1 root root 88524 Sep 23 2020 mv + lrwxrwxrwx 1 root root 8 Nov 8 2019 nisdomainname -> hostname + lrwxrwxrwx 1 root root 14 Apr 19 05:38 pidof -> /sbin/killall5 + -rwxr-xr-x 1 root root 26652 Sep 23 2020 pwd + lrwxrwxrwx 1 root root 4 Aug 5 10:25 rbash -> bash + -rwxr-xr-x 1 root root 30740 Sep 23 2020 readlink + -rwxr-xr-x 1 root root 43104 Sep 23 2020 rm + -rwxr-xr-x 1 root root 30732 Sep 23 2020 rmdir + -rwxr-xr-x 1 root root 14144 Sep 28 2020 run-parts + -rwxr-xr-x 1 root root 76012 Dec 23 2018 sed + lrwxrwxrwx 1 root root 4 Aug 25 23:29 sh -> bash + lrwxrwxrwx 1 root root 4 Aug 21 23:25 sh.distrib -> dash + -rwxr-xr-x 1 root root 22532 Sep 23 2020 sleep + -rwxr-xr-x 1 root root 55360 Sep 23 2020 stty + -rwsr-xr-x 1 root root 46704 Jul 29 09:09 su + -rwxr-xr-x 1 root root 22532 Sep 23 2020 sync + -rwxr-xr-x 1 root root 340872 Feb 17 2021 tar + -rwxr-xr-x 1 root root 9808 Sep 28 2020 tempfile + -rwxr-xr-x 1 root root 67696 Sep 23 2020 touch + -rwxr-xr-x 1 root root 22496 Sep 23 2020 true + -rwxr-xr-x 1 root root 9636 Feb 27 06:12 ulockmgr_server + -rwsr-xr-x 1 root root 22108 Jul 29 09:09 umount + -rwxr-xr-x 1 root root 22520 Sep 23 2020 uname + -rwxr-xr-x 2 root root 2346 Mar 3 13:30 uncompress + -rwxr-xr-x 1 root root 96764 Sep 23 2020 vdir + -rwxr-xr-x 1 root root 38512 Jul 29 09:09 wdctl + lrwxrwxrwx 1 root root 8 Nov 8 2019 ypdomainname -> hostname + -rwxr-xr-x 1 root root 1984 Mar 3 13:30 zcat + -rwxr-xr-x 1 root root 1678 Mar 3 13:30 zcmp + -rwxr-xr-x 1 root root 5880 Mar 3 13:30 zdiff + -rwxr-xr-x 1 root root 29 Mar 3 13:30 zegrep + -rwxr-xr-x 1 root root 29 Mar 3 13:30 zfgrep + -rwxr-xr-x 1 root root 2081 Mar 3 13:30 zforce + -rwxr-xr-x 1 root root 7585 Mar 3 13:30 zgrep + -rwxr-xr-x 1 root root 2206 Mar 3 13:30 zless + -rwxr-xr-x 1 root root 1842 Mar 3 13:30 zmore + -rwxr-xr-x 1 root root 4553 Mar 3 13:30 znew +I: user script /srv/workspace/pbuilder/7057/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -319,7 +353,7 @@ Get: 111 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] -Fetched 91.1 MB in 11s (7992 kB/s) +Fetched 91.1 MB in 9s (9830 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19398 files and directories currently installed.) @@ -792,7 +826,8 @@ fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package -I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes +hostname: Name or service not known +I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable @@ -810,141 +845,141 @@ dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang - cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench -internal/cpu + cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader +internal/cpu runtime/internal/sys internal/race -runtime/internal/math sync/atomic +unicode +runtime/internal/math internal/bytealg runtime/internal/atomic -unicode unicode/utf8 container/list -runtime crypto/internal/subtle crypto/subtle math/bits +runtime internal/testlog -math encoding unicode/utf16 +math internal/nettrace -runtime/cgo vendor/golang.org/x/crypto/cryptobyte/asn1 +runtime/cgo vendor/golang.org/x/crypto/internal/subtle golang.org/x/sys/internal/unsafeheader internal/reflectlite sync internal/singleflight math/rand -errors sort +errors io internal/oserror -syscall strconv +syscall +github.com/hashicorp/golang-lru/simplelru +vendor/golang.org/x/net/dns/dnsmessage bytes strings reflect bufio +github.com/hashicorp/go-immutable-radix time internal/syscall/unix internal/syscall/execenv -github.com/hashicorp/golang-lru/simplelru -github.com/hashicorp/go-immutable-radix -vendor/golang.org/x/net/dns/dnsmessage +hash +crypto/internal/randutil +crypto/rc4 +hash/crc32 +crypto +crypto/hmac +vendor/golang.org/x/text/transform +vendor/golang.org/x/crypto/hkdf +path +regexp/syntax +html context internal/poll -encoding/binary +hash/crc64 internal/fmtsort +encoding/binary os -hash -hash/crc32 +regexp crypto/cipher encoding/base64 -crypto -crypto/aes -crypto/des -crypto/internal/randutil crypto/sha512 -crypto/ed25519/internal/edwards25519 fmt +crypto/aes net -crypto/hmac +crypto/des +crypto/ed25519/internal/edwards25519 crypto/md5 -crypto/rc4 +crypto/sha1 math/big encoding/json log compress/flate -crypto/rand -crypto/elliptic compress/gzip -encoding/asn1 -crypto/ed25519 -crypto/rsa -vendor/golang.org/x/crypto/cryptobyte -crypto/sha1 crypto/sha256 -crypto/ecdsa -crypto/dsa encoding/hex encoding/pem -crypto/x509/pkix path/filepath net/url io/ioutil +crypto/rand +crypto/elliptic +encoding/asn1 +crypto/ed25519 +crypto/rsa +crypto/dsa +vendor/golang.org/x/crypto/cryptobyte +crypto/x509/pkix vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/poly1305 vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/crypto/chacha20poly1305 -vendor/golang.org/x/crypto/hkdf -vendor/golang.org/x/text/transform +crypto/ecdsa vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm -crypto/x509 -vendor/golang.org/x/text/secure/bidirule -net/textproto vendor/golang.org/x/net/http2/hpack mime -vendor/golang.org/x/net/idna -crypto/tls +vendor/golang.org/x/text/secure/bidirule mime/quotedprintable -mime/multipart -vendor/golang.org/x/net/http/httpguts -vendor/golang.org/x/net/http/httpproxy net/http/internal -path os/signal golang.org/x/sys/unix -regexp/syntax -regexp -net/http/httptrace +vendor/golang.org/x/net/idna encoding/gob +go/token +text/template/parse +crypto/x509 +net/textproto +vendor/golang.org/x/net/http/httpguts +vendor/golang.org/x/net/http/httpproxy +mime/multipart github.com/mattn/go-isatty +text/template github.com/mattn/go-colorable -net/http +flag github.com/fatih/color github.com/hashicorp/go-hclog -go/token -html -text/template/parse -hash/crc64 -flag +crypto/tls runtime/debug runtime/trace -text/template testing html/template +net/http/httptrace +net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang - cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench + cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max @@ -986,32 +1021,32 @@ === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir - testing.go:153: 2021-08-25T09:28:00.244Z [INFO] creating new snapshot: path=/tmp/raft487955817/raft203318708/snapshots/3-10-1629883680244.tmp + testing.go:153: 2021-08-25T09:30:55.743Z [INFO] creating new snapshot: path=/tmp/raft214392515/raft220312390/snapshots/3-10-1629883855743.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot - testing.go:153: 2021-08-25T09:28:00.252Z [INFO] creating new snapshot: path=/tmp/raft591350147/snapshots/3-10-1629883680252.tmp - testing.go:153: 2021-08-25T09:28:00.258Z [WARN] found temporary snapshot: name=3-10-1629883680252.tmp + testing.go:153: 2021-08-25T09:30:55.754Z [INFO] creating new snapshot: path=/tmp/raft136735213/snapshots/3-10-1629883855754.tmp + testing.go:153: 2021-08-25T09:30:55.760Z [WARN] found temporary snapshot: name=3-10-1629883855754.tmp --- PASS: TestFileSS_CreateSnapshot (0.02s) === RUN TestFileSS_CancelSnapshot - testing.go:153: 2021-08-25T09:28:00.274Z [INFO] creating new snapshot: path=/tmp/raft181443334/snapshots/3-10-1629883680274.tmp + testing.go:153: 2021-08-25T09:30:55.776Z [INFO] creating new snapshot: path=/tmp/raft121874024/snapshots/3-10-1629883855776.tmp --- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention - testing.go:153: 2021-08-25T09:28:00.284Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-10-1629883680284.tmp - testing.go:153: 2021-08-25T09:28:00.302Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-11-1629883680302.tmp - testing.go:153: 2021-08-25T09:28:00.320Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-12-1629883680320.tmp - testing.go:153: 2021-08-25T09:28:00.337Z [INFO] reaping snapshot: path=/tmp/raft008643757/snapshots/3-10-1629883680284 - testing.go:153: 2021-08-25T09:28:00.338Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-13-1629883680338.tmp - testing.go:153: 2021-08-25T09:28:00.354Z [INFO] reaping snapshot: path=/tmp/raft008643757/snapshots/3-11-1629883680302 - testing.go:153: 2021-08-25T09:28:00.355Z [INFO] creating new snapshot: path=/tmp/raft008643757/snapshots/3-14-1629883680355.tmp - testing.go:153: 2021-08-25T09:28:00.372Z [INFO] reaping snapshot: path=/tmp/raft008643757/snapshots/3-12-1629883680320 ---- PASS: TestFileSS_Retention (0.09s) + testing.go:153: 2021-08-25T09:30:55.791Z [INFO] creating new snapshot: path=/tmp/raft391945383/snapshots/3-10-1629883855791.tmp + testing.go:153: 2021-08-25T09:30:55.817Z [INFO] creating new snapshot: path=/tmp/raft391945383/snapshots/3-11-1629883855817.tmp + testing.go:153: 2021-08-25T09:30:55.840Z [INFO] creating new snapshot: path=/tmp/raft391945383/snapshots/3-12-1629883855839.tmp + testing.go:153: 2021-08-25T09:30:55.859Z [INFO] reaping snapshot: path=/tmp/raft391945383/snapshots/3-10-1629883855791 + testing.go:153: 2021-08-25T09:30:55.860Z [INFO] creating new snapshot: path=/tmp/raft391945383/snapshots/3-13-1629883855860.tmp + testing.go:153: 2021-08-25T09:30:55.883Z [INFO] reaping snapshot: path=/tmp/raft391945383/snapshots/3-11-1629883855817 + testing.go:153: 2021-08-25T09:30:55.884Z [INFO] creating new snapshot: path=/tmp/raft391945383/snapshots/3-14-1629883855884.tmp + testing.go:153: 2021-08-25T09:30:55.910Z [INFO] reaping snapshot: path=/tmp/raft391945383/snapshots/3-12-1629883855839 +--- PASS: TestFileSS_Retention (0.12s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering - testing.go:153: 2021-08-25T09:28:00.377Z [INFO] creating new snapshot: path=/tmp/raft290314716/snapshots/5-130350-1629883680377.tmp - testing.go:153: 2021-08-25T09:28:00.395Z [INFO] creating new snapshot: path=/tmp/raft290314716/snapshots/36-204917-1629883680394.tmp + testing.go:153: 2021-08-25T09:30:55.914Z [INFO] creating new snapshot: path=/tmp/raft177964462/snapshots/5-130350-1629883855914.tmp + testing.go:153: 2021-08-25T09:30:55.936Z [INFO] creating new snapshot: path=/tmp/raft177964462/snapshots/36-204917-1629883855936.tmp --- PASS: TestFileSS_Ordering (0.04s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) @@ -1057,29 +1092,29 @@ === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn ---- PASS: TestNetworkTransport_PooledConn (0.00s) +--- PASS: TestNetworkTransport_PooledConn (0.02s) === RUN TestNetworkTransport_ListenBackoff - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.440Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.002Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.445Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.008Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.455Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.019Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.475Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.039Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.516Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.079Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.596Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.159Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:00.757Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.320Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:01.077Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:56.640Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:01.717Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:57.281Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:02.718Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:58.281Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-08-25T09:28:03.718Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2021-08-25T09:30:59.282Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration @@ -1089,2803 +1124,2901 @@ === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop -testLoggerAdapter verbose: server-57d02664-b621-7445-0603-5d45444cde22: 2021-08-25T09:28:04.444Z [INFO] server-57d02664-b621-7445-0603-5d45444cde22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57d02664-b621-7445-0603-5d45444cde22 Address:57d02664-b621-7445-0603-5d45444cde22}]" - testing.go:149: server-57d02664-b621-7445-0603-5d45444cde22: 2021-08-25T09:28:04.444Z [INFO] server-57d02664-b621-7445-0603-5d45444cde22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57d02664-b621-7445-0603-5d45444cde22 Address:57d02664-b621-7445-0603-5d45444cde22}]" +testLoggerAdapter verbose: server-5a945f62-a102-7777-ba33-5203822559bd: 2021-08-25T09:31:00.007Z [INFO] server-5a945f62-a102-7777-ba33-5203822559bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a945f62-a102-7777-ba33-5203822559bd Address:5a945f62-a102-7777-ba33-5203822559bd}]" + testing.go:149: server-5a945f62-a102-7777-ba33-5203822559bd: 2021-08-25T09:31:00.007Z [INFO] server-5a945f62-a102-7777-ba33-5203822559bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a945f62-a102-7777-ba33-5203822559bd Address:5a945f62-a102-7777-ba33-5203822559bd}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown -testLoggerAdapter verbose: server-1b882f6c-13c3-ac45-0313-fccf176e94fa: 2021-08-25T09:28:04.453Z [INFO] server-1b882f6c-13c3-ac45-0313-fccf176e94fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b882f6c-13c3-ac45-0313-fccf176e94fa Address:1b882f6c-13c3-ac45-0313-fccf176e94fa}]" - testing.go:149: server-1b882f6c-13c3-ac45-0313-fccf176e94fa: 2021-08-25T09:28:04.453Z [INFO] server-1b882f6c-13c3-ac45-0313-fccf176e94fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b882f6c-13c3-ac45-0313-fccf176e94fa Address:1b882f6c-13c3-ac45-0313-fccf176e94fa}]" +testLoggerAdapter verbose: server-93db230f-e40e-c8f2-bce1-49bce958dad3: 2021-08-25T09:31:00.009Z [INFO] server-93db230f-e40e-c8f2-bce1-49bce958dad3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93db230f-e40e-c8f2-bce1-49bce958dad3 Address:93db230f-e40e-c8f2-bce1-49bce958dad3}]" + testing.go:149: server-93db230f-e40e-c8f2-bce1-49bce958dad3: 2021-08-25T09:31:00.009Z [INFO] server-93db230f-e40e-c8f2-bce1-49bce958dad3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93db230f-e40e-c8f2-bce1-49bce958dad3 Address:93db230f-e40e-c8f2-bce1-49bce958dad3}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.455Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: initial configuration: index=0 servers=[] - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.455Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: initial configuration: index=0 servers=[] - testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: initial configuration: index=0 servers=[] - testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.456Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering follower state: follower="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Follower]" leader= - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.456Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering follower state: follower="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Follower]" leader= -testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: entering follower state: follower="Node at de11e8f1-6ded-c95f-940d-f2fe6b54457a [Follower]" leader= - testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.456Z [INFO] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: entering follower state: follower="Node at de11e8f1-6ded-c95f-940d-f2fe6b54457a [Follower]" leader= -testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: entering follower state: follower="Node at 25ad4171-2b97-13ec-8ff6-73422e0b167e [Follower]" leader= - testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.456Z [INFO] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: entering follower state: follower="Node at 25ad4171-2b97-13ec-8ff6-73422e0b167e [Follower]" leader= -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.457Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:04.457Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.536Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: no known peers, aborting election - testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.536Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: no known peers, aborting election -testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.536Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: no known peers, aborting election - testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.536Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: no known peers, aborting election -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering candidate state: node="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Candidate]" term=2 - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.539Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering candidate state: node="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Candidate]" term=2 -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: election won: tally=2 - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: election won: tally=2 -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering leader state: leader="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Leader]" - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: entering leader state: leader="Node at 63e6787b-e3aa-ada4-fa06-4d40330f7d20 [Leader]" -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-de11e8f1-6ded-c95f-940d-f2fe6b54457a - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-de11e8f1-6ded-c95f-940d-f2fe6b54457a -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-25ad4171-2b97-13ec-8ff6-73422e0b167e - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: added peer, starting replication: peer=server-25ad4171-2b97-13ec-8ff6-73422e0b167e -testLoggerAdapter verbose: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.540Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.540Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: 2021-08-25T09:28:04.540Z [WARN] server-de11e8f1-6ded-c95f-940d-f2fe6b54457a: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" next=1 - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" next=1 -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.540Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" - testing.go:149: server-25ad4171-2b97-13ec-8ff6-73422e0b167e: 2021-08-25T09:28:04.540Z [WARN] server-25ad4171-2b97-13ec-8ff6-73422e0b167e: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" next=1 - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [WARN] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: appendEntries rejected, sending older logs: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" next=1 -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.541Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: pipelining replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.645Z [INFO] stable state for Follower reached at 2021-08-25 09:28:04.540230192 +0000 UTC m=+4.304446081 (2 nodes), 83.050301ms from start of poll, 84.361879ms from cluster start. Timeout at 2021-08-25 09:28:04.645406177 +0000 UTC m=+4.409622106, 105.176025ms after stability - testing.go:149: cluster: 2021-08-25T09:28:04.645Z [INFO] stable state for Follower reached at 2021-08-25 09:28:04.540230192 +0000 UTC m=+4.304446081 (2 nodes), 83.050301ms from start of poll, 84.361879ms from cluster start. Timeout at 2021-08-25 09:28:04.645406177 +0000 UTC m=+4.409622106, 105.176025ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.645Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:04.645Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.751Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.645773343 +0000 UTC m=+4.409989232 (1 nodes), 2.28µs from start of poll, 189.90503ms from cluster start. Timeout at 2021-08-25 09:28:04.750904772 +0000 UTC m=+4.515120741, 105.131509ms after stability - testing.go:149: cluster: 2021-08-25T09:28:04.751Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.645773343 +0000 UTC m=+4.409989232 (1 nodes), 2.28µs from start of poll, 189.90503ms from cluster start. Timeout at 2021-08-25 09:28:04.750904772 +0000 UTC m=+4.515120741, 105.131509ms after stability -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-25ad4171-2b97-13ec-8ff6-73422e0b167e 25ad4171-2b97-13ec-8ff6-73422e0b167e}" -testLoggerAdapter verbose: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" - testing.go:149: server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: 2021-08-25T09:28:04.761Z [INFO] server-63e6787b-e3aa-ada4-fa06-4d40330f7d20: aborting pipeline replication: peer="{Voter server-de11e8f1-6ded-c95f-940d-f2fe6b54457a de11e8f1-6ded-c95f-940d-f2fe6b54457a}" ---- PASS: TestRaft_LiveBootstrap (0.31s) +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.012Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: initial configuration: index=0 servers=[] + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.012Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.012Z [INFO] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: initial configuration: index=0 servers=[] + testing.go:149: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.012Z [INFO] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-801e9bb3-0387-ea88-77d9-fb7631183d54: 2021-08-25T09:31:00.012Z [INFO] server-801e9bb3-0387-ea88-77d9-fb7631183d54: initial configuration: index=0 servers=[] + testing.go:149: server-801e9bb3-0387-ea88-77d9-fb7631183d54: 2021-08-25T09:31:00.012Z [INFO] server-801e9bb3-0387-ea88-77d9-fb7631183d54: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.012Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: entering follower state: follower="Node at e6dad585-8c02-5d30-9b8b-22e27aec3544 [Follower]" leader= +testLoggerAdapter verbose: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.012Z [INFO] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: entering follower state: follower="Node at e6a50ae5-5126-81d4-adb0-402613fc9ca2 [Follower]" leader= + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.012Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: entering follower state: follower="Node at e6dad585-8c02-5d30-9b8b-22e27aec3544 [Follower]" leader= +testLoggerAdapter verbose: server-801e9bb3-0387-ea88-77d9-fb7631183d54: 2021-08-25T09:31:00.013Z [INFO] server-801e9bb3-0387-ea88-77d9-fb7631183d54: entering follower state: follower="Node at 801e9bb3-0387-ea88-77d9-fb7631183d54 [Follower]" leader= + testing.go:149: server-801e9bb3-0387-ea88-77d9-fb7631183d54: 2021-08-25T09:31:00.013Z [INFO] server-801e9bb3-0387-ea88-77d9-fb7631183d54: entering follower state: follower="Node at 801e9bb3-0387-ea88-77d9-fb7631183d54 [Follower]" leader= + testing.go:149: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.012Z [INFO] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: entering follower state: follower="Node at e6a50ae5-5126-81d4-adb0-402613fc9ca2 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.013Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:00.013Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.084Z [WARN] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: no known peers, aborting election + testing.go:149: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.084Z [WARN] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: no known peers, aborting election +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [WARN] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [WARN] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: entering candidate state: node="Node at e6dad585-8c02-5d30-9b8b-22e27aec3544 [Candidate]" term=2 + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: entering candidate state: node="Node at e6dad585-8c02-5d30-9b8b-22e27aec3544 [Candidate]" term=2 +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: election won: tally=2 + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: election won: tally=2 +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: entering leader state: leader="Node at e6dad585-8c02-5d30-9b8b-22e27aec3544 [Leader]" + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.103Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: entering leader state: leader="Node at e6dad585-8c02-5d30-9b8b-22e27aec3544 [Leader]" +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: added peer, starting replication: peer=server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: added peer, starting replication: peer=server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: added peer, starting replication: peer=server-801e9bb3-0387-ea88-77d9-fb7631183d54 + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: added peer, starting replication: peer=server-801e9bb3-0387-ea88-77d9-fb7631183d54 +testLoggerAdapter verbose: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.104Z [WARN] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: 2021-08-25T09:31:00.104Z [WARN] server-e6a50ae5-5126-81d4-adb0-402613fc9ca2: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-801e9bb3-0387-ea88-77d9-fb7631183d54: 2021-08-25T09:31:00.104Z [WARN] server-801e9bb3-0387-ea88-77d9-fb7631183d54: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-801e9bb3-0387-ea88-77d9-fb7631183d54: 2021-08-25T09:31:00.104Z [WARN] server-801e9bb3-0387-ea88-77d9-fb7631183d54: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [WARN] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: appendEntries rejected, sending older logs: peer="{Voter server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 e6a50ae5-5126-81d4-adb0-402613fc9ca2}" next=1 + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [WARN] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: appendEntries rejected, sending older logs: peer="{Voter server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 e6a50ae5-5126-81d4-adb0-402613fc9ca2}" next=1 +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [WARN] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: appendEntries rejected, sending older logs: peer="{Voter server-801e9bb3-0387-ea88-77d9-fb7631183d54 801e9bb3-0387-ea88-77d9-fb7631183d54}" next=1 + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [WARN] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: appendEntries rejected, sending older logs: peer="{Voter server-801e9bb3-0387-ea88-77d9-fb7631183d54 801e9bb3-0387-ea88-77d9-fb7631183d54}" next=1 +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: pipelining replication: peer="{Voter server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 e6a50ae5-5126-81d4-adb0-402613fc9ca2}" + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: pipelining replication: peer="{Voter server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 e6a50ae5-5126-81d4-adb0-402613fc9ca2}" +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: pipelining replication: peer="{Voter server-801e9bb3-0387-ea88-77d9-fb7631183d54 801e9bb3-0387-ea88-77d9-fb7631183d54}" + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.104Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: pipelining replication: peer="{Voter server-801e9bb3-0387-ea88-77d9-fb7631183d54 801e9bb3-0387-ea88-77d9-fb7631183d54}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.209Z [INFO] stable state for Follower reached at 2021-08-25 09:31:00.104072742 +0000 UTC m=+4.370289186 (2 nodes), 90.486564ms from start of poll, 91.673532ms from cluster start. Timeout at 2021-08-25 09:31:00.209229904 +0000 UTC m=+4.475446408, 105.157222ms after stability + testing.go:149: cluster: 2021-08-25T09:31:00.209Z [INFO] stable state for Follower reached at 2021-08-25 09:31:00.104072742 +0000 UTC m=+4.370289186 (2 nodes), 90.486564ms from start of poll, 91.673532ms from cluster start. Timeout at 2021-08-25 09:31:00.209229904 +0000 UTC m=+4.475446408, 105.157222ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.209Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:00.209Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.314Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.209624627 +0000 UTC m=+4.475841031 (1 nodes), 2.92µs from start of poll, 197.225377ms from cluster start. Timeout at 2021-08-25 09:31:00.314728175 +0000 UTC m=+4.580944599, 105.103568ms after stability + testing.go:149: cluster: 2021-08-25T09:31:00.314Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.209624627 +0000 UTC m=+4.475841031 (1 nodes), 2.92µs from start of poll, 197.225377ms from cluster start. Timeout at 2021-08-25 09:31:00.314728175 +0000 UTC m=+4.580944599, 105.103568ms after stability +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.325Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: aborting pipeline replication: peer="{Voter server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 e6a50ae5-5126-81d4-adb0-402613fc9ca2}" + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.325Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: aborting pipeline replication: peer="{Voter server-e6a50ae5-5126-81d4-adb0-402613fc9ca2 e6a50ae5-5126-81d4-adb0-402613fc9ca2}" +testLoggerAdapter verbose: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.325Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: aborting pipeline replication: peer="{Voter server-801e9bb3-0387-ea88-77d9-fb7631183d54 801e9bb3-0387-ea88-77d9-fb7631183d54}" + testing.go:149: server-e6dad585-8c02-5d30-9b8b-22e27aec3544: 2021-08-25T09:31:00.325Z [INFO] server-e6dad585-8c02-5d30-9b8b-22e27aec3544: aborting pipeline replication: peer="{Voter server-801e9bb3-0387-ea88-77d9-fb7631183d54 801e9bb3-0387-ea88-77d9-fb7631183d54}" +--- PASS: TestRaft_LiveBootstrap (0.32s) === RUN TestRaft_RecoverCluster_NoState -testLoggerAdapter verbose: server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: 2021-08-25T09:28:04.764Z [INFO] server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: initial configuration: index=0 servers=[] - testing.go:149: server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: 2021-08-25T09:28:04.764Z [INFO] server-c7ae3f63-8e42-8412-14f1-ae6215e494ec: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: 2021-08-25T09:31:00.328Z [INFO] server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: initial configuration: index=0 servers=[] + testing.go:149: server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: 2021-08-25T09:31:00.328Z [INFO] server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: 2021-08-25T09:31:00.328Z [INFO] server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: entering follower state: follower="Node at 8606353d-9f2c-7f8c-8e48-47fb87b81ca9 [Shutdown]" leader= + testing.go:149: server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: 2021-08-25T09:31:00.328Z [INFO] server-8606353d-9f2c-7f8c-8e48-47fb87b81ca9: entering follower state: follower="Node at 8606353d-9f2c-7f8c-8e48-47fb87b81ca9 [Shutdown]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.766Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.766Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.767Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering follower state: follower="Node at c134af69-1f82-2666-538f-df2b135d8202 [Follower]" leader= - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.767Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering follower state: follower="Node at c134af69-1f82-2666-538f-df2b135d8202 [Follower]" leader= -testLoggerAdapter verbose: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" - testing.go:149: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5}]" -testLoggerAdapter verbose: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: entering follower state: follower="Node at caf1f50e-e306-16d6-df1e-1511f7c5cfa5 [Follower]" leader= - testing.go:149: server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: 2021-08-25T09:28:04.767Z [INFO] server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5: entering follower state: follower="Node at caf1f50e-e306-16d6-df1e-1511f7c5cfa5 [Follower]" leader= -testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.767Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: initial configuration: index=0 servers=[] - testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.767Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.768Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:04.768Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.768Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: entering follower state: follower="Node at e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 [Follower]" leader= - testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.768Z [INFO] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: entering follower state: follower="Node at e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 [Follower]" leader= -testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.826Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: no known peers, aborting election - testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.826Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: no known peers, aborting election -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering candidate state: node="Node at c134af69-1f82-2666-538f-df2b135d8202 [Candidate]" term=2 - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.830Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering candidate state: node="Node at c134af69-1f82-2666-538f-df2b135d8202 [Candidate]" term=2 -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: election won: tally=2 - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: election won: tally=2 -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering leader state: leader="Node at c134af69-1f82-2666-538f-df2b135d8202 [Leader]" - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: entering leader state: leader="Node at c134af69-1f82-2666-538f-df2b135d8202 [Leader]" -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.831Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.936Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.831372293 +0000 UTC m=+4.595588182 (1 nodes), 63.191907ms from start of poll, 64.649392ms from cluster start. Timeout at 2021-08-25 09:28:04.936507622 +0000 UTC m=+4.700723591, 105.135409ms after stability - testing.go:149: cluster: 2021-08-25T09:28:04.936Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.831372293 +0000 UTC m=+4.595588182 (1 nodes), 63.191907ms from start of poll, 64.649392ms from cluster start. Timeout at 2021-08-25 09:28:04.936507622 +0000 UTC m=+4.700723591, 105.135409ms after stability -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.936Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: updating configuration: command=AddStaging server-id=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 server-addr=e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5} {Suffrage:Voter ID:server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 Address:e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}]" - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.936Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: updating configuration: command=AddStaging server-id=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 server-addr=e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 servers="[{Suffrage:Voter ID:server-c134af69-1f82-2666-538f-df2b135d8202 Address:c134af69-1f82-2666-538f-df2b135d8202} {Suffrage:Voter ID:server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 Address:caf1f50e-e306-16d6-df1e-1511f7c5cfa5} {Suffrage:Voter ID:server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 Address:e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}]" -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: added peer, starting replication: peer=server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 -testLoggerAdapter verbose: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.937Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: 2021-08-25T09:28:04.937Z [WARN] server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: appendEntries rejected, sending older logs: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" next=1 - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [WARN] server-c134af69-1f82-2666-538f-df2b135d8202: appendEntries rejected, sending older logs: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" next=1 -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:04.937Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: pipelining replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:04.937Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:04.937Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.043Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.937804001 +0000 UTC m=+4.702019890 (1 nodes), 1.5µs from start of poll, 171.0811ms from cluster start. Timeout at 2021-08-25 09:28:05.042923251 +0000 UTC m=+4.807139160, 105.11927ms after stability - testing.go:149: cluster: 2021-08-25T09:28:05.043Z [INFO] stable state for Leader reached at 2021-08-25 09:28:04.937804001 +0000 UTC m=+4.702019890 (1 nodes), 1.5µs from start of poll, 171.0811ms from cluster start. Timeout at 2021-08-25 09:28:05.042923251 +0000 UTC m=+4.807139160, 105.11927ms after stability -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003 e0bb4435-fbed-c0cf-d3e4-0cc3c98e2003}" -testLoggerAdapter verbose: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" - testing.go:149: server-c134af69-1f82-2666-538f-df2b135d8202: 2021-08-25T09:28:05.043Z [INFO] server-c134af69-1f82-2666-538f-df2b135d8202: aborting pipeline replication: peer="{Voter server-caf1f50e-e306-16d6-df1e-1511f7c5cfa5 caf1f50e-e306-16d6-df1e-1511f7c5cfa5}" ---- PASS: TestRaft_HasExistingState (0.28s) +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.330Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f Address:a0ccecb6-c39b-654b-f606-75bb4aec1e9f} {Suffrage:Voter ID:server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 Address:f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}]" + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.330Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f Address:a0ccecb6-c39b-654b-f606-75bb4aec1e9f} {Suffrage:Voter ID:server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 Address:f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}]" +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.330Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: entering follower state: follower="Node at a0ccecb6-c39b-654b-f606-75bb4aec1e9f [Follower]" leader= + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.330Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: entering follower state: follower="Node at a0ccecb6-c39b-654b-f606-75bb4aec1e9f [Follower]" leader= +testLoggerAdapter verbose: server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: 2021-08-25T09:31:00.331Z [INFO] server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f Address:a0ccecb6-c39b-654b-f606-75bb4aec1e9f} {Suffrage:Voter ID:server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 Address:f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}]" + testing.go:149: server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: 2021-08-25T09:31:00.331Z [INFO] server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f Address:a0ccecb6-c39b-654b-f606-75bb4aec1e9f} {Suffrage:Voter ID:server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 Address:f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}]" +testLoggerAdapter verbose: server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: 2021-08-25T09:31:00.331Z [INFO] server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: entering follower state: follower="Node at f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 [Follower]" leader= + testing.go:149: server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: 2021-08-25T09:31:00.331Z [INFO] server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9: entering follower state: follower="Node at f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 [Follower]" leader= +testLoggerAdapter verbose: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.331Z [INFO] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: initial configuration: index=0 servers=[] + testing.go:149: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.331Z [INFO] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.332Z [INFO] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: entering follower state: follower="Node at 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d [Follower]" leader= +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.332Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.332Z [INFO] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: entering follower state: follower="Node at 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d [Follower]" leader= + testing.go:149: cluster: 2021-08-25T09:31:00.332Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.403Z [WARN] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.403Z [WARN] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.403Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: entering candidate state: node="Node at a0ccecb6-c39b-654b-f606-75bb4aec1e9f [Candidate]" term=2 + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.403Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: entering candidate state: node="Node at a0ccecb6-c39b-654b-f606-75bb4aec1e9f [Candidate]" term=2 +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: election won: tally=2 + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: election won: tally=2 +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: entering leader state: leader="Node at a0ccecb6-c39b-654b-f606-75bb4aec1e9f [Leader]" + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: entering leader state: leader="Node at a0ccecb6-c39b-654b-f606-75bb4aec1e9f [Leader]" +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: added peer, starting replication: peer=server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: added peer, starting replication: peer=server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: pipelining replication: peer="{Voter server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}" + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.404Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: pipelining replication: peer="{Voter server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}" +testLoggerAdapter verbose: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.414Z [WARN] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: no known peers, aborting election + testing.go:149: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.414Z [WARN] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.509Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.404300004 +0000 UTC m=+4.670516448 (1 nodes), 72.001883ms from start of poll, 73.777856ms from cluster start. Timeout at 2021-08-25 09:31:00.509403792 +0000 UTC m=+4.775620256, 105.103808ms after stability + testing.go:149: cluster: 2021-08-25T09:31:00.509Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.404300004 +0000 UTC m=+4.670516448 (1 nodes), 72.001883ms from start of poll, 73.777856ms from cluster start. Timeout at 2021-08-25 09:31:00.509403792 +0000 UTC m=+4.775620256, 105.103808ms after stability +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.509Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: updating configuration: command=AddStaging server-id=server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d server-addr=8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d servers="[{Suffrage:Voter ID:server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f Address:a0ccecb6-c39b-654b-f606-75bb4aec1e9f} {Suffrage:Voter ID:server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 Address:f988c234-2c4e-5b85-9d8f-ba864c3dc0d9} {Suffrage:Voter ID:server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d Address:8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}]" + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.509Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: updating configuration: command=AddStaging server-id=server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d server-addr=8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d servers="[{Suffrage:Voter ID:server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f Address:a0ccecb6-c39b-654b-f606-75bb4aec1e9f} {Suffrage:Voter ID:server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 Address:f988c234-2c4e-5b85-9d8f-ba864c3dc0d9} {Suffrage:Voter ID:server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d Address:8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}]" +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.510Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: added peer, starting replication: peer=server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.510Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: added peer, starting replication: peer=server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d +testLoggerAdapter verbose: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.510Z [WARN] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: 2021-08-25T09:31:00.510Z [WARN] server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.510Z [WARN] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: appendEntries rejected, sending older logs: peer="{Voter server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}" next=1 + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.510Z [WARN] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: appendEntries rejected, sending older logs: peer="{Voter server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}" next=1 +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.510Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: pipelining replication: peer="{Voter server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}" + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.510Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: pipelining replication: peer="{Voter server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.515Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:00.515Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.620Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.515731096 +0000 UTC m=+4.781947480 (1 nodes), 1.919µs from start of poll, 185.208888ms from cluster start. Timeout at 2021-08-25 09:31:00.620829965 +0000 UTC m=+4.887046389, 105.098909ms after stability + testing.go:149: cluster: 2021-08-25T09:31:00.620Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.515731096 +0000 UTC m=+4.781947480 (1 nodes), 1.919µs from start of poll, 185.208888ms from cluster start. Timeout at 2021-08-25 09:31:00.620829965 +0000 UTC m=+4.887046389, 105.098909ms after stability +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.621Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: aborting pipeline replication: peer="{Voter server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}" + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.621Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: aborting pipeline replication: peer="{Voter server-8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d 8f5a7bb6-e396-75c8-c28b-5487f1fc8f0d}" +testLoggerAdapter verbose: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.621Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: aborting pipeline replication: peer="{Voter server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}" + testing.go:149: server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: 2021-08-25T09:31:00.621Z [INFO] server-a0ccecb6-c39b-654b-f606-75bb4aec1e9f: aborting pipeline replication: peer="{Voter server-f988c234-2c4e-5b85-9d8f-ba864c3dc0d9 f988c234-2c4e-5b85-9d8f-ba864c3dc0d9}" +--- PASS: TestRaft_HasExistingState (0.29s) === RUN TestRaft_SingleNode -testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa Address:8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa}]" - testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa Address:8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa}]" -testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering follower state: follower="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Follower]" leader= - testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.046Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering follower state: follower="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Follower]" leader= -testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.124Z [WARN] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.124Z [WARN] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering candidate state: node="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Candidate]" term=2 - testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering candidate state: node="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Candidate]" term=2 -testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: election won: tally=1 - testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: election won: tally=1 -testLoggerAdapter verbose: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering leader state: leader="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Leader]" - testing.go:149: server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: 2021-08-25T09:28:05.125Z [INFO] server-8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa: entering leader state: leader="Node at 8ad7033e-fcd9-ecc6-1dfb-df19584a2ffa [Leader]" +testLoggerAdapter verbose: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.623Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff2f47fd-a459-0a31-905d-6a101fe701b6 Address:ff2f47fd-a459-0a31-905d-6a101fe701b6}]" + testing.go:149: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.623Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff2f47fd-a459-0a31-905d-6a101fe701b6 Address:ff2f47fd-a459-0a31-905d-6a101fe701b6}]" +testLoggerAdapter verbose: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.624Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: entering follower state: follower="Node at ff2f47fd-a459-0a31-905d-6a101fe701b6 [Follower]" leader= + testing.go:149: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.624Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: entering follower state: follower="Node at ff2f47fd-a459-0a31-905d-6a101fe701b6 [Follower]" leader= +testLoggerAdapter verbose: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.698Z [WARN] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.698Z [WARN] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.699Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: entering candidate state: node="Node at ff2f47fd-a459-0a31-905d-6a101fe701b6 [Candidate]" term=2 + testing.go:149: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.699Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: entering candidate state: node="Node at ff2f47fd-a459-0a31-905d-6a101fe701b6 [Candidate]" term=2 +testLoggerAdapter verbose: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.699Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: election won: tally=1 + testing.go:149: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.699Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: election won: tally=1 +testLoggerAdapter verbose: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.699Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: entering leader state: leader="Node at ff2f47fd-a459-0a31-905d-6a101fe701b6 [Leader]" + testing.go:149: server-ff2f47fd-a459-0a31-905d-6a101fe701b6: 2021-08-25T09:31:00.699Z [INFO] server-ff2f47fd-a459-0a31-905d-6a101fe701b6: entering leader state: leader="Node at ff2f47fd-a459-0a31-905d-6a101fe701b6 [Leader]" --- PASS: TestRaft_SingleNode (0.08s) === RUN TestRaft_TripleNode -testLoggerAdapter verbose: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.127Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" - testing.go:149: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.127Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" -testLoggerAdapter verbose: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" - testing.go:149: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 Address:ea11f431-3758-3c59-4d9d-3da5a7e750c9} {Suffrage:Voter ID:server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 Address:040fcb8a-9aa2-6b4a-4b67-3f480a7cd767} {Suffrage:Voter ID:server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89 Address:8cb07943-dccb-81e4-0c69-cf4ecf84bd89}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.128Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:05.128Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.128Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: entering follower state: follower="Node at ea11f431-3758-3c59-4d9d-3da5a7e750c9 [Follower]" leader= - testing.go:149: server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: 2021-08-25T09:28:05.128Z [INFO] server-ea11f431-3758-3c59-4d9d-3da5a7e750c9: entering follower state: follower="Node at ea11f431-3758-3c59-4d9d-3da5a7e750c9 [Follower]" leader= -testLoggerAdapter verbose: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: entering follower state: follower="Node at 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 [Follower]" leader= - testing.go:149: server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: 2021-08-25T09:28:05.128Z [INFO] server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767: entering follower state: follower="Node at 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 [Follower]" leader= -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering follower state: follower="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Follower]" leader= - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.128Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering follower state: follower="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Follower]" leader= -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [WARN] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [WARN] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering candidate state: node="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Candidate]" term=2 - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering candidate state: node="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Candidate]" term=2 -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: election won: tally=2 - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: election won: tally=2 -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering leader state: leader="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Leader]" - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: entering leader state: leader="Node at 8cb07943-dccb-81e4-0c69-cf4ecf84bd89 [Leader]" -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.181Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: added peer, starting replication: peer=server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.182Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: pipelining replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.287Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.181798484 +0000 UTC m=+4.946014393 (2 nodes), 53.097565ms from start of poll, 54.11837ms from cluster start. Timeout at 2021-08-25 09:28:05.286920694 +0000 UTC m=+5.051136603, 105.12221ms after stability - testing.go:149: cluster: 2021-08-25T09:28:05.287Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.181798484 +0000 UTC m=+4.946014393 (2 nodes), 53.097565ms from start of poll, 54.11837ms from cluster start. Timeout at 2021-08-25 09:28:05.286920694 +0000 UTC m=+5.051136603, 105.12221ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.287Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:05.287Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.392Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.287208987 +0000 UTC m=+5.051424896 (1 nodes), 1.98µs from start of poll, 159.528873ms from cluster start. Timeout at 2021-08-25 09:28:05.392316878 +0000 UTC m=+5.156532867, 105.107971ms after stability - testing.go:149: cluster: 2021-08-25T09:28:05.392Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.287208987 +0000 UTC m=+5.051424896 (1 nodes), 1.98µs from start of poll, 159.528873ms from cluster start. Timeout at 2021-08-25 09:28:05.392316878 +0000 UTC m=+5.156532867, 105.107971ms after stability -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-040fcb8a-9aa2-6b4a-4b67-3f480a7cd767 040fcb8a-9aa2-6b4a-4b67-3f480a7cd767}" -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.418Z [INFO] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: aborting pipeline replication: peer="{Voter server-ea11f431-3758-3c59-4d9d-3da5a7e750c9 ea11f431-3758-3c59-4d9d-3da5a7e750c9}" -testLoggerAdapter verbose: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.469Z [ERROR] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: failed to heartbeat to: peer=ea11f431-3758-3c59-4d9d-3da5a7e750c9 error="command timed out" - testing.go:149: server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: 2021-08-25T09:28:05.469Z [ERROR] server-8cb07943-dccb-81e4-0c69-cf4ecf84bd89: failed to heartbeat to: peer=ea11f431-3758-3c59-4d9d-3da5a7e750c9 error="command timed out" +testLoggerAdapter verbose: server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: 2021-08-25T09:31:00.702Z [INFO] server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a Address:8e4053e4-853c-c4d2-c2e9-5e2430ed666a} {Suffrage:Voter ID:server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 Address:0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9} {Suffrage:Voter ID:server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 Address:6bf51d3a-dee5-70cc-239c-82e0fe292dd9}]" + testing.go:149: server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: 2021-08-25T09:31:00.702Z [INFO] server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a Address:8e4053e4-853c-c4d2-c2e9-5e2430ed666a} {Suffrage:Voter ID:server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 Address:0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9} {Suffrage:Voter ID:server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 Address:6bf51d3a-dee5-70cc-239c-82e0fe292dd9}]" +testLoggerAdapter verbose: server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: 2021-08-25T09:31:00.702Z [INFO] server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: entering follower state: follower="Node at 8e4053e4-853c-c4d2-c2e9-5e2430ed666a [Follower]" leader= + testing.go:149: server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: 2021-08-25T09:31:00.702Z [INFO] server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a: entering follower state: follower="Node at 8e4053e4-853c-c4d2-c2e9-5e2430ed666a [Follower]" leader= +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.705Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a Address:8e4053e4-853c-c4d2-c2e9-5e2430ed666a} {Suffrage:Voter ID:server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 Address:0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9} {Suffrage:Voter ID:server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 Address:6bf51d3a-dee5-70cc-239c-82e0fe292dd9}]" + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.705Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a Address:8e4053e4-853c-c4d2-c2e9-5e2430ed666a} {Suffrage:Voter ID:server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 Address:0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9} {Suffrage:Voter ID:server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 Address:6bf51d3a-dee5-70cc-239c-82e0fe292dd9}]" +testLoggerAdapter verbose: server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: 2021-08-25T09:31:00.705Z [INFO] server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a Address:8e4053e4-853c-c4d2-c2e9-5e2430ed666a} {Suffrage:Voter ID:server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 Address:0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9} {Suffrage:Voter ID:server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 Address:6bf51d3a-dee5-70cc-239c-82e0fe292dd9}]" + testing.go:149: server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: 2021-08-25T09:31:00.705Z [INFO] server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a Address:8e4053e4-853c-c4d2-c2e9-5e2430ed666a} {Suffrage:Voter ID:server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 Address:0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9} {Suffrage:Voter ID:server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 Address:6bf51d3a-dee5-70cc-239c-82e0fe292dd9}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.705Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:00.705Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.705Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: entering follower state: follower="Node at 0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 [Follower]" leader= + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.705Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: entering follower state: follower="Node at 0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 [Follower]" leader= +testLoggerAdapter verbose: server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: 2021-08-25T09:31:00.705Z [INFO] server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: entering follower state: follower="Node at 6bf51d3a-dee5-70cc-239c-82e0fe292dd9 [Follower]" leader= + testing.go:149: server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: 2021-08-25T09:31:00.705Z [INFO] server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9: entering follower state: follower="Node at 6bf51d3a-dee5-70cc-239c-82e0fe292dd9 [Follower]" leader= +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.765Z [WARN] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.765Z [WARN] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.765Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: entering candidate state: node="Node at 0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 [Candidate]" term=2 + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.765Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: entering candidate state: node="Node at 0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 [Candidate]" term=2 +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: election won: tally=2 + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: election won: tally=2 +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: entering leader state: leader="Node at 0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 [Leader]" + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: entering leader state: leader="Node at 0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9 [Leader]" +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: added peer, starting replication: peer=server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: added peer, starting replication: peer=server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: added peer, starting replication: peer=server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: added peer, starting replication: peer=server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: pipelining replication: peer="{Voter server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a 8e4053e4-853c-c4d2-c2e9-5e2430ed666a}" + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: pipelining replication: peer="{Voter server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a 8e4053e4-853c-c4d2-c2e9-5e2430ed666a}" +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: pipelining replication: peer="{Voter server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 6bf51d3a-dee5-70cc-239c-82e0fe292dd9}" + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.766Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: pipelining replication: peer="{Voter server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 6bf51d3a-dee5-70cc-239c-82e0fe292dd9}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.871Z [INFO] stable state for Follower reached at 2021-08-25 09:31:00.766412802 +0000 UTC m=+5.032629226 (2 nodes), 60.900668ms from start of poll, 64.434355ms from cluster start. Timeout at 2021-08-25 09:31:00.871475454 +0000 UTC m=+5.137691898, 105.062672ms after stability + testing.go:149: cluster: 2021-08-25T09:31:00.871Z [INFO] stable state for Follower reached at 2021-08-25 09:31:00.766412802 +0000 UTC m=+5.032629226 (2 nodes), 60.900668ms from start of poll, 64.434355ms from cluster start. Timeout at 2021-08-25 09:31:00.871475454 +0000 UTC m=+5.137691898, 105.062672ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.871Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:00.871Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:00.977Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.87183084 +0000 UTC m=+5.138047224 (1 nodes), 2.559µs from start of poll, 169.852353ms from cluster start. Timeout at 2021-08-25 09:31:00.976961665 +0000 UTC m=+5.243178129, 105.130905ms after stability + testing.go:149: cluster: 2021-08-25T09:31:00.977Z [INFO] stable state for Leader reached at 2021-08-25 09:31:00.87183084 +0000 UTC m=+5.138047224 (1 nodes), 2.559µs from start of poll, 169.852353ms from cluster start. Timeout at 2021-08-25 09:31:00.976961665 +0000 UTC m=+5.243178129, 105.130905ms after stability +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.987Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: aborting pipeline replication: peer="{Voter server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 6bf51d3a-dee5-70cc-239c-82e0fe292dd9}" + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.987Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: aborting pipeline replication: peer="{Voter server-6bf51d3a-dee5-70cc-239c-82e0fe292dd9 6bf51d3a-dee5-70cc-239c-82e0fe292dd9}" +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.987Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: aborting pipeline replication: peer="{Voter server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a 8e4053e4-853c-c4d2-c2e9-5e2430ed666a}" + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:00.987Z [INFO] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: aborting pipeline replication: peer="{Voter server-8e4053e4-853c-c4d2-c2e9-5e2430ed666a 8e4053e4-853c-c4d2-c2e9-5e2430ed666a}" +testLoggerAdapter verbose: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:01.038Z [ERROR] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: failed to heartbeat to: peer=8e4053e4-853c-c4d2-c2e9-5e2430ed666a error="command timed out" + testing.go:149: server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: 2021-08-25T09:31:01.038Z [ERROR] server-0e93c0c4-0766-b6b7-26e7-bdc19a4a07f9: failed to heartbeat to: peer=8e4053e4-853c-c4d2-c2e9-5e2430ed666a error="command timed out" --- PASS: TestRaft_TripleNode (0.34s) === RUN TestRaft_LeaderFail -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering follower state: follower="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Follower]" leader= - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.473Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering follower state: follower="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Follower]" leader= -testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" - testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" -testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= - testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.474Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 Address:6a039ee3-59cc-2aa4-7ff0-507c5b9a4248} {Suffrage:Voter ID:server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 Address:2d3bdee3-711d-1243-4303-aa6a47ff05b4} {Suffrage:Voter ID:server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 Address:99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.474Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:05.474Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.474Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=2 - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=2 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: election won: tally=2 - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.528Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: election won: tally=2 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering leader state: leader="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Leader]" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering leader state: leader="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Leader]" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.529Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: pipelining replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.634Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.529745746 +0000 UTC m=+5.293961655 (2 nodes), 55.087081ms from start of poll, 56.542025ms from cluster start. Timeout at 2021-08-25 09:28:05.634871835 +0000 UTC m=+5.399087804, 105.126149ms after stability - testing.go:149: cluster: 2021-08-25T09:28:05.634Z [INFO] stable state for Follower reached at 2021-08-25 09:28:05.529745746 +0000 UTC m=+5.293961655 (2 nodes), 55.087081ms from start of poll, 56.542025ms from cluster start. Timeout at 2021-08-25 09:28:05.634871835 +0000 UTC m=+5.399087804, 105.126149ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.635Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:05.635Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.740Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.635205904 +0000 UTC m=+5.399421793 (1 nodes), 2.26µs from start of poll, 162.002163ms from cluster start. Timeout at 2021-08-25 09:28:05.740207365 +0000 UTC m=+5.504423254, 105.001461ms after stability - testing.go:149: cluster: 2021-08-25T09:28:05.740Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.635205904 +0000 UTC m=+5.399421793 (1 nodes), 2.26µs from start of poll, 162.002163ms from cluster start. Timeout at 2021-08-25 09:28:05.740207365 +0000 UTC m=+5.504423254, 105.001461ms after stability - raft_test.go:336: [INFO] Disconnecting Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Leader] -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.752Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="append pipeline closed" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="append pipeline closed" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.755Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.756Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:05.756Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="append pipeline closed" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to pipeline appendEntries: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="append pipeline closed" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.756Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: aborting pipeline replication: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.762Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.762Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.766Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.766Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.768Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.768Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.772Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.772Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.781Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.781Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.786Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.786Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.788Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.788Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.790Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.790Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to heartbeat to: peer=2d3bdee3-711d-1243-4303-aa6a47ff05b4 error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.796Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=50.107663ms - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.796Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=50.107663ms -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.801Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.801Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=3 - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=3 -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.801Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.801Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: rejecting vote request since we have a leader: from=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 - testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.801Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: rejecting vote request since we have a leader: from=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.803Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.803Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to appendEntries to: peer="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.806Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 time=59.466153ms - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.806Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 time=59.466153ms -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=60.398987ms - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact: server-id=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 time=60.398987ms -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact quorum of nodes, stepping down - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.807Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader= -testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 - testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [WARN] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: heartbeat timeout reached, starting election: last-leader=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 -testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering candidate state: node="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Candidate]" term=3 - testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering candidate state: node="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Candidate]" term=3 -testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [ERROR] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.808Z [ERROR] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.808Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: duplicate requestVote for same term: term=3 - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.808Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.860Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.860Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=3 - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=3 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.861Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: Election timeout reached, restarting election - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: Election timeout reached, restarting election -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=4 - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.866Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering candidate state: node="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Candidate]" term=4 -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to make requestVote RPC: target="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.867Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= - testing.go:149: server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: 2021-08-25T09:28:05.867Z [INFO] server-2d3bdee3-711d-1243-4303-aa6a47ff05b4: entering follower state: follower="Node at 2d3bdee3-711d-1243-4303-aa6a47ff05b4 [Follower]" leader= -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: election won: tally=2 - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: election won: tally=2 -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering leader state: leader="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Leader]" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: entering leader state: leader="Node at 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 [Leader]" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: added peer, starting replication: peer=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.867Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.877Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.877Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.885Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.885Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.894Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.894Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.902Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.902Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.914Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.914Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: Election timeout reached, restarting election - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [WARN] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: Election timeout reached, restarting election -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=4 - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering candidate state: node="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate]" term=4 -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" error="failed to connect to peer: 2d3bdee3-711d-1243-4303-aa6a47ff05b4" -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.914Z [ERROR] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: failed to make requestVote RPC: target="{Voter server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248}" error="failed to connect to peer: 6a039ee3-59cc-2aa4-7ff0-507c5b9a4248" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.917Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=50.186736ms - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.917Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=50.186736ms -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.931Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.931Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to appendEntries to: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.940Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.940Z [ERROR] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to heartbeat to: peer=99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 error="failed to connect to peer: 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.967Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=99.74539ms - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.967Z [WARN] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: failed to contact: server-id=server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 time=99.74539ms -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.972Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.867415502 +0000 UTC m=+5.631631431 (1 nodes), 111.215603ms from start of poll, 394.211801ms from cluster start. Timeout at 2021-08-25 09:28:05.972487477 +0000 UTC m=+5.736703386, 105.071955ms after stability - testing.go:149: cluster: 2021-08-25T09:28:05.972Z [INFO] stable state for Leader reached at 2021-08-25 09:28:05.867415502 +0000 UTC m=+5.631631431 (1 nodes), 111.215603ms from start of poll, 394.211801ms from cluster start. Timeout at 2021-08-25 09:28:05.972487477 +0000 UTC m=+5.736703386, 105.071955ms after stability - raft_test.go:371: [INFO] Reconnecting Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Candidate] -testLoggerAdapter verbose: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.977Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 - testing.go:149: server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: 2021-08-25T09:28:05.977Z [INFO] server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118: entering follower state: follower="Node at 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 [Follower]" leader=6a039ee3-59cc-2aa4-7ff0-507c5b9a4248 -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.977Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.977Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: pipelining replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-99c5b0e4-f8f3-f177-2de3-0dc55dcd3118 99c5b0e4-f8f3-f177-2de3-0dc55dcd3118}" -testLoggerAdapter verbose: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" - testing.go:149: server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: 2021-08-25T09:28:05.982Z [INFO] server-6a039ee3-59cc-2aa4-7ff0-507c5b9a4248: aborting pipeline replication: peer="{Voter server-2d3bdee3-711d-1243-4303-aa6a47ff05b4 2d3bdee3-711d-1243-4303-aa6a47ff05b4}" ---- PASS: TestRaft_LeaderFail (0.51s) +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.043Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c880f81a-325f-f987-5534-6efe79a4aa0d Address:c880f81a-325f-f987-5534-6efe79a4aa0d} {Suffrage:Voter ID:server-58633671-e0f2-ffc4-96f8-80811db6bec1 Address:58633671-e0f2-ffc4-96f8-80811db6bec1} {Suffrage:Voter ID:server-c728053a-382d-a4a2-d470-cf8beb8767be Address:c728053a-382d-a4a2-d470-cf8beb8767be}]" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.043Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c880f81a-325f-f987-5534-6efe79a4aa0d Address:c880f81a-325f-f987-5534-6efe79a4aa0d} {Suffrage:Voter ID:server-58633671-e0f2-ffc4-96f8-80811db6bec1 Address:58633671-e0f2-ffc4-96f8-80811db6bec1} {Suffrage:Voter ID:server-c728053a-382d-a4a2-d470-cf8beb8767be Address:c728053a-382d-a4a2-d470-cf8beb8767be}]" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.043Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering follower state: follower="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Follower]" leader= + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.043Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering follower state: follower="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Follower]" leader= +testLoggerAdapter verbose: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.044Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c880f81a-325f-f987-5534-6efe79a4aa0d Address:c880f81a-325f-f987-5534-6efe79a4aa0d} {Suffrage:Voter ID:server-58633671-e0f2-ffc4-96f8-80811db6bec1 Address:58633671-e0f2-ffc4-96f8-80811db6bec1} {Suffrage:Voter ID:server-c728053a-382d-a4a2-d470-cf8beb8767be Address:c728053a-382d-a4a2-d470-cf8beb8767be}]" + testing.go:149: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.044Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c880f81a-325f-f987-5534-6efe79a4aa0d Address:c880f81a-325f-f987-5534-6efe79a4aa0d} {Suffrage:Voter ID:server-58633671-e0f2-ffc4-96f8-80811db6bec1 Address:58633671-e0f2-ffc4-96f8-80811db6bec1} {Suffrage:Voter ID:server-c728053a-382d-a4a2-d470-cf8beb8767be Address:c728053a-382d-a4a2-d470-cf8beb8767be}]" +testLoggerAdapter verbose: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.044Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: entering follower state: follower="Node at 58633671-e0f2-ffc4-96f8-80811db6bec1 [Follower]" leader= + testing.go:149: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.044Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: entering follower state: follower="Node at 58633671-e0f2-ffc4-96f8-80811db6bec1 [Follower]" leader= +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.044Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c880f81a-325f-f987-5534-6efe79a4aa0d Address:c880f81a-325f-f987-5534-6efe79a4aa0d} {Suffrage:Voter ID:server-58633671-e0f2-ffc4-96f8-80811db6bec1 Address:58633671-e0f2-ffc4-96f8-80811db6bec1} {Suffrage:Voter ID:server-c728053a-382d-a4a2-d470-cf8beb8767be Address:c728053a-382d-a4a2-d470-cf8beb8767be}]" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.044Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c880f81a-325f-f987-5534-6efe79a4aa0d Address:c880f81a-325f-f987-5534-6efe79a4aa0d} {Suffrage:Voter ID:server-58633671-e0f2-ffc4-96f8-80811db6bec1 Address:58633671-e0f2-ffc4-96f8-80811db6bec1} {Suffrage:Voter ID:server-c728053a-382d-a4a2-d470-cf8beb8767be Address:c728053a-382d-a4a2-d470-cf8beb8767be}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.044Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:01.044Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.044Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering follower state: follower="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Follower]" leader= + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.044Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering follower state: follower="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Follower]" leader= +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.107Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.107Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.107Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering candidate state: node="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Candidate]" term=2 + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.107Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering candidate state: node="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Candidate]" term=2 +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.107Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: election won: tally=2 + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.107Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: election won: tally=2 +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering leader state: leader="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Leader]" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering leader state: leader="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Leader]" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: added peer, starting replication: peer=server-58633671-e0f2-ffc4-96f8-80811db6bec1 + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: added peer, starting replication: peer=server-58633671-e0f2-ffc4-96f8-80811db6bec1 +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: added peer, starting replication: peer=server-c728053a-382d-a4a2-d470-cf8beb8767be + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: added peer, starting replication: peer=server-c728053a-382d-a4a2-d470-cf8beb8767be +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: pipelining replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: pipelining replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: pipelining replication: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.108Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: pipelining replication: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.213Z [INFO] stable state for Follower reached at 2021-08-25 09:31:01.10823033 +0000 UTC m=+5.374446774 (2 nodes), 63.548538ms from start of poll, 64.844496ms from cluster start. Timeout at 2021-08-25 09:31:01.213382354 +0000 UTC m=+5.479598818, 105.152044ms after stability + testing.go:149: cluster: 2021-08-25T09:31:01.213Z [INFO] stable state for Follower reached at 2021-08-25 09:31:01.10823033 +0000 UTC m=+5.374446774 (2 nodes), 63.548538ms from start of poll, 64.844496ms from cluster start. Timeout at 2021-08-25 09:31:01.213382354 +0000 UTC m=+5.479598818, 105.152044ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.213Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:01.213Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.319Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.213788895 +0000 UTC m=+5.480005279 (1 nodes), 2.499µs from start of poll, 170.403001ms from cluster start. Timeout at 2021-08-25 09:31:01.318948198 +0000 UTC m=+5.585164662, 105.159383ms after stability + testing.go:149: cluster: 2021-08-25T09:31:01.319Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.213788895 +0000 UTC m=+5.480005279 (1 nodes), 2.499µs from start of poll, 170.403001ms from cluster start. Timeout at 2021-08-25 09:31:01.318948198 +0000 UTC m=+5.585164662, 105.159383ms after stability + raft_test.go:336: [INFO] Disconnecting Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Leader] +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.334Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.334Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.334Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.334Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.335Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:01.335Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.335Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to pipeline appendEntries: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="append pipeline closed" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.335Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to pipeline appendEntries: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="append pipeline closed" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.335Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: aborting pipeline replication: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.335Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: aborting pipeline replication: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.336Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to pipeline appendEntries: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="append pipeline closed" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.336Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to pipeline appendEntries: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="append pipeline closed" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.336Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: aborting pipeline replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.336Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: aborting pipeline replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.343Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.343Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.343Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.343Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.350Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.350Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.352Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.352Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.363Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.363Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.363Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.363Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.365Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.365Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.372Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.372Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.378Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact: server-id=server-c728053a-382d-a4a2-d470-cf8beb8767be time=50.128122ms + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.378Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact: server-id=server-c728053a-382d-a4a2-d470-cf8beb8767be time=50.128122ms +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.380Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.380Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.381Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.381Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.382Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: heartbeat timeout reached, starting election: last-leader=c880f81a-325f-f987-5534-6efe79a4aa0d + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.382Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: heartbeat timeout reached, starting election: last-leader=c880f81a-325f-f987-5534-6efe79a4aa0d +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.383Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering candidate state: node="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Candidate]" term=3 + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.383Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering candidate state: node="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Candidate]" term=3 +testLoggerAdapter verbose: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.383Z [WARN] server-58633671-e0f2-ffc4-96f8-80811db6bec1: rejecting vote request since we have a leader: from=c728053a-382d-a4a2-d470-cf8beb8767be leader=c880f81a-325f-f987-5534-6efe79a4aa0d +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.383Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to make requestVote RPC: target="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.383Z [WARN] server-58633671-e0f2-ffc4-96f8-80811db6bec1: rejecting vote request since we have a leader: from=c728053a-382d-a4a2-d470-cf8beb8767be leader=c880f81a-325f-f987-5534-6efe79a4aa0d + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.383Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to make requestVote RPC: target="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact: server-id=server-58633671-e0f2-ffc4-96f8-80811db6bec1 time=59.911821ms + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact: server-id=server-58633671-e0f2-ffc4-96f8-80811db6bec1 time=59.911821ms +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact: server-id=server-c728053a-382d-a4a2-d470-cf8beb8767be time=60.420553ms + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact: server-id=server-c728053a-382d-a4a2-d470-cf8beb8767be time=60.420553ms +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact quorum of nodes, stepping down + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering follower state: follower="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Follower]" leader= + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.389Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering follower state: follower="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Follower]" leader= +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.391Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.391Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=c728053a-382d-a4a2-d470-cf8beb8767be error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.401Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.401Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to heartbeat to: peer=58633671-e0f2-ffc4-96f8-80811db6bec1 error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.406Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.406Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.408Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.408Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to appendEntries to: peer="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.433Z [WARN] server-58633671-e0f2-ffc4-96f8-80811db6bec1: heartbeat timeout reached, starting election: last-leader=c880f81a-325f-f987-5534-6efe79a4aa0d + testing.go:149: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.433Z [WARN] server-58633671-e0f2-ffc4-96f8-80811db6bec1: heartbeat timeout reached, starting election: last-leader=c880f81a-325f-f987-5534-6efe79a4aa0d +testLoggerAdapter verbose: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.433Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: entering candidate state: node="Node at 58633671-e0f2-ffc4-96f8-80811db6bec1 [Candidate]" term=3 + testing.go:149: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.433Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: entering candidate state: node="Node at 58633671-e0f2-ffc4-96f8-80811db6bec1 [Candidate]" term=3 +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.433Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.433Z [ERROR] server-58633671-e0f2-ffc4-96f8-80811db6bec1: failed to make requestVote RPC: target="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.433Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: duplicate requestVote for same term: term=3 + testing.go:149: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.433Z [ERROR] server-58633671-e0f2-ffc4-96f8-80811db6bec1: failed to make requestVote RPC: target="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.476Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: Election timeout reached, restarting election + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.476Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: Election timeout reached, restarting election +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering candidate state: node="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Candidate]" term=4 + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering candidate state: node="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Candidate]" term=4 +testLoggerAdapter verbose: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.477Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: entering follower state: follower="Node at 58633671-e0f2-ffc4-96f8-80811db6bec1 [Follower]" leader= + testing.go:149: server-58633671-e0f2-ffc4-96f8-80811db6bec1: 2021-08-25T09:31:01.477Z [INFO] server-58633671-e0f2-ffc4-96f8-80811db6bec1: entering follower state: follower="Node at 58633671-e0f2-ffc4-96f8-80811db6bec1 [Follower]" leader= +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to make requestVote RPC: target="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to make requestVote RPC: target="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: election won: tally=2 + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: election won: tally=2 +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering leader state: leader="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Leader]" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: entering leader state: leader="Node at c728053a-382d-a4a2-d470-cf8beb8767be [Leader]" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: added peer, starting replication: peer=server-c880f81a-325f-f987-5534-6efe79a4aa0d + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: added peer, starting replication: peer=server-c880f81a-325f-f987-5534-6efe79a4aa0d +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: added peer, starting replication: peer=server-58633671-e0f2-ffc4-96f8-80811db6bec1 + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: added peer, starting replication: peer=server-58633671-e0f2-ffc4-96f8-80811db6bec1 +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: pipelining replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.477Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: pipelining replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.482Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.482Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.483Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering candidate state: node="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Candidate]" term=3 + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.483Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering candidate state: node="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Candidate]" term=3 +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.483Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.483Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.483Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.483Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.486Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.486Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.494Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.494Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.506Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.506Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.512Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.512Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.525Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.525Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.528Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to contact: server-id=server-c880f81a-325f-f987-5534-6efe79a4aa0d time=50.290407ms + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.528Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to contact: server-id=server-c880f81a-325f-f987-5534-6efe79a4aa0d time=50.290407ms +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.541Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.541Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to appendEntries to: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.553Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.553Z [ERROR] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to heartbeat to: peer=c880f81a-325f-f987-5534-6efe79a4aa0d error="failed to connect to peer: c880f81a-325f-f987-5534-6efe79a4aa0d" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: Election timeout reached, restarting election + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [WARN] server-c880f81a-325f-f987-5534-6efe79a4aa0d: Election timeout reached, restarting election +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering candidate state: node="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Candidate]" term=4 + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering candidate state: node="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Candidate]" term=4 +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-c728053a-382d-a4a2-d470-cf8beb8767be c728053a-382d-a4a2-d470-cf8beb8767be}" error="failed to connect to peer: c728053a-382d-a4a2-d470-cf8beb8767be" +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.560Z [ERROR] server-c880f81a-325f-f987-5534-6efe79a4aa0d: failed to make requestVote RPC: target="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" error="failed to connect to peer: 58633671-e0f2-ffc4-96f8-80811db6bec1" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.577Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to contact: server-id=server-c880f81a-325f-f987-5534-6efe79a4aa0d time=100.123976ms + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.577Z [WARN] server-c728053a-382d-a4a2-d470-cf8beb8767be: failed to contact: server-id=server-c880f81a-325f-f987-5534-6efe79a4aa0d time=100.123976ms +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.588Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.483154463 +0000 UTC m=+5.749370887 (1 nodes), 148.023088ms from start of poll, 439.768609ms from cluster start. Timeout at 2021-08-25 09:31:01.588328404 +0000 UTC m=+5.854544888, 105.174001ms after stability + testing.go:149: cluster: 2021-08-25T09:31:01.588Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.483154463 +0000 UTC m=+5.749370887 (1 nodes), 148.023088ms from start of poll, 439.768609ms from cluster start. Timeout at 2021-08-25 09:31:01.588328404 +0000 UTC m=+5.854544888, 105.174001ms after stability + raft_test.go:371: [INFO] Reconnecting Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Candidate] +testLoggerAdapter verbose: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.592Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering follower state: follower="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Follower]" leader=c728053a-382d-a4a2-d470-cf8beb8767be + testing.go:149: server-c880f81a-325f-f987-5534-6efe79a4aa0d: 2021-08-25T09:31:01.592Z [INFO] server-c880f81a-325f-f987-5534-6efe79a4aa0d: entering follower state: follower="Node at c880f81a-325f-f987-5534-6efe79a4aa0d [Follower]" leader=c728053a-382d-a4a2-d470-cf8beb8767be +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.592Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: pipelining replication: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.592Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: pipelining replication: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.597Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: aborting pipeline replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.597Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: aborting pipeline replication: peer="{Voter server-58633671-e0f2-ffc4-96f8-80811db6bec1 58633671-e0f2-ffc4-96f8-80811db6bec1}" +testLoggerAdapter verbose: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.597Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: aborting pipeline replication: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" + testing.go:149: server-c728053a-382d-a4a2-d470-cf8beb8767be: 2021-08-25T09:31:01.597Z [INFO] server-c728053a-382d-a4a2-d470-cf8beb8767be: aborting pipeline replication: peer="{Voter server-c880f81a-325f-f987-5534-6efe79a4aa0d c880f81a-325f-f987-5534-6efe79a4aa0d}" +--- PASS: TestRaft_LeaderFail (0.56s) === RUN TestRaft_BehindFollower -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.986Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.986Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.987Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering follower state: follower="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Follower]" leader= - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:05.987Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering follower state: follower="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Follower]" leader= -testLoggerAdapter verbose: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" - testing.go:149: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" -testLoggerAdapter verbose: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: entering follower state: follower="Node at 55ac66af-99a1-bfc9-072b-c3a2671898b1 [Follower]" leader= - testing.go:149: server-55ac66af-99a1-bfc9-072b-c3a2671898b1: 2021-08-25T09:28:05.987Z [INFO] server-55ac66af-99a1-bfc9-072b-c3a2671898b1: entering follower state: follower="Node at 55ac66af-99a1-bfc9-072b-c3a2671898b1 [Follower]" leader= -testLoggerAdapter verbose: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" - testing.go:149: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7f134ac-941b-8470-0852-b7f284e3abb4 Address:e7f134ac-941b-8470-0852-b7f284e3abb4} {Suffrage:Voter ID:server-55ac66af-99a1-bfc9-072b-c3a2671898b1 Address:55ac66af-99a1-bfc9-072b-c3a2671898b1} {Suffrage:Voter ID:server-126bbe33-9f7c-2831-55a6-a34e29d66dae Address:126bbe33-9f7c-2831-55a6-a34e29d66dae}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:05.987Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:05.987Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: entering follower state: follower="Node at 126bbe33-9f7c-2831-55a6-a34e29d66dae [Follower]" leader= - testing.go:149: server-126bbe33-9f7c-2831-55a6-a34e29d66dae: 2021-08-25T09:28:05.987Z [INFO] server-126bbe33-9f7c-2831-55a6-a34e29d66dae: entering follower state: follower="Node at 126bbe33-9f7c-2831-55a6-a34e29d66dae [Follower]" leader= -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [WARN] server-e7f134ac-941b-8470-0852-b7f284e3abb4: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [WARN] server-e7f134ac-941b-8470-0852-b7f284e3abb4: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering candidate state: node="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Candidate]" term=2 - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.041Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering candidate state: node="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Candidate]" term=2 -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: election won: tally=2 - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: election won: tally=2 -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering leader state: leader="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Leader]" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: entering leader state: leader="Node at e7f134ac-941b-8470-0852-b7f284e3abb4 [Leader]" -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-55ac66af-99a1-bfc9-072b-c3a2671898b1 - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-55ac66af-99a1-bfc9-072b-c3a2671898b1 -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-126bbe33-9f7c-2831-55a6-a34e29d66dae - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: added peer, starting replication: peer=server-126bbe33-9f7c-2831-55a6-a34e29d66dae -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.042Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.147Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.042558864 +0000 UTC m=+5.806774753 (1 nodes), 54.628322ms from start of poll, 55.872667ms from cluster start. Timeout at 2021-08-25 09:28:06.147702852 +0000 UTC m=+5.911918781, 105.144028ms after stability - testing.go:149: cluster: 2021-08-25T09:28:06.147Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.042558864 +0000 UTC m=+5.806774753 (1 nodes), 54.628322ms from start of poll, 55.872667ms from cluster start. Timeout at 2021-08-25 09:28:06.147702852 +0000 UTC m=+5.911918781, 105.144028ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.147Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:06.147Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.253Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.147986506 +0000 UTC m=+5.912202395 (2 nodes), 2.78µs from start of poll, 161.300309ms from cluster start. Timeout at 2021-08-25 09:28:06.253147772 +0000 UTC m=+6.017363761, 105.161366ms after stability - testing.go:149: cluster: 2021-08-25T09:28:06.253Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.147986506 +0000 UTC m=+5.912202395 (2 nodes), 2.78µs from start of poll, 161.300309ms from cluster start. Timeout at 2021-08-25 09:28:06.253147772 +0000 UTC m=+6.017363761, 105.161366ms after stability -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to pipeline appendEntries: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="append pipeline closed" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to pipeline appendEntries: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="append pipeline closed" -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to appendEntries to: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="failed to connect to peer: 55ac66af-99a1-bfc9-072b-c3a2671898b1" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.253Z [ERROR] server-e7f134ac-941b-8470-0852-b7f284e3abb4: failed to appendEntries to: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" error="failed to connect to peer: 55ac66af-99a1-bfc9-072b-c3a2671898b1" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.601Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a07c722-1d1e-b639-1c47-528b0a9a382d Address:4a07c722-1d1e-b639-1c47-528b0a9a382d} {Suffrage:Voter ID:server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 Address:e7880f18-7dcf-d42f-06e9-7ae18e1d1354} {Suffrage:Voter ID:server-a190ce1e-00f4-9803-b86b-4b844d5efd7c Address:a190ce1e-00f4-9803-b86b-4b844d5efd7c}]" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.601Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a07c722-1d1e-b639-1c47-528b0a9a382d Address:4a07c722-1d1e-b639-1c47-528b0a9a382d} {Suffrage:Voter ID:server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 Address:e7880f18-7dcf-d42f-06e9-7ae18e1d1354} {Suffrage:Voter ID:server-a190ce1e-00f4-9803-b86b-4b844d5efd7c Address:a190ce1e-00f4-9803-b86b-4b844d5efd7c}]" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.601Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: entering follower state: follower="Node at 4a07c722-1d1e-b639-1c47-528b0a9a382d [Follower]" leader= + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.601Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: entering follower state: follower="Node at 4a07c722-1d1e-b639-1c47-528b0a9a382d [Follower]" leader= +testLoggerAdapter verbose: server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: 2021-08-25T09:31:01.602Z [INFO] server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a07c722-1d1e-b639-1c47-528b0a9a382d Address:4a07c722-1d1e-b639-1c47-528b0a9a382d} {Suffrage:Voter ID:server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 Address:e7880f18-7dcf-d42f-06e9-7ae18e1d1354} {Suffrage:Voter ID:server-a190ce1e-00f4-9803-b86b-4b844d5efd7c Address:a190ce1e-00f4-9803-b86b-4b844d5efd7c}]" + testing.go:149: server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: 2021-08-25T09:31:01.602Z [INFO] server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a07c722-1d1e-b639-1c47-528b0a9a382d Address:4a07c722-1d1e-b639-1c47-528b0a9a382d} {Suffrage:Voter ID:server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 Address:e7880f18-7dcf-d42f-06e9-7ae18e1d1354} {Suffrage:Voter ID:server-a190ce1e-00f4-9803-b86b-4b844d5efd7c Address:a190ce1e-00f4-9803-b86b-4b844d5efd7c}]" +testLoggerAdapter verbose: server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: 2021-08-25T09:31:01.602Z [INFO] server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: entering follower state: follower="Node at e7880f18-7dcf-d42f-06e9-7ae18e1d1354 [Follower]" leader= + testing.go:149: server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: 2021-08-25T09:31:01.602Z [INFO] server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354: entering follower state: follower="Node at e7880f18-7dcf-d42f-06e9-7ae18e1d1354 [Follower]" leader= +testLoggerAdapter verbose: server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: 2021-08-25T09:31:01.602Z [INFO] server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a07c722-1d1e-b639-1c47-528b0a9a382d Address:4a07c722-1d1e-b639-1c47-528b0a9a382d} {Suffrage:Voter ID:server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 Address:e7880f18-7dcf-d42f-06e9-7ae18e1d1354} {Suffrage:Voter ID:server-a190ce1e-00f4-9803-b86b-4b844d5efd7c Address:a190ce1e-00f4-9803-b86b-4b844d5efd7c}]" + testing.go:149: server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: 2021-08-25T09:31:01.602Z [INFO] server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a07c722-1d1e-b639-1c47-528b0a9a382d Address:4a07c722-1d1e-b639-1c47-528b0a9a382d} {Suffrage:Voter ID:server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 Address:e7880f18-7dcf-d42f-06e9-7ae18e1d1354} {Suffrage:Voter ID:server-a190ce1e-00f4-9803-b86b-4b844d5efd7c Address:a190ce1e-00f4-9803-b86b-4b844d5efd7c}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.602Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:01.602Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: 2021-08-25T09:31:01.602Z [INFO] server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: entering follower state: follower="Node at a190ce1e-00f4-9803-b86b-4b844d5efd7c [Follower]" leader= + testing.go:149: server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: 2021-08-25T09:31:01.602Z [INFO] server-a190ce1e-00f4-9803-b86b-4b844d5efd7c: entering follower state: follower="Node at a190ce1e-00f4-9803-b86b-4b844d5efd7c [Follower]" leader= +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [WARN] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [WARN] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: entering candidate state: node="Node at 4a07c722-1d1e-b639-1c47-528b0a9a382d [Candidate]" term=2 + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: entering candidate state: node="Node at 4a07c722-1d1e-b639-1c47-528b0a9a382d [Candidate]" term=2 +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: election won: tally=2 + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: election won: tally=2 +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: entering leader state: leader="Node at 4a07c722-1d1e-b639-1c47-528b0a9a382d [Leader]" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: entering leader state: leader="Node at 4a07c722-1d1e-b639-1c47-528b0a9a382d [Leader]" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: added peer, starting replication: peer=server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: added peer, starting replication: peer=server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: added peer, starting replication: peer=server-a190ce1e-00f4-9803-b86b-4b844d5efd7c + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: added peer, starting replication: peer=server-a190ce1e-00f4-9803-b86b-4b844d5efd7c +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: pipelining replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.663Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: pipelining replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.664Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: pipelining replication: peer="{Voter server-a190ce1e-00f4-9803-b86b-4b844d5efd7c a190ce1e-00f4-9803-b86b-4b844d5efd7c}" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.664Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: pipelining replication: peer="{Voter server-a190ce1e-00f4-9803-b86b-4b844d5efd7c a190ce1e-00f4-9803-b86b-4b844d5efd7c}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.769Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.663715648 +0000 UTC m=+5.929932092 (1 nodes), 61.143725ms from start of poll, 62.492698ms from cluster start. Timeout at 2021-08-25 09:31:01.76899416 +0000 UTC m=+6.035210624, 105.278532ms after stability + testing.go:149: cluster: 2021-08-25T09:31:01.769Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.663715648 +0000 UTC m=+5.929932092 (1 nodes), 61.143725ms from start of poll, 62.492698ms from cluster start. Timeout at 2021-08-25 09:31:01.76899416 +0000 UTC m=+6.035210624, 105.278532ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.769Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:01.769Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.874Z [INFO] stable state for Follower reached at 2021-08-25 09:31:01.769317109 +0000 UTC m=+6.035533493 (2 nodes), 3.579µs from start of poll, 168.094099ms from cluster start. Timeout at 2021-08-25 09:31:01.874356003 +0000 UTC m=+6.140572407, 105.038914ms after stability + testing.go:149: cluster: 2021-08-25T09:31:01.874Z [INFO] stable state for Follower reached at 2021-08-25 09:31:01.769317109 +0000 UTC m=+6.035533493 (2 nodes), 3.579µs from start of poll, 168.094099ms from cluster start. Timeout at 2021-08-25 09:31:01.874356003 +0000 UTC m=+6.140572407, 105.038914ms after stability +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.874Z [ERROR] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: failed to pipeline appendEntries: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" error="append pipeline closed" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.874Z [ERROR] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: failed to pipeline appendEntries: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" error="append pipeline closed" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.874Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: aborting pipeline replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.874Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: aborting pipeline replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.874Z [ERROR] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: failed to appendEntries to: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" error="failed to connect to peer: e7880f18-7dcf-d42f-06e9-7ae18e1d1354" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.874Z [ERROR] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: failed to appendEntries to: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" error="failed to connect to peer: e7880f18-7dcf-d42f-06e9-7ae18e1d1354" raft_test.go:420: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.263Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.263Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: pipelining replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.277Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:06.277Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.382Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.277147002 +0000 UTC m=+6.041362891 (1 nodes), 2.16µs from start of poll, 290.460805ms from cluster start. Timeout at 2021-08-25 09:28:06.382257253 +0000 UTC m=+6.146473142, 105.110251ms after stability - testing.go:149: cluster: 2021-08-25T09:28:06.382Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.277147002 +0000 UTC m=+6.041362891 (1 nodes), 2.16µs from start of poll, 290.460805ms from cluster start. Timeout at 2021-08-25 09:28:06.382257253 +0000 UTC m=+6.146473142, 105.110251ms after stability -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-126bbe33-9f7c-2831-55a6-a34e29d66dae 126bbe33-9f7c-2831-55a6-a34e29d66dae}" -testLoggerAdapter verbose: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" - testing.go:149: server-e7f134ac-941b-8470-0852-b7f284e3abb4: 2021-08-25T09:28:06.382Z [INFO] server-e7f134ac-941b-8470-0852-b7f284e3abb4: aborting pipeline replication: peer="{Voter server-55ac66af-99a1-bfc9-072b-c3a2671898b1 55ac66af-99a1-bfc9-072b-c3a2671898b1}" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.885Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: pipelining replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.885Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: pipelining replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.894Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:01.894Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:01.999Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.894405956 +0000 UTC m=+6.160622440 (1 nodes), 15.459µs from start of poll, 293.183046ms from cluster start. Timeout at 2021-08-25 09:31:01.999533881 +0000 UTC m=+6.265750325, 105.127885ms after stability + testing.go:149: cluster: 2021-08-25T09:31:01.999Z [INFO] stable state for Leader reached at 2021-08-25 09:31:01.894405956 +0000 UTC m=+6.160622440 (1 nodes), 15.459µs from start of poll, 293.183046ms from cluster start. Timeout at 2021-08-25 09:31:01.999533881 +0000 UTC m=+6.265750325, 105.127885ms after stability +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.999Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: aborting pipeline replication: peer="{Voter server-a190ce1e-00f4-9803-b86b-4b844d5efd7c a190ce1e-00f4-9803-b86b-4b844d5efd7c}" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.999Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: aborting pipeline replication: peer="{Voter server-a190ce1e-00f4-9803-b86b-4b844d5efd7c a190ce1e-00f4-9803-b86b-4b844d5efd7c}" +testLoggerAdapter verbose: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.999Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: aborting pipeline replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" + testing.go:149: server-4a07c722-1d1e-b639-1c47-528b0a9a382d: 2021-08-25T09:31:01.999Z [INFO] server-4a07c722-1d1e-b639-1c47-528b0a9a382d: aborting pipeline replication: peer="{Voter server-e7880f18-7dcf-d42f-06e9-7ae18e1d1354 e7880f18-7dcf-d42f-06e9-7ae18e1d1354}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering follower state: follower="Node at 27343939-c938-1f03-3869-4d341d79d44b [Follower]" leader= - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.386Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering follower state: follower="Node at 27343939-c938-1f03-3869-4d341d79d44b [Follower]" leader= -testLoggerAdapter verbose: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" - testing.go:149: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" -testLoggerAdapter verbose: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: entering follower state: follower="Node at e962818f-1d0a-0cb1-423c-a122388f9ed5 [Follower]" leader= - testing.go:149: server-e962818f-1d0a-0cb1-423c-a122388f9ed5: 2021-08-25T09:28:06.386Z [INFO] server-e962818f-1d0a-0cb1-423c-a122388f9ed5: entering follower state: follower="Node at e962818f-1d0a-0cb1-423c-a122388f9ed5 [Follower]" leader= -testLoggerAdapter verbose: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" - testing.go:149: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27343939-c938-1f03-3869-4d341d79d44b Address:27343939-c938-1f03-3869-4d341d79d44b} {Suffrage:Voter ID:server-e962818f-1d0a-0cb1-423c-a122388f9ed5 Address:e962818f-1d0a-0cb1-423c-a122388f9ed5} {Suffrage:Voter ID:server-227f3f58-3e33-e7b6-2932-9026b2d38170 Address:227f3f58-3e33-e7b6-2932-9026b2d38170}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.387Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:06.387Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: entering follower state: follower="Node at 227f3f58-3e33-e7b6-2932-9026b2d38170 [Follower]" leader= - testing.go:149: server-227f3f58-3e33-e7b6-2932-9026b2d38170: 2021-08-25T09:28:06.387Z [INFO] server-227f3f58-3e33-e7b6-2932-9026b2d38170: entering follower state: follower="Node at 227f3f58-3e33-e7b6-2932-9026b2d38170 [Follower]" leader= -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [WARN] server-27343939-c938-1f03-3869-4d341d79d44b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [WARN] server-27343939-c938-1f03-3869-4d341d79d44b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering candidate state: node="Node at 27343939-c938-1f03-3869-4d341d79d44b [Candidate]" term=2 - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering candidate state: node="Node at 27343939-c938-1f03-3869-4d341d79d44b [Candidate]" term=2 -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: election won: tally=2 - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: election won: tally=2 -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering leader state: leader="Node at 27343939-c938-1f03-3869-4d341d79d44b [Leader]" - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: entering leader state: leader="Node at 27343939-c938-1f03-3869-4d341d79d44b [Leader]" -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-e962818f-1d0a-0cb1-423c-a122388f9ed5 - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.439Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-e962818f-1d0a-0cb1-423c-a122388f9ed5 -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-227f3f58-3e33-e7b6-2932-9026b2d38170 - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: added peer, starting replication: peer=server-227f3f58-3e33-e7b6-2932-9026b2d38170 -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.440Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: pipelining replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.545Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.440482781 +0000 UTC m=+6.204698710 (1 nodes), 53.282368ms from start of poll, 54.487336ms from cluster start. Timeout at 2021-08-25 09:28:06.545491182 +0000 UTC m=+6.309707131, 105.008421ms after stability - testing.go:149: cluster: 2021-08-25T09:28:06.545Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.440482781 +0000 UTC m=+6.204698710 (1 nodes), 53.282368ms from start of poll, 54.487336ms from cluster start. Timeout at 2021-08-25 09:28:06.545491182 +0000 UTC m=+6.309707131, 105.008421ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.545Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:06.545Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.651Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.545841669 +0000 UTC m=+6.310057538 (2 nodes), 3.22µs from start of poll, 159.846164ms from cluster start. Timeout at 2021-08-25 09:28:06.650938901 +0000 UTC m=+6.415154870, 105.097332ms after stability - testing.go:149: cluster: 2021-08-25T09:28:06.651Z [INFO] stable state for Follower reached at 2021-08-25 09:28:06.545841669 +0000 UTC m=+6.310057538 (2 nodes), 3.22µs from start of poll, 159.846164ms from cluster start. Timeout at 2021-08-25 09:28:06.650938901 +0000 UTC m=+6.415154870, 105.097332ms after stability -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-227f3f58-3e33-e7b6-2932-9026b2d38170 227f3f58-3e33-e7b6-2932-9026b2d38170}" -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.651Z [INFO] server-27343939-c938-1f03-3869-4d341d79d44b: aborting pipeline replication: peer="{Voter server-e962818f-1d0a-0cb1-423c-a122388f9ed5 e962818f-1d0a-0cb1-423c-a122388f9ed5}" -testLoggerAdapter verbose: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.701Z [ERROR] server-27343939-c938-1f03-3869-4d341d79d44b: failed to heartbeat to: peer=e962818f-1d0a-0cb1-423c-a122388f9ed5 error="command timed out" - testing.go:149: server-27343939-c938-1f03-3869-4d341d79d44b: 2021-08-25T09:28:06.701Z [ERROR] server-27343939-c938-1f03-3869-4d341d79d44b: failed to heartbeat to: peer=e962818f-1d0a-0cb1-423c-a122388f9ed5 error="command timed out" ---- PASS: TestRaft_ApplyNonLeader (0.32s) +testLoggerAdapter verbose: server-4bb98f98-907a-c904-eb90-f9587034a211: 2021-08-25T09:31:02.004Z [INFO] server-4bb98f98-907a-c904-eb90-f9587034a211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bb98f98-907a-c904-eb90-f9587034a211 Address:4bb98f98-907a-c904-eb90-f9587034a211} {Suffrage:Voter ID:server-625ecbd8-5b58-6828-6e89-ac680de13424 Address:625ecbd8-5b58-6828-6e89-ac680de13424} {Suffrage:Voter ID:server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 Address:4b1b7608-b90b-e9db-134a-6cbc2ea994d8}]" + testing.go:149: server-4bb98f98-907a-c904-eb90-f9587034a211: 2021-08-25T09:31:02.004Z [INFO] server-4bb98f98-907a-c904-eb90-f9587034a211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bb98f98-907a-c904-eb90-f9587034a211 Address:4bb98f98-907a-c904-eb90-f9587034a211} {Suffrage:Voter ID:server-625ecbd8-5b58-6828-6e89-ac680de13424 Address:625ecbd8-5b58-6828-6e89-ac680de13424} {Suffrage:Voter ID:server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 Address:4b1b7608-b90b-e9db-134a-6cbc2ea994d8}]" +testLoggerAdapter verbose: server-4bb98f98-907a-c904-eb90-f9587034a211: 2021-08-25T09:31:02.004Z [INFO] server-4bb98f98-907a-c904-eb90-f9587034a211: entering follower state: follower="Node at 4bb98f98-907a-c904-eb90-f9587034a211 [Follower]" leader= + testing.go:149: server-4bb98f98-907a-c904-eb90-f9587034a211: 2021-08-25T09:31:02.004Z [INFO] server-4bb98f98-907a-c904-eb90-f9587034a211: entering follower state: follower="Node at 4bb98f98-907a-c904-eb90-f9587034a211 [Follower]" leader= +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.004Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bb98f98-907a-c904-eb90-f9587034a211 Address:4bb98f98-907a-c904-eb90-f9587034a211} {Suffrage:Voter ID:server-625ecbd8-5b58-6828-6e89-ac680de13424 Address:625ecbd8-5b58-6828-6e89-ac680de13424} {Suffrage:Voter ID:server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 Address:4b1b7608-b90b-e9db-134a-6cbc2ea994d8}]" + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.004Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bb98f98-907a-c904-eb90-f9587034a211 Address:4bb98f98-907a-c904-eb90-f9587034a211} {Suffrage:Voter ID:server-625ecbd8-5b58-6828-6e89-ac680de13424 Address:625ecbd8-5b58-6828-6e89-ac680de13424} {Suffrage:Voter ID:server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 Address:4b1b7608-b90b-e9db-134a-6cbc2ea994d8}]" +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.005Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: entering follower state: follower="Node at 625ecbd8-5b58-6828-6e89-ac680de13424 [Follower]" leader= + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.005Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: entering follower state: follower="Node at 625ecbd8-5b58-6828-6e89-ac680de13424 [Follower]" leader= +testLoggerAdapter verbose: server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: 2021-08-25T09:31:02.005Z [INFO] server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bb98f98-907a-c904-eb90-f9587034a211 Address:4bb98f98-907a-c904-eb90-f9587034a211} {Suffrage:Voter ID:server-625ecbd8-5b58-6828-6e89-ac680de13424 Address:625ecbd8-5b58-6828-6e89-ac680de13424} {Suffrage:Voter ID:server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 Address:4b1b7608-b90b-e9db-134a-6cbc2ea994d8}]" + testing.go:149: server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: 2021-08-25T09:31:02.005Z [INFO] server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bb98f98-907a-c904-eb90-f9587034a211 Address:4bb98f98-907a-c904-eb90-f9587034a211} {Suffrage:Voter ID:server-625ecbd8-5b58-6828-6e89-ac680de13424 Address:625ecbd8-5b58-6828-6e89-ac680de13424} {Suffrage:Voter ID:server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 Address:4b1b7608-b90b-e9db-134a-6cbc2ea994d8}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.005Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:02.005Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: 2021-08-25T09:31:02.005Z [INFO] server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: entering follower state: follower="Node at 4b1b7608-b90b-e9db-134a-6cbc2ea994d8 [Follower]" leader= + testing.go:149: server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: 2021-08-25T09:31:02.005Z [INFO] server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8: entering follower state: follower="Node at 4b1b7608-b90b-e9db-134a-6cbc2ea994d8 [Follower]" leader= +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [WARN] server-625ecbd8-5b58-6828-6e89-ac680de13424: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [WARN] server-625ecbd8-5b58-6828-6e89-ac680de13424: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: entering candidate state: node="Node at 625ecbd8-5b58-6828-6e89-ac680de13424 [Candidate]" term=2 + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: entering candidate state: node="Node at 625ecbd8-5b58-6828-6e89-ac680de13424 [Candidate]" term=2 +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: election won: tally=2 + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: election won: tally=2 +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: entering leader state: leader="Node at 625ecbd8-5b58-6828-6e89-ac680de13424 [Leader]" + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: entering leader state: leader="Node at 625ecbd8-5b58-6828-6e89-ac680de13424 [Leader]" +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: added peer, starting replication: peer=server-4bb98f98-907a-c904-eb90-f9587034a211 + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: added peer, starting replication: peer=server-4bb98f98-907a-c904-eb90-f9587034a211 +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: added peer, starting replication: peer=server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.058Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: added peer, starting replication: peer=server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.059Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: pipelining replication: peer="{Voter server-4bb98f98-907a-c904-eb90-f9587034a211 4bb98f98-907a-c904-eb90-f9587034a211}" + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.059Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: pipelining replication: peer="{Voter server-4bb98f98-907a-c904-eb90-f9587034a211 4bb98f98-907a-c904-eb90-f9587034a211}" +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.059Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: pipelining replication: peer="{Voter server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 4b1b7608-b90b-e9db-134a-6cbc2ea994d8}" + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.059Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: pipelining replication: peer="{Voter server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 4b1b7608-b90b-e9db-134a-6cbc2ea994d8}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.164Z [INFO] stable state for Leader reached at 2021-08-25 09:31:02.058863137 +0000 UTC m=+6.325079541 (1 nodes), 53.340639ms from start of poll, 54.696412ms from cluster start. Timeout at 2021-08-25 09:31:02.163994942 +0000 UTC m=+6.430211426, 105.131885ms after stability + testing.go:149: cluster: 2021-08-25T09:31:02.164Z [INFO] stable state for Leader reached at 2021-08-25 09:31:02.058863137 +0000 UTC m=+6.325079541 (1 nodes), 53.340639ms from start of poll, 54.696412ms from cluster start. Timeout at 2021-08-25 09:31:02.163994942 +0000 UTC m=+6.430211426, 105.131885ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.164Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:02.164Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.269Z [INFO] stable state for Follower reached at 2021-08-25 09:31:02.164266137 +0000 UTC m=+6.430482541 (2 nodes), 3.06µs from start of poll, 160.099412ms from cluster start. Timeout at 2021-08-25 09:31:02.269285412 +0000 UTC m=+6.535501896, 105.019355ms after stability + testing.go:149: cluster: 2021-08-25T09:31:02.269Z [INFO] stable state for Follower reached at 2021-08-25 09:31:02.164266137 +0000 UTC m=+6.430482541 (2 nodes), 3.06µs from start of poll, 160.099412ms from cluster start. Timeout at 2021-08-25 09:31:02.269285412 +0000 UTC m=+6.535501896, 105.019355ms after stability +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.269Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: aborting pipeline replication: peer="{Voter server-4bb98f98-907a-c904-eb90-f9587034a211 4bb98f98-907a-c904-eb90-f9587034a211}" + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.269Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: aborting pipeline replication: peer="{Voter server-4bb98f98-907a-c904-eb90-f9587034a211 4bb98f98-907a-c904-eb90-f9587034a211}" +testLoggerAdapter verbose: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.269Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: aborting pipeline replication: peer="{Voter server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 4b1b7608-b90b-e9db-134a-6cbc2ea994d8}" + testing.go:149: server-625ecbd8-5b58-6828-6e89-ac680de13424: 2021-08-25T09:31:02.269Z [INFO] server-625ecbd8-5b58-6828-6e89-ac680de13424: aborting pipeline replication: peer="{Voter server-4b1b7608-b90b-e9db-134a-6cbc2ea994d8 4b1b7608-b90b-e9db-134a-6cbc2ea994d8}" +--- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.705Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.705Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.706Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering follower state: follower="Node at b13f6cde-a725-f012-0112-f10901efd360 [Follower]" leader= - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.706Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering follower state: follower="Node at b13f6cde-a725-f012-0112-f10901efd360 [Follower]" leader= -testLoggerAdapter verbose: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" - testing.go:149: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" -testLoggerAdapter verbose: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: entering follower state: follower="Node at 7a1edcec-5dc4-5b3f-b39d-9f11a877f904 [Follower]" leader= - testing.go:149: server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: 2021-08-25T09:28:06.706Z [INFO] server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904: entering follower state: follower="Node at 7a1edcec-5dc4-5b3f-b39d-9f11a877f904 [Follower]" leader= -testLoggerAdapter verbose: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" - testing.go:149: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b13f6cde-a725-f012-0112-f10901efd360 Address:b13f6cde-a725-f012-0112-f10901efd360} {Suffrage:Voter ID:server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 Address:7a1edcec-5dc4-5b3f-b39d-9f11a877f904} {Suffrage:Voter ID:server-bd21579c-c57b-59de-76c4-b761e79393ee Address:bd21579c-c57b-59de-76c4-b761e79393ee}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:06.706Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:06.706Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: entering follower state: follower="Node at bd21579c-c57b-59de-76c4-b761e79393ee [Follower]" leader= - testing.go:149: server-bd21579c-c57b-59de-76c4-b761e79393ee: 2021-08-25T09:28:06.706Z [INFO] server-bd21579c-c57b-59de-76c4-b761e79393ee: entering follower state: follower="Node at bd21579c-c57b-59de-76c4-b761e79393ee [Follower]" leader= -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [WARN] server-b13f6cde-a725-f012-0112-f10901efd360: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [WARN] server-b13f6cde-a725-f012-0112-f10901efd360: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering candidate state: node="Node at b13f6cde-a725-f012-0112-f10901efd360 [Candidate]" term=2 - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering candidate state: node="Node at b13f6cde-a725-f012-0112-f10901efd360 [Candidate]" term=2 -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: election won: tally=2 - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: election won: tally=2 -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering leader state: leader="Node at b13f6cde-a725-f012-0112-f10901efd360 [Leader]" - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: entering leader state: leader="Node at b13f6cde-a725-f012-0112-f10901efd360 [Leader]" -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-bd21579c-c57b-59de-76c4-b761e79393ee - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.817Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: added peer, starting replication: peer=server-bd21579c-c57b-59de-76c4-b761e79393ee -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:06.818Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: pipelining replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.023Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.817929921 +0000 UTC m=+6.582145830 (1 nodes), 111.049159ms from start of poll, 112.305662ms from cluster start. Timeout at 2021-08-25 09:28:07.023068355 +0000 UTC m=+6.787284264, 205.138434ms after stability - testing.go:149: cluster: 2021-08-25T09:28:07.023Z [INFO] stable state for Leader reached at 2021-08-25 09:28:06.817929921 +0000 UTC m=+6.582145830 (1 nodes), 111.049159ms from start of poll, 112.305662ms from cluster start. Timeout at 2021-08-25 09:28:07.023068355 +0000 UTC m=+6.787284264, 205.138434ms after stability -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-bd21579c-c57b-59de-76c4-b761e79393ee bd21579c-c57b-59de-76c4-b761e79393ee}" -testLoggerAdapter verbose: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" - testing.go:149: server-b13f6cde-a725-f012-0112-f10901efd360: 2021-08-25T09:28:07.035Z [INFO] server-b13f6cde-a725-f012-0112-f10901efd360: aborting pipeline replication: peer="{Voter server-7a1edcec-5dc4-5b3f-b39d-9f11a877f904 7a1edcec-5dc4-5b3f-b39d-9f11a877f904}" ---- PASS: TestRaft_ApplyConcurrent (0.33s) +testLoggerAdapter verbose: server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: 2021-08-25T09:31:02.274Z [INFO] server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0af17e88-12bf-32ef-b543-a3c70f4aa29d Address:0af17e88-12bf-32ef-b543-a3c70f4aa29d} {Suffrage:Voter ID:server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 Address:c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52} {Suffrage:Voter ID:server-ca44bb78-0870-8913-45b0-9e9a2a417d49 Address:ca44bb78-0870-8913-45b0-9e9a2a417d49}]" + testing.go:149: server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: 2021-08-25T09:31:02.274Z [INFO] server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0af17e88-12bf-32ef-b543-a3c70f4aa29d Address:0af17e88-12bf-32ef-b543-a3c70f4aa29d} {Suffrage:Voter ID:server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 Address:c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52} {Suffrage:Voter ID:server-ca44bb78-0870-8913-45b0-9e9a2a417d49 Address:ca44bb78-0870-8913-45b0-9e9a2a417d49}]" +testLoggerAdapter verbose: server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: 2021-08-25T09:31:02.274Z [INFO] server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: entering follower state: follower="Node at 0af17e88-12bf-32ef-b543-a3c70f4aa29d [Follower]" leader= + testing.go:149: server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: 2021-08-25T09:31:02.274Z [INFO] server-0af17e88-12bf-32ef-b543-a3c70f4aa29d: entering follower state: follower="Node at 0af17e88-12bf-32ef-b543-a3c70f4aa29d [Follower]" leader= +testLoggerAdapter verbose: server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: 2021-08-25T09:31:02.274Z [INFO] server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0af17e88-12bf-32ef-b543-a3c70f4aa29d Address:0af17e88-12bf-32ef-b543-a3c70f4aa29d} {Suffrage:Voter ID:server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 Address:c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52} {Suffrage:Voter ID:server-ca44bb78-0870-8913-45b0-9e9a2a417d49 Address:ca44bb78-0870-8913-45b0-9e9a2a417d49}]" + testing.go:149: server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: 2021-08-25T09:31:02.274Z [INFO] server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0af17e88-12bf-32ef-b543-a3c70f4aa29d Address:0af17e88-12bf-32ef-b543-a3c70f4aa29d} {Suffrage:Voter ID:server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 Address:c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52} {Suffrage:Voter ID:server-ca44bb78-0870-8913-45b0-9e9a2a417d49 Address:ca44bb78-0870-8913-45b0-9e9a2a417d49}]" +testLoggerAdapter verbose: server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: 2021-08-25T09:31:02.274Z [INFO] server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: entering follower state: follower="Node at c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 [Follower]" leader= + testing.go:149: server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: 2021-08-25T09:31:02.274Z [INFO] server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52: entering follower state: follower="Node at c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 [Follower]" leader= +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.274Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0af17e88-12bf-32ef-b543-a3c70f4aa29d Address:0af17e88-12bf-32ef-b543-a3c70f4aa29d} {Suffrage:Voter ID:server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 Address:c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52} {Suffrage:Voter ID:server-ca44bb78-0870-8913-45b0-9e9a2a417d49 Address:ca44bb78-0870-8913-45b0-9e9a2a417d49}]" + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.274Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0af17e88-12bf-32ef-b543-a3c70f4aa29d Address:0af17e88-12bf-32ef-b543-a3c70f4aa29d} {Suffrage:Voter ID:server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 Address:c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52} {Suffrage:Voter ID:server-ca44bb78-0870-8913-45b0-9e9a2a417d49 Address:ca44bb78-0870-8913-45b0-9e9a2a417d49}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.275Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:02.275Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.275Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: entering follower state: follower="Node at ca44bb78-0870-8913-45b0-9e9a2a417d49 [Follower]" leader= + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.275Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: entering follower state: follower="Node at ca44bb78-0870-8913-45b0-9e9a2a417d49 [Follower]" leader= +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [WARN] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [WARN] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: entering candidate state: node="Node at ca44bb78-0870-8913-45b0-9e9a2a417d49 [Candidate]" term=2 + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: entering candidate state: node="Node at ca44bb78-0870-8913-45b0-9e9a2a417d49 [Candidate]" term=2 +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: election won: tally=2 + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: election won: tally=2 +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: entering leader state: leader="Node at ca44bb78-0870-8913-45b0-9e9a2a417d49 [Leader]" + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: entering leader state: leader="Node at ca44bb78-0870-8913-45b0-9e9a2a417d49 [Leader]" +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: added peer, starting replication: peer=server-0af17e88-12bf-32ef-b543-a3c70f4aa29d + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: added peer, starting replication: peer=server-0af17e88-12bf-32ef-b543-a3c70f4aa29d +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: added peer, starting replication: peer=server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: added peer, starting replication: peer=server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: pipelining replication: peer="{Voter server-0af17e88-12bf-32ef-b543-a3c70f4aa29d 0af17e88-12bf-32ef-b543-a3c70f4aa29d}" + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: pipelining replication: peer="{Voter server-0af17e88-12bf-32ef-b543-a3c70f4aa29d 0af17e88-12bf-32ef-b543-a3c70f4aa29d}" +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: pipelining replication: peer="{Voter server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52}" + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.390Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: pipelining replication: peer="{Voter server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.596Z [INFO] stable state for Leader reached at 2021-08-25 09:31:02.390785697 +0000 UTC m=+6.657002121 (1 nodes), 115.554064ms from start of poll, 116.988909ms from cluster start. Timeout at 2021-08-25 09:31:02.595973365 +0000 UTC m=+6.862189809, 205.187688ms after stability + testing.go:149: cluster: 2021-08-25T09:31:02.596Z [INFO] stable state for Leader reached at 2021-08-25 09:31:02.390785697 +0000 UTC m=+6.657002121 (1 nodes), 115.554064ms from start of poll, 116.988909ms from cluster start. Timeout at 2021-08-25 09:31:02.595973365 +0000 UTC m=+6.862189809, 205.187688ms after stability +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.608Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: aborting pipeline replication: peer="{Voter server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52}" + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.608Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: aborting pipeline replication: peer="{Voter server-c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52 c65d1fb8-e8ef-f4a8-2a1d-9de701e1af52}" +testLoggerAdapter verbose: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.608Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: aborting pipeline replication: peer="{Voter server-0af17e88-12bf-32ef-b543-a3c70f4aa29d 0af17e88-12bf-32ef-b543-a3c70f4aa29d}" + testing.go:149: server-ca44bb78-0870-8913-45b0-9e9a2a417d49: 2021-08-25T09:31:02.608Z [INFO] server-ca44bb78-0870-8913-45b0-9e9a2a417d49: aborting pipeline replication: peer="{Voter server-0af17e88-12bf-32ef-b543-a3c70f4aa29d 0af17e88-12bf-32ef-b543-a3c70f4aa29d}" +--- PASS: TestRaft_ApplyConcurrent (0.34s) === RUN TestRaft_ApplyConcurrent_Timeout -testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64230c52-0049-cf61-f6b3-03d525a099bd Address:64230c52-0049-cf61-f6b3-03d525a099bd}]" - testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64230c52-0049-cf61-f6b3-03d525a099bd Address:64230c52-0049-cf61-f6b3-03d525a099bd}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.038Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:07.038Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering follower state: follower="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Follower]" leader= - testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.038Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering follower state: follower="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Follower]" leader= -testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [WARN] server-64230c52-0049-cf61-f6b3-03d525a099bd: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [WARN] server-64230c52-0049-cf61-f6b3-03d525a099bd: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering candidate state: node="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Candidate]" term=2 - testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering candidate state: node="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Candidate]" term=2 -testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: election won: tally=1 - testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: election won: tally=1 -testLoggerAdapter verbose: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering leader state: leader="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Leader]" - testing.go:149: server-64230c52-0049-cf61-f6b3-03d525a099bd: 2021-08-25T09:28:07.230Z [INFO] server-64230c52-0049-cf61-f6b3-03d525a099bd: entering leader state: leader="Node at 64230c52-0049-cf61-f6b3-03d525a099bd [Leader]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.431Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.230685219 +0000 UTC m=+6.994901128 (1 nodes), 191.980557ms from start of poll, 192.369661ms from cluster start. Timeout at 2021-08-25 09:28:07.431798048 +0000 UTC m=+7.196014017, 201.112889ms after stability - testing.go:149: cluster: 2021-08-25T09:28:07.431Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.230685219 +0000 UTC m=+6.994901128 (1 nodes), 191.980557ms from start of poll, 192.369661ms from cluster start. Timeout at 2021-08-25 09:28:07.431798048 +0000 UTC m=+7.196014017, 201.112889ms after stability ---- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s) +testLoggerAdapter verbose: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.611Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26e42226-8c40-0211-14dd-2e25486586eb Address:26e42226-8c40-0211-14dd-2e25486586eb}]" + testing.go:149: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.611Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26e42226-8c40-0211-14dd-2e25486586eb Address:26e42226-8c40-0211-14dd-2e25486586eb}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.611Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:02.611Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.611Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: entering follower state: follower="Node at 26e42226-8c40-0211-14dd-2e25486586eb [Follower]" leader= + testing.go:149: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.611Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: entering follower state: follower="Node at 26e42226-8c40-0211-14dd-2e25486586eb [Follower]" leader= +testLoggerAdapter verbose: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.786Z [WARN] server-26e42226-8c40-0211-14dd-2e25486586eb: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.786Z [WARN] server-26e42226-8c40-0211-14dd-2e25486586eb: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.786Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: entering candidate state: node="Node at 26e42226-8c40-0211-14dd-2e25486586eb [Candidate]" term=2 + testing.go:149: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.786Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: entering candidate state: node="Node at 26e42226-8c40-0211-14dd-2e25486586eb [Candidate]" term=2 +testLoggerAdapter verbose: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.786Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: election won: tally=1 + testing.go:149: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.786Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: election won: tally=1 +testLoggerAdapter verbose: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.787Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: entering leader state: leader="Node at 26e42226-8c40-0211-14dd-2e25486586eb [Leader]" + testing.go:149: server-26e42226-8c40-0211-14dd-2e25486586eb: 2021-08-25T09:31:02.787Z [INFO] server-26e42226-8c40-0211-14dd-2e25486586eb: entering leader state: leader="Node at 26e42226-8c40-0211-14dd-2e25486586eb [Leader]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.988Z [INFO] stable state for Leader reached at 2021-08-25 09:31:02.787082398 +0000 UTC m=+7.053298822 (1 nodes), 175.446487ms from start of poll, 175.913983ms from cluster start. Timeout at 2021-08-25 09:31:02.988202548 +0000 UTC m=+7.254419012, 201.12019ms after stability + testing.go:149: cluster: 2021-08-25T09:31:02.988Z [INFO] stable state for Leader reached at 2021-08-25 09:31:02.787082398 +0000 UTC m=+7.053298822 (1 nodes), 175.446487ms from start of poll, 175.913983ms from cluster start. Timeout at 2021-08-25 09:31:02.988202548 +0000 UTC m=+7.254419012, 201.12019ms after stability +--- PASS: TestRaft_ApplyConcurrent_Timeout (0.38s) === RUN TestRaft_JoinNode -testLoggerAdapter verbose: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" - testing.go:149: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" -testLoggerAdapter verbose: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: entering follower state: follower="Node at 7123646f-1f46-adbf-b3d0-b3f4673ff95f [Follower]" leader= - testing.go:149: server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: 2021-08-25T09:28:07.434Z [INFO] server-7123646f-1f46-adbf-b3d0-b3f4673ff95f: entering follower state: follower="Node at 7123646f-1f46-adbf-b3d0-b3f4673ff95f [Follower]" leader= -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a}]" -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering follower state: follower="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Follower]" leader= - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.434Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering follower state: follower="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Follower]" leader= -testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: initial configuration: index=0 servers=[] - testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.435Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:07.435Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: entering follower state: follower="Node at 6f6c6b55-66a9-f17c-bb21-87481a40843f [Follower]" leader= - testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.435Z [INFO] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: entering follower state: follower="Node at 6f6c6b55-66a9-f17c-bb21-87481a40843f [Follower]" leader= -testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.488Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: no known peers, aborting election - testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.488Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: no known peers, aborting election -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering candidate state: node="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Candidate]" term=2 - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.491Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering candidate state: node="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Candidate]" term=2 -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: election won: tally=2 - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: election won: tally=2 -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering leader state: leader="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Leader]" - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: entering leader state: leader="Node at 4d015a90-64bd-4573-253e-3dbecc1e887a [Leader]" -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-7123646f-1f46-adbf-b3d0-b3f4673ff95f - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-7123646f-1f46-adbf-b3d0-b3f4673ff95f -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.492Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.597Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.492304164 +0000 UTC m=+7.256520073 (1 nodes), 56.676552ms from start of poll, 58.048465ms from cluster start. Timeout at 2021-08-25 09:28:07.597487889 +0000 UTC m=+7.361703798, 105.183725ms after stability - testing.go:149: cluster: 2021-08-25T09:28:07.597Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.492304164 +0000 UTC m=+7.256520073 (1 nodes), 56.676552ms from start of poll, 58.048465ms from cluster start. Timeout at 2021-08-25 09:28:07.597487889 +0000 UTC m=+7.361703798, 105.183725ms after stability -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: updating configuration: command=AddStaging server-id=server-6f6c6b55-66a9-f17c-bb21-87481a40843f server-addr=6f6c6b55-66a9-f17c-bb21-87481a40843f servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a} {Suffrage:Voter ID:server-6f6c6b55-66a9-f17c-bb21-87481a40843f Address:6f6c6b55-66a9-f17c-bb21-87481a40843f}]" - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: updating configuration: command=AddStaging server-id=server-6f6c6b55-66a9-f17c-bb21-87481a40843f server-addr=6f6c6b55-66a9-f17c-bb21-87481a40843f servers="[{Suffrage:Voter ID:server-7123646f-1f46-adbf-b3d0-b3f4673ff95f Address:7123646f-1f46-adbf-b3d0-b3f4673ff95f} {Suffrage:Voter ID:server-4d015a90-64bd-4573-253e-3dbecc1e887a Address:4d015a90-64bd-4573-253e-3dbecc1e887a} {Suffrage:Voter ID:server-6f6c6b55-66a9-f17c-bb21-87481a40843f Address:6f6c6b55-66a9-f17c-bb21-87481a40843f}]" -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-6f6c6b55-66a9-f17c-bb21-87481a40843f - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.597Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: added peer, starting replication: peer=server-6f6c6b55-66a9-f17c-bb21-87481a40843f -testLoggerAdapter verbose: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.598Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-6f6c6b55-66a9-f17c-bb21-87481a40843f: 2021-08-25T09:28:07.598Z [WARN] server-6f6c6b55-66a9-f17c-bb21-87481a40843f: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: appendEntries rejected, sending older logs: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" next=1 - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [WARN] server-4d015a90-64bd-4573-253e-3dbecc1e887a: appendEntries rejected, sending older logs: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" next=1 -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.598Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:07.598Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.598Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: pipelining replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.703Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.598341009 +0000 UTC m=+7.362556898 (1 nodes), 2.099µs from start of poll, 164.08529ms from cluster start. Timeout at 2021-08-25 09:28:07.703461939 +0000 UTC m=+7.467677848, 105.12095ms after stability - testing.go:149: cluster: 2021-08-25T09:28:07.703Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.598341009 +0000 UTC m=+7.362556898 (1 nodes), 2.099µs from start of poll, 164.08529ms from cluster start. Timeout at 2021-08-25 09:28:07.703461939 +0000 UTC m=+7.467677848, 105.12095ms after stability -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-7123646f-1f46-adbf-b3d0-b3f4673ff95f 7123646f-1f46-adbf-b3d0-b3f4673ff95f}" -testLoggerAdapter verbose: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" - testing.go:149: server-4d015a90-64bd-4573-253e-3dbecc1e887a: 2021-08-25T09:28:07.703Z [INFO] server-4d015a90-64bd-4573-253e-3dbecc1e887a: aborting pipeline replication: peer="{Voter server-6f6c6b55-66a9-f17c-bb21-87481a40843f 6f6c6b55-66a9-f17c-bb21-87481a40843f}" +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:02.991Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c983cac5-6e9a-1659-243f-896ef4285906 Address:c983cac5-6e9a-1659-243f-896ef4285906} {Suffrage:Voter ID:server-6dc2cf86-8bae-cc4e-8127-b5b754df438e Address:6dc2cf86-8bae-cc4e-8127-b5b754df438e}]" + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:02.991Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c983cac5-6e9a-1659-243f-896ef4285906 Address:c983cac5-6e9a-1659-243f-896ef4285906} {Suffrage:Voter ID:server-6dc2cf86-8bae-cc4e-8127-b5b754df438e Address:6dc2cf86-8bae-cc4e-8127-b5b754df438e}]" +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:02.991Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: entering follower state: follower="Node at c983cac5-6e9a-1659-243f-896ef4285906 [Follower]" leader= + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:02.991Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: entering follower state: follower="Node at c983cac5-6e9a-1659-243f-896ef4285906 [Follower]" leader= +testLoggerAdapter verbose: server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: 2021-08-25T09:31:02.991Z [INFO] server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c983cac5-6e9a-1659-243f-896ef4285906 Address:c983cac5-6e9a-1659-243f-896ef4285906} {Suffrage:Voter ID:server-6dc2cf86-8bae-cc4e-8127-b5b754df438e Address:6dc2cf86-8bae-cc4e-8127-b5b754df438e}]" + testing.go:149: server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: 2021-08-25T09:31:02.991Z [INFO] server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c983cac5-6e9a-1659-243f-896ef4285906 Address:c983cac5-6e9a-1659-243f-896ef4285906} {Suffrage:Voter ID:server-6dc2cf86-8bae-cc4e-8127-b5b754df438e Address:6dc2cf86-8bae-cc4e-8127-b5b754df438e}]" +testLoggerAdapter verbose: server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: 2021-08-25T09:31:02.991Z [INFO] server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: entering follower state: follower="Node at 6dc2cf86-8bae-cc4e-8127-b5b754df438e [Follower]" leader= + testing.go:149: server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: 2021-08-25T09:31:02.991Z [INFO] server-6dc2cf86-8bae-cc4e-8127-b5b754df438e: entering follower state: follower="Node at 6dc2cf86-8bae-cc4e-8127-b5b754df438e [Follower]" leader= +testLoggerAdapter verbose: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:02.992Z [INFO] server-5638b3ad-b602-3f2b-a735-5da284e5a868: initial configuration: index=0 servers=[] + testing.go:149: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:02.992Z [INFO] server-5638b3ad-b602-3f2b-a735-5da284e5a868: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:02.992Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:02.992Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:02.992Z [INFO] server-5638b3ad-b602-3f2b-a735-5da284e5a868: entering follower state: follower="Node at 5638b3ad-b602-3f2b-a735-5da284e5a868 [Follower]" leader= + testing.go:149: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:02.992Z [INFO] server-5638b3ad-b602-3f2b-a735-5da284e5a868: entering follower state: follower="Node at 5638b3ad-b602-3f2b-a735-5da284e5a868 [Follower]" leader= +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [WARN] server-c983cac5-6e9a-1659-243f-896ef4285906: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [WARN] server-c983cac5-6e9a-1659-243f-896ef4285906: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: entering candidate state: node="Node at c983cac5-6e9a-1659-243f-896ef4285906 [Candidate]" term=2 + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: entering candidate state: node="Node at c983cac5-6e9a-1659-243f-896ef4285906 [Candidate]" term=2 +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: election won: tally=2 + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: election won: tally=2 +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: entering leader state: leader="Node at c983cac5-6e9a-1659-243f-896ef4285906 [Leader]" + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: entering leader state: leader="Node at c983cac5-6e9a-1659-243f-896ef4285906 [Leader]" +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: added peer, starting replication: peer=server-6dc2cf86-8bae-cc4e-8127-b5b754df438e + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: added peer, starting replication: peer=server-6dc2cf86-8bae-cc4e-8127-b5b754df438e +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: pipelining replication: peer="{Voter server-6dc2cf86-8bae-cc4e-8127-b5b754df438e 6dc2cf86-8bae-cc4e-8127-b5b754df438e}" + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.050Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: pipelining replication: peer="{Voter server-6dc2cf86-8bae-cc4e-8127-b5b754df438e 6dc2cf86-8bae-cc4e-8127-b5b754df438e}" +testLoggerAdapter verbose: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:03.090Z [WARN] server-5638b3ad-b602-3f2b-a735-5da284e5a868: no known peers, aborting election + testing.go:149: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:03.090Z [WARN] server-5638b3ad-b602-3f2b-a735-5da284e5a868: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.155Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.050559339 +0000 UTC m=+7.316775783 (1 nodes), 57.925808ms from start of poll, 59.623728ms from cluster start. Timeout at 2021-08-25 09:31:03.155705643 +0000 UTC m=+7.421922087, 105.146304ms after stability + testing.go:149: cluster: 2021-08-25T09:31:03.155Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.050559339 +0000 UTC m=+7.316775783 (1 nodes), 57.925808ms from start of poll, 59.623728ms from cluster start. Timeout at 2021-08-25 09:31:03.155705643 +0000 UTC m=+7.421922087, 105.146304ms after stability +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.155Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: updating configuration: command=AddStaging server-id=server-5638b3ad-b602-3f2b-a735-5da284e5a868 server-addr=5638b3ad-b602-3f2b-a735-5da284e5a868 servers="[{Suffrage:Voter ID:server-c983cac5-6e9a-1659-243f-896ef4285906 Address:c983cac5-6e9a-1659-243f-896ef4285906} {Suffrage:Voter ID:server-6dc2cf86-8bae-cc4e-8127-b5b754df438e Address:6dc2cf86-8bae-cc4e-8127-b5b754df438e} {Suffrage:Voter ID:server-5638b3ad-b602-3f2b-a735-5da284e5a868 Address:5638b3ad-b602-3f2b-a735-5da284e5a868}]" + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.155Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: updating configuration: command=AddStaging server-id=server-5638b3ad-b602-3f2b-a735-5da284e5a868 server-addr=5638b3ad-b602-3f2b-a735-5da284e5a868 servers="[{Suffrage:Voter ID:server-c983cac5-6e9a-1659-243f-896ef4285906 Address:c983cac5-6e9a-1659-243f-896ef4285906} {Suffrage:Voter ID:server-6dc2cf86-8bae-cc4e-8127-b5b754df438e Address:6dc2cf86-8bae-cc4e-8127-b5b754df438e} {Suffrage:Voter ID:server-5638b3ad-b602-3f2b-a735-5da284e5a868 Address:5638b3ad-b602-3f2b-a735-5da284e5a868}]" +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.156Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: added peer, starting replication: peer=server-5638b3ad-b602-3f2b-a735-5da284e5a868 + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.156Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: added peer, starting replication: peer=server-5638b3ad-b602-3f2b-a735-5da284e5a868 +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.156Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:03.156Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:03.156Z [WARN] server-5638b3ad-b602-3f2b-a735-5da284e5a868: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-5638b3ad-b602-3f2b-a735-5da284e5a868: 2021-08-25T09:31:03.156Z [WARN] server-5638b3ad-b602-3f2b-a735-5da284e5a868: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.156Z [WARN] server-c983cac5-6e9a-1659-243f-896ef4285906: appendEntries rejected, sending older logs: peer="{Voter server-5638b3ad-b602-3f2b-a735-5da284e5a868 5638b3ad-b602-3f2b-a735-5da284e5a868}" next=1 + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.156Z [WARN] server-c983cac5-6e9a-1659-243f-896ef4285906: appendEntries rejected, sending older logs: peer="{Voter server-5638b3ad-b602-3f2b-a735-5da284e5a868 5638b3ad-b602-3f2b-a735-5da284e5a868}" next=1 +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.156Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: pipelining replication: peer="{Voter server-5638b3ad-b602-3f2b-a735-5da284e5a868 5638b3ad-b602-3f2b-a735-5da284e5a868}" + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.156Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: pipelining replication: peer="{Voter server-5638b3ad-b602-3f2b-a735-5da284e5a868 5638b3ad-b602-3f2b-a735-5da284e5a868}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.261Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.156431615 +0000 UTC m=+7.422648019 (1 nodes), 1.84µs from start of poll, 165.495964ms from cluster start. Timeout at 2021-08-25 09:31:03.261601696 +0000 UTC m=+7.527818160, 105.170141ms after stability + testing.go:149: cluster: 2021-08-25T09:31:03.261Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.156431615 +0000 UTC m=+7.422648019 (1 nodes), 1.84µs from start of poll, 165.495964ms from cluster start. Timeout at 2021-08-25 09:31:03.261601696 +0000 UTC m=+7.527818160, 105.170141ms after stability +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.261Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: aborting pipeline replication: peer="{Voter server-6dc2cf86-8bae-cc4e-8127-b5b754df438e 6dc2cf86-8bae-cc4e-8127-b5b754df438e}" + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.261Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: aborting pipeline replication: peer="{Voter server-6dc2cf86-8bae-cc4e-8127-b5b754df438e 6dc2cf86-8bae-cc4e-8127-b5b754df438e}" +testLoggerAdapter verbose: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.261Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: aborting pipeline replication: peer="{Voter server-5638b3ad-b602-3f2b-a735-5da284e5a868 5638b3ad-b602-3f2b-a735-5da284e5a868}" + testing.go:149: server-c983cac5-6e9a-1659-243f-896ef4285906: 2021-08-25T09:31:03.261Z [INFO] server-c983cac5-6e9a-1659-243f-896ef4285906: aborting pipeline replication: peer="{Voter server-5638b3ad-b602-3f2b-a735-5da284e5a868 5638b3ad-b602-3f2b-a735-5da284e5a868}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9}]" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9}]" -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering follower state: follower="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Follower]" leader= - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.706Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering follower state: follower="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Follower]" leader= -testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: initial configuration: index=0 servers=[] - testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: entering follower state: follower="Node at c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c [Follower]" leader= - testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.707Z [INFO] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: entering follower state: follower="Node at c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c [Follower]" leader= -testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: initial configuration: index=0 servers=[] - testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.707Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:07.707Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: entering follower state: follower="Node at 774530ec-2a41-28cd-d722-da8f2b52b59b [Follower]" leader= - testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.707Z [INFO] server-774530ec-2a41-28cd-d722-da8f2b52b59b: entering follower state: follower="Node at 774530ec-2a41-28cd-d722-da8f2b52b59b [Follower]" leader= -testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.787Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: no known peers, aborting election - testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:07.787Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: no known peers, aborting election -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.789Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.789Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering candidate state: node="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Candidate]" term=2 - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering candidate state: node="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Candidate]" term=2 -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: election won: tally=1 - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: election won: tally=1 -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering leader state: leader="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Leader]" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.790Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: entering leader state: leader="Node at 09f20c95-f8df-5131-f93f-3cd5e1bfcba9 [Leader]" -testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.795Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: no known peers, aborting election - testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.795Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.895Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.790453134 +0000 UTC m=+7.554669023 (1 nodes), 82.593403ms from start of poll, 84.289366ms from cluster start. Timeout at 2021-08-25 09:28:07.895609621 +0000 UTC m=+7.659825570, 105.156547ms after stability - testing.go:149: cluster: 2021-08-25T09:28:07.895Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.790453134 +0000 UTC m=+7.554669023 (1 nodes), 82.593403ms from start of poll, 84.289366ms from cluster start. Timeout at 2021-08-25 09:28:07.895609621 +0000 UTC m=+7.659825570, 105.156547ms after stability -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.895Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c server-addr=c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}]" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.895Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c server-addr=c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}]" -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:07.896Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:07.896Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.896Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: 2021-08-25T09:28:07.896Z [WARN] server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" next=1 - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" next=1 -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:07.896Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.001Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.896359911 +0000 UTC m=+7.660575800 (1 nodes), 1.859µs from start of poll, 190.196143ms from cluster start. Timeout at 2021-08-25 09:28:08.001450124 +0000 UTC m=+7.765666073, 105.090273ms after stability - testing.go:149: cluster: 2021-08-25T09:28:08.001Z [INFO] stable state for Leader reached at 2021-08-25 09:28:07.896359911 +0000 UTC m=+7.660575800 (1 nodes), 1.859µs from start of poll, 190.196143ms from cluster start. Timeout at 2021-08-25 09:28:08.001450124 +0000 UTC m=+7.765666073, 105.090273ms after stability -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-774530ec-2a41-28cd-d722-da8f2b52b59b server-addr=774530ec-2a41-28cd-d722-da8f2b52b59b servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c} {Suffrage:Voter ID:server-774530ec-2a41-28cd-d722-da8f2b52b59b Address:774530ec-2a41-28cd-d722-da8f2b52b59b}]" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: updating configuration: command=AddStaging server-id=server-774530ec-2a41-28cd-d722-da8f2b52b59b server-addr=774530ec-2a41-28cd-d722-da8f2b52b59b servers="[{Suffrage:Voter ID:server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9 Address:09f20c95-f8df-5131-f93f-3cd5e1bfcba9} {Suffrage:Voter ID:server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c Address:c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c} {Suffrage:Voter ID:server-774530ec-2a41-28cd-d722-da8f2b52b59b Address:774530ec-2a41-28cd-d722-da8f2b52b59b}]" -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-774530ec-2a41-28cd-d722-da8f2b52b59b - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.001Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: added peer, starting replication: peer=server-774530ec-2a41-28cd-d722-da8f2b52b59b -testLoggerAdapter verbose: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:08.002Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: failed to get previous log: previous-index=4 last-index=0 error="log not found" - testing.go:149: server-774530ec-2a41-28cd-d722-da8f2b52b59b: 2021-08-25T09:28:08.002Z [WARN] server-774530ec-2a41-28cd-d722-da8f2b52b59b: failed to get previous log: previous-index=4 last-index=0 error="log not found" -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" next=1 - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [WARN] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: appendEntries rejected, sending older logs: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" next=1 -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.002Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:08.002Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.002Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: pipelining replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.107Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.002346421 +0000 UTC m=+7.766562310 (1 nodes), 1.94µs from start of poll, 296.182653ms from cluster start. Timeout at 2021-08-25 09:28:08.107428395 +0000 UTC m=+7.871644304, 105.081994ms after stability - testing.go:149: cluster: 2021-08-25T09:28:08.107Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.002346421 +0000 UTC m=+7.766562310 (1 nodes), 1.94µs from start of poll, 296.182653ms from cluster start. Timeout at 2021-08-25 09:28:08.107428395 +0000 UTC m=+7.871644304, 105.081994ms after stability -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.107Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.107Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-774530ec-2a41-28cd-d722-da8f2b52b59b 774530ec-2a41-28cd-d722-da8f2b52b59b}" -testLoggerAdapter verbose: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.108Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" - testing.go:149: server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: 2021-08-25T09:28:08.108Z [INFO] server-09f20c95-f8df-5131-f93f-3cd5e1bfcba9: aborting pipeline replication: peer="{Voter server-c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c c98d5cd3-0f7b-5891-d82e-62d5be8bfd9c}" +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.264Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63c344e4-4e0d-987f-e671-609be6b63c40 Address:63c344e4-4e0d-987f-e671-609be6b63c40}]" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.264Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63c344e4-4e0d-987f-e671-609be6b63c40 Address:63c344e4-4e0d-987f-e671-609be6b63c40}]" +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.264Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: entering follower state: follower="Node at 63c344e4-4e0d-987f-e671-609be6b63c40 [Follower]" leader= + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.264Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: entering follower state: follower="Node at 63c344e4-4e0d-987f-e671-609be6b63c40 [Follower]" leader= +testLoggerAdapter verbose: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.265Z [INFO] server-77035e64-5963-ae15-507b-66131657cc44: initial configuration: index=0 servers=[] + testing.go:149: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.265Z [INFO] server-77035e64-5963-ae15-507b-66131657cc44: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.265Z [INFO] server-77035e64-5963-ae15-507b-66131657cc44: entering follower state: follower="Node at 77035e64-5963-ae15-507b-66131657cc44 [Follower]" leader= + testing.go:149: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.265Z [INFO] server-77035e64-5963-ae15-507b-66131657cc44: entering follower state: follower="Node at 77035e64-5963-ae15-507b-66131657cc44 [Follower]" leader= +testLoggerAdapter verbose: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.266Z [INFO] server-7da55b48-5800-9531-12b8-e97106fd93b0: initial configuration: index=0 servers=[] + testing.go:149: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.266Z [INFO] server-7da55b48-5800-9531-12b8-e97106fd93b0: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.267Z [INFO] server-7da55b48-5800-9531-12b8-e97106fd93b0: entering follower state: follower="Node at 7da55b48-5800-9531-12b8-e97106fd93b0 [Follower]" leader= + testing.go:149: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.267Z [INFO] server-7da55b48-5800-9531-12b8-e97106fd93b0: entering follower state: follower="Node at 7da55b48-5800-9531-12b8-e97106fd93b0 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.268Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:03.268Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.325Z [WARN] server-77035e64-5963-ae15-507b-66131657cc44: no known peers, aborting election + testing.go:149: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.325Z [WARN] server-77035e64-5963-ae15-507b-66131657cc44: no known peers, aborting election +testLoggerAdapter verbose: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.336Z [WARN] server-7da55b48-5800-9531-12b8-e97106fd93b0: no known peers, aborting election + testing.go:149: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.336Z [WARN] server-7da55b48-5800-9531-12b8-e97106fd93b0: no known peers, aborting election +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [WARN] server-63c344e4-4e0d-987f-e671-609be6b63c40: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [WARN] server-63c344e4-4e0d-987f-e671-609be6b63c40: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: entering candidate state: node="Node at 63c344e4-4e0d-987f-e671-609be6b63c40 [Candidate]" term=2 + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: entering candidate state: node="Node at 63c344e4-4e0d-987f-e671-609be6b63c40 [Candidate]" term=2 +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: election won: tally=1 + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: election won: tally=1 +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: entering leader state: leader="Node at 63c344e4-4e0d-987f-e671-609be6b63c40 [Leader]" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.358Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: entering leader state: leader="Node at 63c344e4-4e0d-987f-e671-609be6b63c40 [Leader]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.464Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.358806847 +0000 UTC m=+7.625023251 (1 nodes), 90.664306ms from start of poll, 94.3415ms from cluster start. Timeout at 2021-08-25 09:31:03.463946272 +0000 UTC m=+7.730162776, 105.139525ms after stability + testing.go:149: cluster: 2021-08-25T09:31:03.464Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.358806847 +0000 UTC m=+7.625023251 (1 nodes), 90.664306ms from start of poll, 94.3415ms from cluster start. Timeout at 2021-08-25 09:31:03.463946272 +0000 UTC m=+7.730162776, 105.139525ms after stability +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.464Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: updating configuration: command=AddStaging server-id=server-77035e64-5963-ae15-507b-66131657cc44 server-addr=77035e64-5963-ae15-507b-66131657cc44 servers="[{Suffrage:Voter ID:server-63c344e4-4e0d-987f-e671-609be6b63c40 Address:63c344e4-4e0d-987f-e671-609be6b63c40} {Suffrage:Voter ID:server-77035e64-5963-ae15-507b-66131657cc44 Address:77035e64-5963-ae15-507b-66131657cc44}]" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.464Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: updating configuration: command=AddStaging server-id=server-77035e64-5963-ae15-507b-66131657cc44 server-addr=77035e64-5963-ae15-507b-66131657cc44 servers="[{Suffrage:Voter ID:server-63c344e4-4e0d-987f-e671-609be6b63c40 Address:63c344e4-4e0d-987f-e671-609be6b63c40} {Suffrage:Voter ID:server-77035e64-5963-ae15-507b-66131657cc44 Address:77035e64-5963-ae15-507b-66131657cc44}]" +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.464Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: added peer, starting replication: peer=server-77035e64-5963-ae15-507b-66131657cc44 + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.464Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: added peer, starting replication: peer=server-77035e64-5963-ae15-507b-66131657cc44 +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.464Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.464Z [WARN] server-77035e64-5963-ae15-507b-66131657cc44: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: cluster: 2021-08-25T09:31:03.464Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-77035e64-5963-ae15-507b-66131657cc44: 2021-08-25T09:31:03.464Z [WARN] server-77035e64-5963-ae15-507b-66131657cc44: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.464Z [WARN] server-63c344e4-4e0d-987f-e671-609be6b63c40: appendEntries rejected, sending older logs: peer="{Voter server-77035e64-5963-ae15-507b-66131657cc44 77035e64-5963-ae15-507b-66131657cc44}" next=1 + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.464Z [WARN] server-63c344e4-4e0d-987f-e671-609be6b63c40: appendEntries rejected, sending older logs: peer="{Voter server-77035e64-5963-ae15-507b-66131657cc44 77035e64-5963-ae15-507b-66131657cc44}" next=1 +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.465Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: pipelining replication: peer="{Voter server-77035e64-5963-ae15-507b-66131657cc44 77035e64-5963-ae15-507b-66131657cc44}" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.465Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: pipelining replication: peer="{Voter server-77035e64-5963-ae15-507b-66131657cc44 77035e64-5963-ae15-507b-66131657cc44}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.570Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.464793772 +0000 UTC m=+7.731010176 (1 nodes), 1.98µs from start of poll, 200.328425ms from cluster start. Timeout at 2021-08-25 09:31:03.569904599 +0000 UTC m=+7.836121003, 105.110827ms after stability + testing.go:149: cluster: 2021-08-25T09:31:03.570Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.464793772 +0000 UTC m=+7.731010176 (1 nodes), 1.98µs from start of poll, 200.328425ms from cluster start. Timeout at 2021-08-25 09:31:03.569904599 +0000 UTC m=+7.836121003, 105.110827ms after stability +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.570Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: updating configuration: command=AddStaging server-id=server-7da55b48-5800-9531-12b8-e97106fd93b0 server-addr=7da55b48-5800-9531-12b8-e97106fd93b0 servers="[{Suffrage:Voter ID:server-63c344e4-4e0d-987f-e671-609be6b63c40 Address:63c344e4-4e0d-987f-e671-609be6b63c40} {Suffrage:Voter ID:server-77035e64-5963-ae15-507b-66131657cc44 Address:77035e64-5963-ae15-507b-66131657cc44} {Suffrage:Voter ID:server-7da55b48-5800-9531-12b8-e97106fd93b0 Address:7da55b48-5800-9531-12b8-e97106fd93b0}]" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.570Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: updating configuration: command=AddStaging server-id=server-7da55b48-5800-9531-12b8-e97106fd93b0 server-addr=7da55b48-5800-9531-12b8-e97106fd93b0 servers="[{Suffrage:Voter ID:server-63c344e4-4e0d-987f-e671-609be6b63c40 Address:63c344e4-4e0d-987f-e671-609be6b63c40} {Suffrage:Voter ID:server-77035e64-5963-ae15-507b-66131657cc44 Address:77035e64-5963-ae15-507b-66131657cc44} {Suffrage:Voter ID:server-7da55b48-5800-9531-12b8-e97106fd93b0 Address:7da55b48-5800-9531-12b8-e97106fd93b0}]" +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.570Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: added peer, starting replication: peer=server-7da55b48-5800-9531-12b8-e97106fd93b0 + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.570Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: added peer, starting replication: peer=server-7da55b48-5800-9531-12b8-e97106fd93b0 +testLoggerAdapter verbose: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.570Z [WARN] server-7da55b48-5800-9531-12b8-e97106fd93b0: failed to get previous log: previous-index=4 last-index=0 error="log not found" + testing.go:149: server-7da55b48-5800-9531-12b8-e97106fd93b0: 2021-08-25T09:31:03.570Z [WARN] server-7da55b48-5800-9531-12b8-e97106fd93b0: failed to get previous log: previous-index=4 last-index=0 error="log not found" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.570Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.570Z [WARN] server-63c344e4-4e0d-987f-e671-609be6b63c40: appendEntries rejected, sending older logs: peer="{Voter server-7da55b48-5800-9531-12b8-e97106fd93b0 7da55b48-5800-9531-12b8-e97106fd93b0}" next=1 + testing.go:149: cluster: 2021-08-25T09:31:03.570Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.570Z [WARN] server-63c344e4-4e0d-987f-e671-609be6b63c40: appendEntries rejected, sending older logs: peer="{Voter server-7da55b48-5800-9531-12b8-e97106fd93b0 7da55b48-5800-9531-12b8-e97106fd93b0}" next=1 +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.571Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: pipelining replication: peer="{Voter server-7da55b48-5800-9531-12b8-e97106fd93b0 7da55b48-5800-9531-12b8-e97106fd93b0}" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.571Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: pipelining replication: peer="{Voter server-7da55b48-5800-9531-12b8-e97106fd93b0 7da55b48-5800-9531-12b8-e97106fd93b0}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.676Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.570873908 +0000 UTC m=+7.837090312 (1 nodes), 2.02µs from start of poll, 306.408561ms from cluster start. Timeout at 2021-08-25 09:31:03.676022131 +0000 UTC m=+7.942238575, 105.148263ms after stability + testing.go:149: cluster: 2021-08-25T09:31:03.676Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.570873908 +0000 UTC m=+7.837090312 (1 nodes), 2.02µs from start of poll, 306.408561ms from cluster start. Timeout at 2021-08-25 09:31:03.676022131 +0000 UTC m=+7.942238575, 105.148263ms after stability +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.676Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: aborting pipeline replication: peer="{Voter server-7da55b48-5800-9531-12b8-e97106fd93b0 7da55b48-5800-9531-12b8-e97106fd93b0}" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.676Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: aborting pipeline replication: peer="{Voter server-7da55b48-5800-9531-12b8-e97106fd93b0 7da55b48-5800-9531-12b8-e97106fd93b0}" +testLoggerAdapter verbose: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.676Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: aborting pipeline replication: peer="{Voter server-77035e64-5963-ae15-507b-66131657cc44 77035e64-5963-ae15-507b-66131657cc44}" + testing.go:149: server-63c344e4-4e0d-987f-e671-609be6b63c40: 2021-08-25T09:31:03.676Z [INFO] server-63c344e4-4e0d-987f-e671-609be6b63c40: aborting pipeline replication: peer="{Voter server-77035e64-5963-ae15-507b-66131657cc44 77035e64-5963-ae15-507b-66131657cc44}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower -testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" - testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" -testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering follower state: follower="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Follower]" leader= - testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.113Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering follower state: follower="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Follower]" leader= -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering follower state: follower="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Follower]" leader= - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.113Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering follower state: follower="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Follower]" leader= -testLoggerAdapter verbose: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" - testing.go:149: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b89ba74-8aee-e236-8eee-b67204f5351a Address:9b89ba74-8aee-e236-8eee-b67204f5351a} {Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.114Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:08.114Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: entering follower state: follower="Node at be5e9eef-f4ac-cc10-3b82-83df05e62605 [Follower]" leader= - testing.go:149: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.114Z [INFO] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: entering follower state: follower="Node at be5e9eef-f4ac-cc10-3b82-83df05e62605 [Follower]" leader= -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering candidate state: node="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Candidate]" term=2 - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering candidate state: node="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Candidate]" term=2 -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: election won: tally=2 - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: election won: tally=2 -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering leader state: leader="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Leader]" - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: entering leader state: leader="Node at dd3282c0-b14d-5efa-0e74-3a06ac823338 [Leader]" -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-be5e9eef-f4ac-cc10-3b82-83df05e62605 - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: added peer, starting replication: peer=server-be5e9eef-f4ac-cc10-3b82-83df05e62605 -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.183Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.184Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.184Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: pipelining replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.288Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.183739862 +0000 UTC m=+7.947955771 (1 nodes), 69.447545ms from start of poll, 70.613077ms from cluster start. Timeout at 2021-08-25 09:28:08.288864912 +0000 UTC m=+8.053080841, 105.12507ms after stability - testing.go:149: cluster: 2021-08-25T09:28:08.288Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.183739862 +0000 UTC m=+7.947955771 (1 nodes), 69.447545ms from start of poll, 70.613077ms from cluster start. Timeout at 2021-08-25 09:28:08.288864912 +0000 UTC m=+8.053080841, 105.12507ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.294Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:08.294Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.399Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.294394998 +0000 UTC m=+8.058610907 (2 nodes), 2.98µs from start of poll, 181.268213ms from cluster start. Timeout at 2021-08-25 09:28:08.399516308 +0000 UTC m=+8.163732217, 105.12131ms after stability - testing.go:149: cluster: 2021-08-25T09:28:08.399Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.294394998 +0000 UTC m=+8.058610907 (2 nodes), 2.98µs from start of poll, 181.268213ms from cluster start. Timeout at 2021-08-25 09:28:08.399516308 +0000 UTC m=+8.163732217, 105.12131ms after stability -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.399Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: updating configuration: command=RemoveServer server-id=server-9b89ba74-8aee-e236-8eee-b67204f5351a server-addr= servers="[{Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.399Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: updating configuration: command=RemoveServer server-id=server-9b89ba74-8aee-e236-8eee-b67204f5351a server-addr= servers="[{Suffrage:Voter ID:server-dd3282c0-b14d-5efa-0e74-3a06ac823338 Address:dd3282c0-b14d-5efa-0e74-3a06ac823338} {Suffrage:Voter ID:server-be5e9eef-f4ac-cc10-3b82-83df05e62605 Address:be5e9eef-f4ac-cc10-3b82-83df05e62605}]" -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: removed peer, stopping replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a last-index=3 - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: removed peer, stopping replication: peer=server-9b89ba74-8aee-e236-8eee-b67204f5351a last-index=3 -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.400Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-9b89ba74-8aee-e236-8eee-b67204f5351a 9b89ba74-8aee-e236-8eee-b67204f5351a}" -testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [WARN] server-9b89ba74-8aee-e236-8eee-b67204f5351a: heartbeat timeout reached, starting election: last-leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 - testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [WARN] server-9b89ba74-8aee-e236-8eee-b67204f5351a: heartbeat timeout reached, starting election: last-leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 -testLoggerAdapter verbose: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering candidate state: node="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Candidate]" term=3 - testing.go:149: server-9b89ba74-8aee-e236-8eee-b67204f5351a: 2021-08-25T09:28:08.461Z [INFO] server-9b89ba74-8aee-e236-8eee-b67204f5351a: entering candidate state: node="Node at 9b89ba74-8aee-e236-8eee-b67204f5351a [Candidate]" term=3 -testLoggerAdapter verbose: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.461Z [WARN] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 - testing.go:149: server-be5e9eef-f4ac-cc10-3b82-83df05e62605: 2021-08-25T09:28:08.461Z [WARN] server-be5e9eef-f4ac-cc10-3b82-83df05e62605: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.461Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.461Z [WARN] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: rejecting vote request since we have a leader: from=9b89ba74-8aee-e236-8eee-b67204f5351a leader=dd3282c0-b14d-5efa-0e74-3a06ac823338 -testLoggerAdapter verbose: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.505Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" - testing.go:149: server-dd3282c0-b14d-5efa-0e74-3a06ac823338: 2021-08-25T09:28:08.505Z [INFO] server-dd3282c0-b14d-5efa-0e74-3a06ac823338: aborting pipeline replication: peer="{Voter server-be5e9eef-f4ac-cc10-3b82-83df05e62605 be5e9eef-f4ac-cc10-3b82-83df05e62605}" ---- PASS: TestRaft_RemoveFollower (0.40s) +testLoggerAdapter verbose: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:03.680Z [INFO] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 Address:1b839e59-bc0e-bf81-3aa2-7d9fe09243e8} {Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" + testing.go:149: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:03.680Z [INFO] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 Address:1b839e59-bc0e-bf81-3aa2-7d9fe09243e8} {Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" +testLoggerAdapter verbose: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:03.680Z [INFO] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: entering follower state: follower="Node at 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 [Follower]" leader= + testing.go:149: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:03.680Z [INFO] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: entering follower state: follower="Node at 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 [Follower]" leader= +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.680Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 Address:1b839e59-bc0e-bf81-3aa2-7d9fe09243e8} {Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.680Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 Address:1b839e59-bc0e-bf81-3aa2-7d9fe09243e8} {Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.680Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: entering follower state: follower="Node at f874226f-fe5a-d360-de96-f971dbb7204f [Follower]" leader= + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.680Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: entering follower state: follower="Node at f874226f-fe5a-d360-de96-f971dbb7204f [Follower]" leader= +testLoggerAdapter verbose: server-23d1332c-166c-a241-6eec-467a9e6a71df: 2021-08-25T09:31:03.680Z [INFO] server-23d1332c-166c-a241-6eec-467a9e6a71df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 Address:1b839e59-bc0e-bf81-3aa2-7d9fe09243e8} {Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" + testing.go:149: server-23d1332c-166c-a241-6eec-467a9e6a71df: 2021-08-25T09:31:03.680Z [INFO] server-23d1332c-166c-a241-6eec-467a9e6a71df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 Address:1b839e59-bc0e-bf81-3aa2-7d9fe09243e8} {Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.681Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:03.681Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-23d1332c-166c-a241-6eec-467a9e6a71df: 2021-08-25T09:31:03.681Z [INFO] server-23d1332c-166c-a241-6eec-467a9e6a71df: entering follower state: follower="Node at 23d1332c-166c-a241-6eec-467a9e6a71df [Follower]" leader= + testing.go:149: server-23d1332c-166c-a241-6eec-467a9e6a71df: 2021-08-25T09:31:03.681Z [INFO] server-23d1332c-166c-a241-6eec-467a9e6a71df: entering follower state: follower="Node at 23d1332c-166c-a241-6eec-467a9e6a71df [Follower]" leader= +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [WARN] server-f874226f-fe5a-d360-de96-f971dbb7204f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [WARN] server-f874226f-fe5a-d360-de96-f971dbb7204f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: entering candidate state: node="Node at f874226f-fe5a-d360-de96-f971dbb7204f [Candidate]" term=2 + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: entering candidate state: node="Node at f874226f-fe5a-d360-de96-f971dbb7204f [Candidate]" term=2 +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: election won: tally=2 + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: election won: tally=2 +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: entering leader state: leader="Node at f874226f-fe5a-d360-de96-f971dbb7204f [Leader]" + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: entering leader state: leader="Node at f874226f-fe5a-d360-de96-f971dbb7204f [Leader]" +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: added peer, starting replication: peer=server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: added peer, starting replication: peer=server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: added peer, starting replication: peer=server-23d1332c-166c-a241-6eec-467a9e6a71df + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.747Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: added peer, starting replication: peer=server-23d1332c-166c-a241-6eec-467a9e6a71df +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.748Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: pipelining replication: peer="{Voter server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8}" + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.748Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: pipelining replication: peer="{Voter server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8}" +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.748Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: pipelining replication: peer="{Voter server-23d1332c-166c-a241-6eec-467a9e6a71df 23d1332c-166c-a241-6eec-467a9e6a71df}" + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.748Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: pipelining replication: peer="{Voter server-23d1332c-166c-a241-6eec-467a9e6a71df 23d1332c-166c-a241-6eec-467a9e6a71df}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.852Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.747844432 +0000 UTC m=+8.014060836 (1 nodes), 66.738039ms from start of poll, 67.835496ms from cluster start. Timeout at 2021-08-25 09:31:03.852871027 +0000 UTC m=+8.119087471, 105.026635ms after stability + testing.go:149: cluster: 2021-08-25T09:31:03.852Z [INFO] stable state for Leader reached at 2021-08-25 09:31:03.747844432 +0000 UTC m=+8.014060836 (1 nodes), 66.738039ms from start of poll, 67.835496ms from cluster start. Timeout at 2021-08-25 09:31:03.852871027 +0000 UTC m=+8.119087471, 105.026635ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.858Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:03.858Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:03.963Z [INFO] stable state for Follower reached at 2021-08-25 09:31:03.858427804 +0000 UTC m=+8.124644188 (2 nodes), 2.7µs from start of poll, 178.418848ms from cluster start. Timeout at 2021-08-25 09:31:03.963558529 +0000 UTC m=+8.229775033, 105.130845ms after stability + testing.go:149: cluster: 2021-08-25T09:31:03.963Z [INFO] stable state for Follower reached at 2021-08-25 09:31:03.858427804 +0000 UTC m=+8.124644188 (2 nodes), 2.7µs from start of poll, 178.418848ms from cluster start. Timeout at 2021-08-25 09:31:03.963558529 +0000 UTC m=+8.229775033, 105.130845ms after stability +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.963Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: updating configuration: command=RemoveServer server-id=server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 server-addr= servers="[{Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.963Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: updating configuration: command=RemoveServer server-id=server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 server-addr= servers="[{Suffrage:Voter ID:server-f874226f-fe5a-d360-de96-f971dbb7204f Address:f874226f-fe5a-d360-de96-f971dbb7204f} {Suffrage:Voter ID:server-23d1332c-166c-a241-6eec-467a9e6a71df Address:23d1332c-166c-a241-6eec-467a9e6a71df}]" +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.964Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: removed peer, stopping replication: peer=server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 last-index=3 + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.964Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: removed peer, stopping replication: peer=server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 last-index=3 +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.964Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: aborting pipeline replication: peer="{Voter server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8}" + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:03.964Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: aborting pipeline replication: peer="{Voter server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8}" +testLoggerAdapter verbose: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:04.054Z [WARN] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: heartbeat timeout reached, starting election: last-leader=f874226f-fe5a-d360-de96-f971dbb7204f + testing.go:149: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:04.054Z [WARN] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: heartbeat timeout reached, starting election: last-leader=f874226f-fe5a-d360-de96-f971dbb7204f +testLoggerAdapter verbose: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:04.054Z [INFO] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: entering candidate state: node="Node at 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 [Candidate]" term=3 + testing.go:149: server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: 2021-08-25T09:31:04.054Z [INFO] server-1b839e59-bc0e-bf81-3aa2-7d9fe09243e8: entering candidate state: node="Node at 1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 [Candidate]" term=3 +testLoggerAdapter verbose: server-23d1332c-166c-a241-6eec-467a9e6a71df: 2021-08-25T09:31:04.054Z [WARN] server-23d1332c-166c-a241-6eec-467a9e6a71df: rejecting vote request since we have a leader: from=1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 leader=f874226f-fe5a-d360-de96-f971dbb7204f + testing.go:149: server-23d1332c-166c-a241-6eec-467a9e6a71df: 2021-08-25T09:31:04.054Z [WARN] server-23d1332c-166c-a241-6eec-467a9e6a71df: rejecting vote request since we have a leader: from=1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 leader=f874226f-fe5a-d360-de96-f971dbb7204f +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:04.054Z [WARN] server-f874226f-fe5a-d360-de96-f971dbb7204f: rejecting vote request since we have a leader: from=1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 leader=f874226f-fe5a-d360-de96-f971dbb7204f + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:04.054Z [WARN] server-f874226f-fe5a-d360-de96-f971dbb7204f: rejecting vote request since we have a leader: from=1b839e59-bc0e-bf81-3aa2-7d9fe09243e8 leader=f874226f-fe5a-d360-de96-f971dbb7204f +testLoggerAdapter verbose: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:04.069Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: aborting pipeline replication: peer="{Voter server-23d1332c-166c-a241-6eec-467a9e6a71df 23d1332c-166c-a241-6eec-467a9e6a71df}" + testing.go:149: server-f874226f-fe5a-d360-de96-f971dbb7204f: 2021-08-25T09:31:04.069Z [INFO] server-f874226f-fe5a-d360-de96-f971dbb7204f: aborting pipeline replication: peer="{Voter server-23d1332c-166c-a241-6eec-467a9e6a71df 23d1332c-166c-a241-6eec-467a9e6a71df}" +--- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader -testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.509Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" - testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.509Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" -testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.510Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= - testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.510Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering follower state: follower="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Follower]" leader= - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.510Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering follower state: follower="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Follower]" leader= -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-000e1006-437f-af1b-8db7-148ccf97d74b Address:000e1006-437f-af1b-8db7-148ccf97d74b} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.510Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:08.510Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering follower state: follower="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Follower]" leader= - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.510Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering follower state: follower="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Follower]" leader= -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [WARN] server-000e1006-437f-af1b-8db7-148ccf97d74b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [WARN] server-000e1006-437f-af1b-8db7-148ccf97d74b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering candidate state: node="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Candidate]" term=2 - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering candidate state: node="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Candidate]" term=2 -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: election won: tally=2 - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: election won: tally=2 -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering leader state: leader="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Leader]" - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: entering leader state: leader="Node at 000e1006-437f-af1b-8db7-148ccf97d74b [Leader]" -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.581Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: added peer, starting replication: peer=server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.582Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.687Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.582279522 +0000 UTC m=+8.346495411 (1 nodes), 71.405804ms from start of poll, 72.73966ms from cluster start. Timeout at 2021-08-25 09:28:08.687336438 +0000 UTC m=+8.451552407, 105.056996ms after stability - testing.go:149: cluster: 2021-08-25T09:28:08.687Z [INFO] stable state for Leader reached at 2021-08-25 09:28:08.582279522 +0000 UTC m=+8.346495411 (1 nodes), 71.405804ms from start of poll, 72.73966ms from cluster start. Timeout at 2021-08-25 09:28:08.687336438 +0000 UTC m=+8.451552407, 105.056996ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.692Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:08.692Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:08.798Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.692850705 +0000 UTC m=+8.457066614 (2 nodes), 2.619µs from start of poll, 183.310863ms from cluster start. Timeout at 2021-08-25 09:28:08.797965836 +0000 UTC m=+8.562181825, 105.115211ms after stability - testing.go:149: cluster: 2021-08-25T09:28:08.798Z [INFO] stable state for Follower reached at 2021-08-25 09:28:08.692850705 +0000 UTC m=+8.457066614 (2 nodes), 2.619µs from start of poll, 183.310863ms from cluster start. Timeout at 2021-08-25 09:28:08.797965836 +0000 UTC m=+8.562181825, 105.115211ms after stability -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: updating configuration: command=RemoveServer server-id=server-000e1006-437f-af1b-8db7-148ccf97d74b server-addr= servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: updating configuration: command=RemoveServer server-id=server-000e1006-437f-af1b-8db7-148ccf97d74b server-addr= servers="[{Suffrage:Voter ID:server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 Address:b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349} {Suffrage:Voter ID:server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 Address:051c4c20-115e-e3ed-8769-6c28e69fb7f3}]" -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: removed ourself, shutting down - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: removed ourself, shutting down -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-051c4c20-115e-e3ed-8769-6c28e69fb7f3 051c4c20-115e-e3ed-8769-6c28e69fb7f3}" -testLoggerAdapter verbose: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" - testing.go:149: server-000e1006-437f-af1b-8db7-148ccf97d74b: 2021-08-25T09:28:08.798Z [INFO] server-000e1006-437f-af1b-8db7-148ccf97d74b: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=3 - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.886Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=3 -testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.886Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: rejecting vote request since we have a leader: from=051c4c20-115e-e3ed-8769-6c28e69fb7f3 leader=000e1006-437f-af1b-8db7-148ccf97d74b - testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.886Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: rejecting vote request since we have a leader: from=051c4c20-115e-e3ed-8769-6c28e69fb7f3 leader=000e1006-437f-af1b-8db7-148ccf97d74b -testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b - testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [WARN] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: heartbeat timeout reached, starting election: last-leader=000e1006-437f-af1b-8db7-148ccf97d74b -testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering candidate state: node="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Candidate]" term=3 - testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.907Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering candidate state: node="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Candidate]" term=3 -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.908Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: duplicate requestVote for same term: term=3 - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.908Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: Election timeout reached, restarting election - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [WARN] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: Election timeout reached, restarting election -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=4 - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering candidate state: node="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Candidate]" term=4 -testLoggerAdapter verbose: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.953Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= - testing.go:149: server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: 2021-08-25T09:28:08.953Z [INFO] server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349: entering follower state: follower="Node at b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 [Follower]" leader= -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: election won: tally=2 - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: election won: tally=2 -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering leader state: leader="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Leader]" - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: entering leader state: leader="Node at 051c4c20-115e-e3ed-8769-6c28e69fb7f3 [Leader]" -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: added peer, starting replication: peer=server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:08.953Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: pipelining replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.009Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:09.009Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.114Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.009637643 +0000 UTC m=+8.773853512 (1 nodes), 123.788µs from start of poll, 500.097761ms from cluster start. Timeout at 2021-08-25 09:28:09.114769392 +0000 UTC m=+8.878985421, 105.131909ms after stability - testing.go:149: cluster: 2021-08-25T09:28:09.114Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.009637643 +0000 UTC m=+8.773853512 (1 nodes), 123.788µs from start of poll, 500.097761ms from cluster start. Timeout at 2021-08-25 09:28:09.114769392 +0000 UTC m=+8.878985421, 105.131909ms after stability -testLoggerAdapter verbose: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:09.115Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" - testing.go:149: server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: 2021-08-25T09:28:09.115Z [INFO] server-051c4c20-115e-e3ed-8769-6c28e69fb7f3: aborting pipeline replication: peer="{Voter server-b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349 b3b7d1f7-e5dd-d06f-eff6-ca0b9a0a2349}" ---- PASS: TestRaft_RemoveLeader (0.61s) +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.073Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb} {Suffrage:Voter ID:server-ecfd4bd9-12df-2443-487c-4ed94def9de8 Address:ecfd4bd9-12df-2443-487c-4ed94def9de8}]" + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.073Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb} {Suffrage:Voter ID:server-ecfd4bd9-12df-2443-487c-4ed94def9de8 Address:ecfd4bd9-12df-2443-487c-4ed94def9de8}]" +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.073Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering follower state: follower="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Follower]" leader= + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.073Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering follower state: follower="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Follower]" leader= +testLoggerAdapter verbose: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.074Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb} {Suffrage:Voter ID:server-ecfd4bd9-12df-2443-487c-4ed94def9de8 Address:ecfd4bd9-12df-2443-487c-4ed94def9de8}]" + testing.go:149: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.074Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb} {Suffrage:Voter ID:server-ecfd4bd9-12df-2443-487c-4ed94def9de8 Address:ecfd4bd9-12df-2443-487c-4ed94def9de8}]" +testLoggerAdapter verbose: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.074Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: entering follower state: follower="Node at 54ca903a-4ab2-420b-6088-a1afdededecb [Follower]" leader= + testing.go:149: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.074Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: entering follower state: follower="Node at 54ca903a-4ab2-420b-6088-a1afdededecb [Follower]" leader= +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.074Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb} {Suffrage:Voter ID:server-ecfd4bd9-12df-2443-487c-4ed94def9de8 Address:ecfd4bd9-12df-2443-487c-4ed94def9de8}]" + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.074Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb} {Suffrage:Voter ID:server-ecfd4bd9-12df-2443-487c-4ed94def9de8 Address:ecfd4bd9-12df-2443-487c-4ed94def9de8}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.074Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:04.074Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.074Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: entering follower state: follower="Node at ecfd4bd9-12df-2443-487c-4ed94def9de8 [Follower]" leader= + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.074Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: entering follower state: follower="Node at ecfd4bd9-12df-2443-487c-4ed94def9de8 [Follower]" leader= +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.140Z [WARN] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.140Z [WARN] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.140Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: entering candidate state: node="Node at ecfd4bd9-12df-2443-487c-4ed94def9de8 [Candidate]" term=2 + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.140Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: entering candidate state: node="Node at ecfd4bd9-12df-2443-487c-4ed94def9de8 [Candidate]" term=2 +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.140Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: election won: tally=2 + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.140Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: election won: tally=2 +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: entering leader state: leader="Node at ecfd4bd9-12df-2443-487c-4ed94def9de8 [Leader]" + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: entering leader state: leader="Node at ecfd4bd9-12df-2443-487c-4ed94def9de8 [Leader]" +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: added peer, starting replication: peer=server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: added peer, starting replication: peer=server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: added peer, starting replication: peer=server-54ca903a-4ab2-420b-6088-a1afdededecb + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: added peer, starting replication: peer=server-54ca903a-4ab2-420b-6088-a1afdededecb +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: pipelining replication: peer="{Voter server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232}" + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: pipelining replication: peer="{Voter server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232}" +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: pipelining replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.141Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: pipelining replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.246Z [INFO] stable state for Leader reached at 2021-08-25 09:31:04.141148774 +0000 UTC m=+8.407365178 (1 nodes), 66.51212ms from start of poll, 67.758942ms from cluster start. Timeout at 2021-08-25 09:31:04.24627084 +0000 UTC m=+8.512487264, 105.122086ms after stability + testing.go:149: cluster: 2021-08-25T09:31:04.246Z [INFO] stable state for Leader reached at 2021-08-25 09:31:04.141148774 +0000 UTC m=+8.407365178 (1 nodes), 66.51212ms from start of poll, 67.758942ms from cluster start. Timeout at 2021-08-25 09:31:04.24627084 +0000 UTC m=+8.512487264, 105.122086ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.251Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:04.251Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.356Z [INFO] stable state for Follower reached at 2021-08-25 09:31:04.251818038 +0000 UTC m=+8.518034422 (2 nodes), 2.9µs from start of poll, 178.428186ms from cluster start. Timeout at 2021-08-25 09:31:04.356820775 +0000 UTC m=+8.623037179, 105.002757ms after stability + testing.go:149: cluster: 2021-08-25T09:31:04.356Z [INFO] stable state for Follower reached at 2021-08-25 09:31:04.251818038 +0000 UTC m=+8.518034422 (2 nodes), 2.9µs from start of poll, 178.428186ms from cluster start. Timeout at 2021-08-25 09:31:04.356820775 +0000 UTC m=+8.623037179, 105.002757ms after stability +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: updating configuration: command=RemoveServer server-id=server-ecfd4bd9-12df-2443-487c-4ed94def9de8 server-addr= servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb}]" + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: updating configuration: command=RemoveServer server-id=server-ecfd4bd9-12df-2443-487c-4ed94def9de8 server-addr= servers="[{Suffrage:Voter ID:server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 Address:6ed4721a-7e76-0bd4-50d1-bcb0de4b2232} {Suffrage:Voter ID:server-54ca903a-4ab2-420b-6088-a1afdededecb Address:54ca903a-4ab2-420b-6088-a1afdededecb}]" +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: removed ourself, shutting down + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: removed ourself, shutting down +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: aborting pipeline replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: aborting pipeline replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" +testLoggerAdapter verbose: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: aborting pipeline replication: peer="{Voter server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232}" + testing.go:149: server-ecfd4bd9-12df-2443-487c-4ed94def9de8: 2021-08-25T09:31:04.357Z [INFO] server-ecfd4bd9-12df-2443-487c-4ed94def9de8: aborting pipeline replication: peer="{Voter server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232}" +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.420Z [WARN] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: heartbeat timeout reached, starting election: last-leader=ecfd4bd9-12df-2443-487c-4ed94def9de8 + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.420Z [WARN] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: heartbeat timeout reached, starting election: last-leader=ecfd4bd9-12df-2443-487c-4ed94def9de8 +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.420Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering candidate state: node="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Candidate]" term=3 + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.420Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering candidate state: node="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Candidate]" term=3 +testLoggerAdapter verbose: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.420Z [WARN] server-54ca903a-4ab2-420b-6088-a1afdededecb: rejecting vote request since we have a leader: from=6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 leader=ecfd4bd9-12df-2443-487c-4ed94def9de8 + testing.go:149: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.420Z [WARN] server-54ca903a-4ab2-420b-6088-a1afdededecb: rejecting vote request since we have a leader: from=6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 leader=ecfd4bd9-12df-2443-487c-4ed94def9de8 +testLoggerAdapter verbose: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.460Z [WARN] server-54ca903a-4ab2-420b-6088-a1afdededecb: heartbeat timeout reached, starting election: last-leader=ecfd4bd9-12df-2443-487c-4ed94def9de8 + testing.go:149: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.460Z [WARN] server-54ca903a-4ab2-420b-6088-a1afdededecb: heartbeat timeout reached, starting election: last-leader=ecfd4bd9-12df-2443-487c-4ed94def9de8 +testLoggerAdapter verbose: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.460Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: entering candidate state: node="Node at 54ca903a-4ab2-420b-6088-a1afdededecb [Candidate]" term=3 + testing.go:149: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.460Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: entering candidate state: node="Node at 54ca903a-4ab2-420b-6088-a1afdededecb [Candidate]" term=3 +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.460Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: duplicate requestVote for same term: term=3 + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.460Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [WARN] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: Election timeout reached, restarting election + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [WARN] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: Election timeout reached, restarting election +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering candidate state: node="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Candidate]" term=4 + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering candidate state: node="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Candidate]" term=4 +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: election won: tally=2 + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: election won: tally=2 +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering leader state: leader="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Leader]" + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: entering leader state: leader="Node at 6ed4721a-7e76-0bd4-50d1-bcb0de4b2232 [Leader]" +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: added peer, starting replication: peer=server-54ca903a-4ab2-420b-6088-a1afdededecb + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: added peer, starting replication: peer=server-54ca903a-4ab2-420b-6088-a1afdededecb +testLoggerAdapter verbose: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.479Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: entering follower state: follower="Node at 54ca903a-4ab2-420b-6088-a1afdededecb [Follower]" leader= + testing.go:149: server-54ca903a-4ab2-420b-6088-a1afdededecb: 2021-08-25T09:31:04.479Z [INFO] server-54ca903a-4ab2-420b-6088-a1afdededecb: entering follower state: follower="Node at 54ca903a-4ab2-420b-6088-a1afdededecb [Follower]" leader= +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: pipelining replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.479Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: pipelining replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.567Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:04.567Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.673Z [INFO] stable state for Leader reached at 2021-08-25 09:31:04.568237557 +0000 UTC m=+8.834453961 (1 nodes), 100.59µs from start of poll, 494.847725ms from cluster start. Timeout at 2021-08-25 09:31:04.67328701 +0000 UTC m=+8.939503494, 105.049533ms after stability + testing.go:149: cluster: 2021-08-25T09:31:04.673Z [INFO] stable state for Leader reached at 2021-08-25 09:31:04.568237557 +0000 UTC m=+8.834453961 (1 nodes), 100.59µs from start of poll, 494.847725ms from cluster start. Timeout at 2021-08-25 09:31:04.67328701 +0000 UTC m=+8.939503494, 105.049533ms after stability +testLoggerAdapter verbose: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.673Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: aborting pipeline replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" + testing.go:149: server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: 2021-08-25T09:31:04.673Z [INFO] server-6ed4721a-7e76-0bd4-50d1-bcb0de4b2232: aborting pipeline replication: peer="{Voter server-54ca903a-4ab2-420b-6088-a1afdededecb 54ca903a-4ab2-420b-6088-a1afdededecb}" +--- PASS: TestRaft_RemoveLeader (0.60s) === RUN TestRaft_RemoveLeader_NoShutdown -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering follower state: follower="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Follower]" leader= - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.118Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering follower state: follower="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Follower]" leader= -testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" - testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" -testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= - testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.118Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b} {Suffrage:Voter ID:server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 Address:8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.119Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:09.119Z [INFO] starting stability test: raft-state=Follower - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.119Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering candidate state: node="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Candidate]" term=2 - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering candidate state: node="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Candidate]" term=2 -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: election won: tally=2 - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: election won: tally=2 -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering leader state: leader="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Leader]" - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering leader state: leader="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Leader]" -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-30230337-8642-d28a-9e21-942a5cb1c648 - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.186Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-30230337-8642-d28a-9e21-942a5cb1c648 -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.187Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.292Z [INFO] stable state for Follower reached at 2021-08-25 09:28:09.187123548 +0000 UTC m=+8.951339476 (2 nodes), 67.818796ms from start of poll, 68.890557ms from cluster start. Timeout at 2021-08-25 09:28:09.292176504 +0000 UTC m=+9.056392493, 105.053017ms after stability - testing.go:149: cluster: 2021-08-25T09:28:09.292Z [INFO] stable state for Follower reached at 2021-08-25 09:28:09.187123548 +0000 UTC m=+8.951339476 (2 nodes), 67.818796ms from start of poll, 68.890557ms from cluster start. Timeout at 2021-08-25 09:28:09.292176504 +0000 UTC m=+9.056392493, 105.053017ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.292Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:09.292Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.397Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.292426941 +0000 UTC m=+9.056642830 (1 nodes), 2.54µs from start of poll, 174.193911ms from cluster start. Timeout at 2021-08-25 09:28:09.397479657 +0000 UTC m=+9.161695566, 105.052736ms after stability - testing.go:149: cluster: 2021-08-25T09:28:09.397Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.292426941 +0000 UTC m=+9.056642830 (1 nodes), 2.54µs from start of poll, 174.193911ms from cluster start. Timeout at 2021-08-25 09:28:09.397479657 +0000 UTC m=+9.161695566, 105.052736ms after stability -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: updating configuration: command=RemoveServer server-id=server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 server-addr= servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}]" - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: updating configuration: command=RemoveServer server-id=server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 server-addr= servers="[{Suffrage:Voter ID:server-30230337-8642-d28a-9e21-942a5cb1c648 Address:30230337-8642-d28a-9e21-942a5cb1c648} {Suffrage:Voter ID:server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b Address:5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}]" -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: removed ourself, transitioning to follower - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.400Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: removed ourself, transitioning to follower -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: entering follower state: follower="Node at 8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 [Follower]" leader= -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.401Z [INFO] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: aborting pipeline replication: peer="{Voter server-30230337-8642-d28a-9e21-942a5cb1c648 30230337-8642-d28a-9e21-942a5cb1c648}" -testLoggerAdapter verbose: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.463Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: not part of stable configuration, aborting election - testing.go:149: server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: 2021-08-25T09:28:09.463Z [WARN] server-8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31: not part of stable configuration, aborting election -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=3 - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.494Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=3 -testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.494Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: rejecting vote request since we have a leader: from=30230337-8642-d28a-9e21-942a5cb1c648 leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 - testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.494Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: rejecting vote request since we have a leader: from=30230337-8642-d28a-9e21-942a5cb1c648 leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.506Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:09.506Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 - testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [WARN] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: heartbeat timeout reached, starting election: last-leader=8dbdd131-7d9e-ab97-aa0a-137c6ac6ea31 -testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering candidate state: node="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Candidate]" term=3 - testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.518Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering candidate state: node="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Candidate]" term=3 -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.519Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: duplicate requestVote for same term: term=3 - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.519Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: Election timeout reached, restarting election - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [WARN] server-30230337-8642-d28a-9e21-942a5cb1c648: Election timeout reached, restarting election -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=4 - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering candidate state: node="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Candidate]" term=4 -testLoggerAdapter verbose: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.552Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= - testing.go:149: server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: 2021-08-25T09:28:09.552Z [INFO] server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b: entering follower state: follower="Node at 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b [Follower]" leader= -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: election won: tally=2 - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: election won: tally=2 -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering leader state: leader="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Leader]" - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: entering leader state: leader="Node at 30230337-8642-d28a-9e21-942a5cb1c648 [Leader]" -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.552Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: added peer, starting replication: peer=server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.553Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.553Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: pipelining replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.658Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.553221882 +0000 UTC m=+9.317437791 (1 nodes), 46.613027ms from start of poll, 434.988872ms from cluster start. Timeout at 2021-08-25 09:28:09.65836279 +0000 UTC m=+9.422578759, 105.140968ms after stability - testing.go:149: cluster: 2021-08-25T09:28:09.658Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.553221882 +0000 UTC m=+9.317437791 (1 nodes), 46.613027ms from start of poll, 434.988872ms from cluster start. Timeout at 2021-08-25 09:28:09.65836279 +0000 UTC m=+9.422578759, 105.140968ms after stability -testLoggerAdapter verbose: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.763Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" - testing.go:149: server-30230337-8642-d28a-9e21-942a5cb1c648: 2021-08-25T09:28:09.763Z [INFO] server-30230337-8642-d28a-9e21-942a5cb1c648: aborting pipeline replication: peer="{Voter server-5f9267ed-84b2-25ec-98ff-9f2ed4e6394b 5f9267ed-84b2-25ec-98ff-9f2ed4e6394b}" ---- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s) +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.677Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 Address:4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9} {Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.677Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 Address:4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9} {Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.677Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering follower state: follower="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Follower]" leader= + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.677Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering follower state: follower="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Follower]" leader= +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:04.677Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 Address:4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9} {Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:04.677Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 Address:4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9} {Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:04.678Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering follower state: follower="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Follower]" leader= + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:04.678Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering follower state: follower="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Follower]" leader= +testLoggerAdapter verbose: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:04.678Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 Address:4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9} {Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" + testing.go:149: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:04.678Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 Address:4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9} {Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.678Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:04.678Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:04.678Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: entering follower state: follower="Node at e56fef11-7326-8fbd-73a1-b576a906494c [Follower]" leader= + testing.go:149: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:04.678Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: entering follower state: follower="Node at e56fef11-7326-8fbd-73a1-b576a906494c [Follower]" leader= +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.736Z [WARN] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.736Z [WARN] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.736Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering candidate state: node="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Candidate]" term=2 + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.736Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering candidate state: node="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Candidate]" term=2 +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.736Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: election won: tally=2 + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.736Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: election won: tally=2 +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering leader state: leader="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Leader]" + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering leader state: leader="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Leader]" +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: added peer, starting replication: peer=server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: added peer, starting replication: peer=server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: added peer, starting replication: peer=server-e56fef11-7326-8fbd-73a1-b576a906494c + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: added peer, starting replication: peer=server-e56fef11-7326-8fbd-73a1-b576a906494c +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: pipelining replication: peer="{Voter server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 fe6ed2ae-4788-f9f5-8990-73aa1ad45c74}" + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: pipelining replication: peer="{Voter server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 fe6ed2ae-4788-f9f5-8990-73aa1ad45c74}" +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: pipelining replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.737Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: pipelining replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.842Z [INFO] stable state for Follower reached at 2021-08-25 09:31:04.737044469 +0000 UTC m=+9.003260873 (2 nodes), 58.686876ms from start of poll, 59.767195ms from cluster start. Timeout at 2021-08-25 09:31:04.842186753 +0000 UTC m=+9.108403157, 105.142284ms after stability + testing.go:149: cluster: 2021-08-25T09:31:04.842Z [INFO] stable state for Follower reached at 2021-08-25 09:31:04.737044469 +0000 UTC m=+9.003260873 (2 nodes), 58.686876ms from start of poll, 59.767195ms from cluster start. Timeout at 2021-08-25 09:31:04.842186753 +0000 UTC m=+9.108403157, 105.142284ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.842Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:04.842Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:04.947Z [INFO] stable state for Leader reached at 2021-08-25 09:31:04.842468167 +0000 UTC m=+9.108684571 (1 nodes), 2.6µs from start of poll, 165.190893ms from cluster start. Timeout at 2021-08-25 09:31:04.947577194 +0000 UTC m=+9.213793598, 105.109027ms after stability + testing.go:149: cluster: 2021-08-25T09:31:04.947Z [INFO] stable state for Leader reached at 2021-08-25 09:31:04.842468167 +0000 UTC m=+9.108684571 (1 nodes), 2.6µs from start of poll, 165.190893ms from cluster start. Timeout at 2021-08-25 09:31:04.947577194 +0000 UTC m=+9.213793598, 105.109027ms after stability +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: updating configuration: command=RemoveServer server-id=server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 server-addr= servers="[{Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: updating configuration: command=RemoveServer server-id=server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 server-addr= servers="[{Suffrage:Voter ID:server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 Address:fe6ed2ae-4788-f9f5-8990-73aa1ad45c74} {Suffrage:Voter ID:server-e56fef11-7326-8fbd-73a1-b576a906494c Address:e56fef11-7326-8fbd-73a1-b576a906494c}]" +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: removed ourself, transitioning to follower + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: removed ourself, transitioning to follower +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering follower state: follower="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Follower]" leader= + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: entering follower state: follower="Node at 4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 [Follower]" leader= +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: aborting pipeline replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: aborting pipeline replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: aborting pipeline replication: peer="{Voter server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 fe6ed2ae-4788-f9f5-8990-73aa1ad45c74}" + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:04.950Z [INFO] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: aborting pipeline replication: peer="{Voter server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 fe6ed2ae-4788-f9f5-8990-73aa1ad45c74}" +testLoggerAdapter verbose: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.019Z [WARN] server-e56fef11-7326-8fbd-73a1-b576a906494c: heartbeat timeout reached, starting election: last-leader=4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 + testing.go:149: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.019Z [WARN] server-e56fef11-7326-8fbd-73a1-b576a906494c: heartbeat timeout reached, starting election: last-leader=4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 +testLoggerAdapter verbose: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.019Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: entering candidate state: node="Node at e56fef11-7326-8fbd-73a1-b576a906494c [Candidate]" term=3 + testing.go:149: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.019Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: entering candidate state: node="Node at e56fef11-7326-8fbd-73a1-b576a906494c [Candidate]" term=3 +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.020Z [WARN] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: rejecting vote request since we have a leader: from=e56fef11-7326-8fbd-73a1-b576a906494c leader=4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.020Z [WARN] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: rejecting vote request since we have a leader: from=e56fef11-7326-8fbd-73a1-b576a906494c leader=4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 +testLoggerAdapter verbose: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:05.030Z [WARN] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: not part of stable configuration, aborting election + testing.go:149: server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: 2021-08-25T09:31:05.030Z [WARN] server-4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9: not part of stable configuration, aborting election +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.037Z [WARN] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: heartbeat timeout reached, starting election: last-leader=4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.037Z [WARN] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: heartbeat timeout reached, starting election: last-leader=4abb3925-2bc0-5dd0-2cb6-eb7a96035fc9 +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.037Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering candidate state: node="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Candidate]" term=3 + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.037Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering candidate state: node="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Candidate]" term=3 +testLoggerAdapter verbose: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.037Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: duplicate requestVote for same term: term=3 + testing.go:149: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.037Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.056Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:05.056Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [WARN] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: Election timeout reached, restarting election + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [WARN] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: Election timeout reached, restarting election +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering candidate state: node="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Candidate]" term=4 + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering candidate state: node="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Candidate]" term=4 +testLoggerAdapter verbose: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.091Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: entering follower state: follower="Node at e56fef11-7326-8fbd-73a1-b576a906494c [Follower]" leader= + testing.go:149: server-e56fef11-7326-8fbd-73a1-b576a906494c: 2021-08-25T09:31:05.091Z [INFO] server-e56fef11-7326-8fbd-73a1-b576a906494c: entering follower state: follower="Node at e56fef11-7326-8fbd-73a1-b576a906494c [Follower]" leader= +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: election won: tally=2 + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: election won: tally=2 +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering leader state: leader="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Leader]" + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: entering leader state: leader="Node at fe6ed2ae-4788-f9f5-8990-73aa1ad45c74 [Leader]" +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: added peer, starting replication: peer=server-e56fef11-7326-8fbd-73a1-b576a906494c + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.091Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: added peer, starting replication: peer=server-e56fef11-7326-8fbd-73a1-b576a906494c +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.092Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: pipelining replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.092Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: pipelining replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.197Z [INFO] stable state for Leader reached at 2021-08-25 09:31:05.092163926 +0000 UTC m=+9.358380390 (1 nodes), 35.764934ms from start of poll, 414.886712ms from cluster start. Timeout at 2021-08-25 09:31:05.197275413 +0000 UTC m=+9.463491817, 105.111427ms after stability + testing.go:149: cluster: 2021-08-25T09:31:05.197Z [INFO] stable state for Leader reached at 2021-08-25 09:31:05.092163926 +0000 UTC m=+9.358380390 (1 nodes), 35.764934ms from start of poll, 414.886712ms from cluster start. Timeout at 2021-08-25 09:31:05.197275413 +0000 UTC m=+9.463491817, 105.111427ms after stability +testLoggerAdapter verbose: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.302Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: aborting pipeline replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" + testing.go:149: server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: 2021-08-25T09:31:05.302Z [INFO] server-fe6ed2ae-4788-f9f5-8990-73aa1ad45c74: aborting pipeline replication: peer="{Voter server-e56fef11-7326-8fbd-73a1-b576a906494c e56fef11-7326-8fbd-73a1-b576a906494c}" +--- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s) === RUN TestRaft_RemoveFollower_SplitCluster -testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.767Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" - testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.767Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" -testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.768Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering follower state: follower="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Follower]" leader= - testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:09.768Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering follower state: follower="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Follower]" leader= -testLoggerAdapter verbose: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.768Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" - testing.go:149: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.768Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" -testLoggerAdapter verbose: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.768Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" - testing.go:149: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.768Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.768Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.768Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-30d61c37-79cc-9c5d-feab-0046126d51ca Address:30d61c37-79cc-9c5d-feab-0046126d51ca} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.768Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:09.768Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.769Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: entering follower state: follower="Node at c7612fa6-4ceb-63b0-97ad-63dd16716550 [Follower]" leader= - testing.go:149: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:09.769Z [INFO] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: entering follower state: follower="Node at c7612fa6-4ceb-63b0-97ad-63dd16716550 [Follower]" leader= -testLoggerAdapter verbose: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.769Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: entering follower state: follower="Node at 30d61c37-79cc-9c5d-feab-0046126d51ca [Follower]" leader= - testing.go:149: server-30d61c37-79cc-9c5d-feab-0046126d51ca: 2021-08-25T09:28:09.769Z [INFO] server-30d61c37-79cc-9c5d-feab-0046126d51ca: entering follower state: follower="Node at 30d61c37-79cc-9c5d-feab-0046126d51ca [Follower]" leader= -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.769Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.769Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering candidate state: node="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Candidate]" term=2 - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering candidate state: node="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Candidate]" term=2 -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: election won: tally=3 - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.831Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: election won: tally=3 -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering leader state: leader="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Leader]" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering leader state: leader="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Leader]" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: added peer, starting replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:09.832Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: pipelining replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:09.937Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.832253849 +0000 UTC m=+9.596469738 (1 nodes), 63.244242ms from start of poll, 64.715725ms from cluster start. Timeout at 2021-08-25 09:28:09.937392377 +0000 UTC m=+9.701608406, 105.138668ms after stability - testing.go:149: cluster: 2021-08-25T09:28:09.937Z [INFO] stable state for Leader reached at 2021-08-25 09:28:09.832253849 +0000 UTC m=+9.596469738 (1 nodes), 63.244242ms from start of poll, 64.715725ms from cluster start. Timeout at 2021-08-25 09:28:09.937392377 +0000 UTC m=+9.701608406, 105.138668ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.042Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:10.042Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.148Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.042953807 +0000 UTC m=+9.807169696 (3 nodes), 3.24µs from start of poll, 275.415683ms from cluster start. Timeout at 2021-08-25 09:28:10.148092315 +0000 UTC m=+9.912308264, 105.138568ms after stability - testing.go:149: cluster: 2021-08-25T09:28:10.148Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.042953807 +0000 UTC m=+9.807169696 (3 nodes), 3.24µs from start of poll, 275.415683ms from cluster start. Timeout at 2021-08-25 09:28:10.148092315 +0000 UTC m=+9.912308264, 105.138568ms after stability -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: updating configuration: command=RemoveServer server-id=server-30d61c37-79cc-9c5d-feab-0046126d51ca server-addr= servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: updating configuration: command=RemoveServer server-id=server-30d61c37-79cc-9c5d-feab-0046126d51ca server-addr= servers="[{Suffrage:Voter ID:server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 Address:1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1} {Suffrage:Voter ID:server-c7612fa6-4ceb-63b0-97ad-63dd16716550 Address:c7612fa6-4ceb-63b0-97ad-63dd16716550} {Suffrage:Voter ID:server-fd8b4f14-3588-16aa-f3ea-cff79821e401 Address:fd8b4f14-3588-16aa-f3ea-cff79821e401}]" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: removed peer, stopping replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca last-index=3 - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: removed peer, stopping replication: peer=server-30d61c37-79cc-9c5d-feab-0046126d51ca last-index=3 -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="append pipeline closed" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="append pipeline closed" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="append pipeline closed" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to pipeline appendEntries: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="append pipeline closed" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.148Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: aborting pipeline replication: peer="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.149Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.154Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.154Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.157Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.157Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.165Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.165Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.170Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.170Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.173Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.173Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.176Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.176Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.183Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.183Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=c7612fa6-4ceb-63b0-97ad-63dd16716550 error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.186Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.186Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to heartbeat to: peer=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.191Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=50.103124ms - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.191Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=50.103124ms -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.193Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" -testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [WARN] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: heartbeat timeout reached, starting election: last-leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 - testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [WARN] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: heartbeat timeout reached, starting election: last-leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 -testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering candidate state: node="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Candidate]" term=3 - testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [INFO] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: entering candidate state: node="Node at 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 [Candidate]" term=3 -testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-fd8b4f14-3588-16aa-f3ea-cff79821e401 fd8b4f14-3588-16aa-f3ea-cff79821e401}" error="failed to connect to peer: fd8b4f14-3588-16aa-f3ea-cff79821e401" - testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-fd8b4f14-3588-16aa-f3ea-cff79821e401 fd8b4f14-3588-16aa-f3ea-cff79821e401}" error="failed to connect to peer: fd8b4f14-3588-16aa-f3ea-cff79821e401" -testLoggerAdapter verbose: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:10.195Z [WARN] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: rejecting vote request since we have a leader: from=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 - testing.go:149: server-c7612fa6-4ceb-63b0-97ad-63dd16716550: 2021-08-25T09:28:10.195Z [WARN] server-c7612fa6-4ceb-63b0-97ad-63dd16716550: rejecting vote request since we have a leader: from=1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 leader=fd8b4f14-3588-16aa-f3ea-cff79821e401 -testLoggerAdapter verbose: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" error="failed to connect to peer: 30d61c37-79cc-9c5d-feab-0046126d51ca" - testing.go:149: server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: 2021-08-25T09:28:10.195Z [ERROR] server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1: failed to make requestVote RPC: target="{Voter server-30d61c37-79cc-9c5d-feab-0046126d51ca 30d61c37-79cc-9c5d-feab-0046126d51ca}" error="failed to connect to peer: 30d61c37-79cc-9c5d-feab-0046126d51ca" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=60.376569ms - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 time=60.376569ms -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 time=55.241106ms - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact: server-id=server-c7612fa6-4ceb-63b0-97ad-63dd16716550 time=55.241106ms -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact quorum of nodes, stepping down - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [WARN] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [INFO] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: entering follower state: follower="Node at fd8b4f14-3588-16aa-f3ea-cff79821e401 [Follower]" leader= -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-c7612fa6-4ceb-63b0-97ad-63dd16716550 c7612fa6-4ceb-63b0-97ad-63dd16716550}" error="failed to connect to peer: c7612fa6-4ceb-63b0-97ad-63dd16716550" -testLoggerAdapter verbose: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" - testing.go:149: server-fd8b4f14-3588-16aa-f3ea-cff79821e401: 2021-08-25T09:28:10.201Z [ERROR] server-fd8b4f14-3588-16aa-f3ea-cff79821e401: failed to appendEntries to: peer="{Voter server-1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1}" error="failed to connect to peer: 1f76d72f-fc93-5c45-9f9b-58b6c3bfaeb1" ---- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.308Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.308Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.308Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering follower state: follower="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Follower]" leader= + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.308Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering follower state: follower="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Follower]" leader= +testLoggerAdapter verbose: server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: 2021-08-25T09:31:05.308Z [INFO] server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" + testing.go:149: server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: 2021-08-25T09:31:05.308Z [INFO] server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" +testLoggerAdapter verbose: server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: 2021-08-25T09:31:05.308Z [INFO] server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: entering follower state: follower="Node at c778aa7e-ec4c-add6-15f9-47dc1da58f15 [Follower]" leader= + testing.go:149: server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: 2021-08-25T09:31:05.308Z [INFO] server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: entering follower state: follower="Node at c778aa7e-ec4c-add6-15f9-47dc1da58f15 [Follower]" leader= +testLoggerAdapter verbose: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.309Z [INFO] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" + testing.go:149: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.309Z [INFO] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" +testLoggerAdapter verbose: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.309Z [INFO] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: entering follower state: follower="Node at 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d [Follower]" leader= + testing.go:149: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.309Z [INFO] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: entering follower state: follower="Node at 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d [Follower]" leader= +testLoggerAdapter verbose: server-aaaaae65-4033-e76b-a58c-653a480cbac8: 2021-08-25T09:31:05.309Z [INFO] server-aaaaae65-4033-e76b-a58c-653a480cbac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" + testing.go:149: server-aaaaae65-4033-e76b-a58c-653a480cbac8: 2021-08-25T09:31:05.309Z [INFO] server-aaaaae65-4033-e76b-a58c-653a480cbac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d} {Suffrage:Voter ID:server-aaaaae65-4033-e76b-a58c-653a480cbac8 Address:aaaaae65-4033-e76b-a58c-653a480cbac8}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.309Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:05.309Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-aaaaae65-4033-e76b-a58c-653a480cbac8: 2021-08-25T09:31:05.309Z [INFO] server-aaaaae65-4033-e76b-a58c-653a480cbac8: entering follower state: follower="Node at aaaaae65-4033-e76b-a58c-653a480cbac8 [Follower]" leader= + testing.go:149: server-aaaaae65-4033-e76b-a58c-653a480cbac8: 2021-08-25T09:31:05.309Z [INFO] server-aaaaae65-4033-e76b-a58c-653a480cbac8: entering follower state: follower="Node at aaaaae65-4033-e76b-a58c-653a480cbac8 [Follower]" leader= +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.360Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.360Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering candidate state: node="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Candidate]" term=2 + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering candidate state: node="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Candidate]" term=2 +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: election won: tally=3 + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: election won: tally=3 +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering leader state: leader="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Leader]" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering leader state: leader="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Leader]" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: added peer, starting replication: peer=server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: added peer, starting replication: peer=server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: added peer, starting replication: peer=server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: added peer, starting replication: peer=server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: added peer, starting replication: peer=server-aaaaae65-4033-e76b-a58c-653a480cbac8 + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: added peer, starting replication: peer=server-aaaaae65-4033-e76b-a58c-653a480cbac8 +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: pipelining replication: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: pipelining replication: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: pipelining replication: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.361Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: pipelining replication: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.362Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: pipelining replication: peer="{Voter server-aaaaae65-4033-e76b-a58c-653a480cbac8 aaaaae65-4033-e76b-a58c-653a480cbac8}" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.362Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: pipelining replication: peer="{Voter server-aaaaae65-4033-e76b-a58c-653a480cbac8 aaaaae65-4033-e76b-a58c-653a480cbac8}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.466Z [INFO] stable state for Leader reached at 2021-08-25 09:31:05.361711237 +0000 UTC m=+9.627927681 (1 nodes), 51.996067ms from start of poll, 53.76548ms from cluster start. Timeout at 2021-08-25 09:31:05.466881378 +0000 UTC m=+9.733097842, 105.170161ms after stability + testing.go:149: cluster: 2021-08-25T09:31:05.466Z [INFO] stable state for Leader reached at 2021-08-25 09:31:05.361711237 +0000 UTC m=+9.627927681 (1 nodes), 51.996067ms from start of poll, 53.76548ms from cluster start. Timeout at 2021-08-25 09:31:05.466881378 +0000 UTC m=+9.733097842, 105.170161ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.572Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:05.572Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.677Z [INFO] stable state for Follower reached at 2021-08-25 09:31:05.572568931 +0000 UTC m=+9.838785315 (3 nodes), 3.48µs from start of poll, 264.623114ms from cluster start. Timeout at 2021-08-25 09:31:05.677706476 +0000 UTC m=+9.943922960, 105.137645ms after stability + testing.go:149: cluster: 2021-08-25T09:31:05.677Z [INFO] stable state for Follower reached at 2021-08-25 09:31:05.572568931 +0000 UTC m=+9.838785315 (3 nodes), 3.48µs from start of poll, 264.623114ms from cluster start. Timeout at 2021-08-25 09:31:05.677706476 +0000 UTC m=+9.943922960, 105.137645ms after stability +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: updating configuration: command=RemoveServer server-id=server-aaaaae65-4033-e76b-a58c-653a480cbac8 server-addr= servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}]" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: updating configuration: command=RemoveServer server-id=server-aaaaae65-4033-e76b-a58c-653a480cbac8 server-addr= servers="[{Suffrage:Voter ID:server-207f0be7-9230-744d-939c-5efa7e5a0a8e Address:207f0be7-9230-744d-939c-5efa7e5a0a8e} {Suffrage:Voter ID:server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 Address:c778aa7e-ec4c-add6-15f9-47dc1da58f15} {Suffrage:Voter ID:server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d Address:17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}]" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: removed peer, stopping replication: peer=server-aaaaae65-4033-e76b-a58c-653a480cbac8 last-index=3 + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: removed peer, stopping replication: peer=server-aaaaae65-4033-e76b-a58c-653a480cbac8 last-index=3 +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: aborting pipeline replication: peer="{Voter server-aaaaae65-4033-e76b-a58c-653a480cbac8 aaaaae65-4033-e76b-a58c-653a480cbac8}" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: aborting pipeline replication: peer="{Voter server-aaaaae65-4033-e76b-a58c-653a480cbac8 aaaaae65-4033-e76b-a58c-653a480cbac8}" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to pipeline appendEntries: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="append pipeline closed" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to pipeline appendEntries: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="append pipeline closed" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: aborting pipeline replication: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: aborting pipeline replication: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to pipeline appendEntries: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="append pipeline closed" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to pipeline appendEntries: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="append pipeline closed" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: aborting pipeline replication: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: aborting pipeline replication: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=c778aa7e-ec4c-add6-15f9-47dc1da58f15 error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.678Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=c778aa7e-ec4c-add6-15f9-47dc1da58f15 error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.682Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.682Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.684Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.684Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.686Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.686Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.696Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=c778aa7e-ec4c-add6-15f9-47dc1da58f15 error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.696Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=c778aa7e-ec4c-add6-15f9-47dc1da58f15 error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.699Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.699Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.702Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.702Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.703Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.703Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.715Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=c778aa7e-ec4c-add6-15f9-47dc1da58f15 error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.715Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=c778aa7e-ec4c-add6-15f9-47dc1da58f15 error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.718Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.718Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to heartbeat to: peer=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.719Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.719Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.719Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.719Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.723Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact: server-id=server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 time=50.109284ms + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.723Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact: server-id=server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 time=50.109284ms +testLoggerAdapter verbose: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [WARN] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: heartbeat timeout reached, starting election: last-leader=207f0be7-9230-744d-939c-5efa7e5a0a8e + testing.go:149: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [WARN] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: heartbeat timeout reached, starting election: last-leader=207f0be7-9230-744d-939c-5efa7e5a0a8e +testLoggerAdapter verbose: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [INFO] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: entering candidate state: node="Node at 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d [Candidate]" term=3 + testing.go:149: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [INFO] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: entering candidate state: node="Node at 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d [Candidate]" term=3 +testLoggerAdapter verbose: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [ERROR] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: failed to make requestVote RPC: target="{Voter server-aaaaae65-4033-e76b-a58c-653a480cbac8 aaaaae65-4033-e76b-a58c-653a480cbac8}" error="failed to connect to peer: aaaaae65-4033-e76b-a58c-653a480cbac8" + testing.go:149: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [ERROR] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: failed to make requestVote RPC: target="{Voter server-aaaaae65-4033-e76b-a58c-653a480cbac8 aaaaae65-4033-e76b-a58c-653a480cbac8}" error="failed to connect to peer: aaaaae65-4033-e76b-a58c-653a480cbac8" +testLoggerAdapter verbose: server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: 2021-08-25T09:31:05.732Z [WARN] server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: rejecting vote request since we have a leader: from=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d leader=207f0be7-9230-744d-939c-5efa7e5a0a8e + testing.go:149: server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: 2021-08-25T09:31:05.732Z [WARN] server-c778aa7e-ec4c-add6-15f9-47dc1da58f15: rejecting vote request since we have a leader: from=17f9739f-bdf2-fc3a-d284-e52a21ee6f7d leader=207f0be7-9230-744d-939c-5efa7e5a0a8e +testLoggerAdapter verbose: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [ERROR] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: failed to make requestVote RPC: target="{Voter server-207f0be7-9230-744d-939c-5efa7e5a0a8e 207f0be7-9230-744d-939c-5efa7e5a0a8e}" error="failed to connect to peer: 207f0be7-9230-744d-939c-5efa7e5a0a8e" + testing.go:149: server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: 2021-08-25T09:31:05.732Z [ERROR] server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d: failed to make requestVote RPC: target="{Voter server-207f0be7-9230-744d-939c-5efa7e5a0a8e 207f0be7-9230-744d-939c-5efa7e5a0a8e}" error="failed to connect to peer: 207f0be7-9230-744d-939c-5efa7e5a0a8e" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact: server-id=server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 time=60.447371ms + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact: server-id=server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 time=60.447371ms +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact: server-id=server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d time=58.0063ms + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact: server-id=server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d time=58.0063ms +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact quorum of nodes, stepping down + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [WARN] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering follower state: follower="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Follower]" leader= + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [INFO] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: entering follower state: follower="Node at 207f0be7-9230-744d-939c-5efa7e5a0a8e [Follower]" leader= +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-17f9739f-bdf2-fc3a-d284-e52a21ee6f7d 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d}" error="failed to connect to peer: 17f9739f-bdf2-fc3a-d284-e52a21ee6f7d" +testLoggerAdapter verbose: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" + testing.go:149: server-207f0be7-9230-744d-939c-5efa7e5a0a8e: 2021-08-25T09:31:05.734Z [ERROR] server-207f0be7-9230-744d-939c-5efa7e5a0a8e: failed to appendEntries to: peer="{Voter server-c778aa7e-ec4c-add6-15f9-47dc1da58f15 c778aa7e-ec4c-add6-15f9-47dc1da58f15}" error="failed to connect to peer: c778aa7e-ec4c-add6-15f9-47dc1da58f15" +--- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) === RUN TestRaft_AddKnownPeer -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.205Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.205Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" -testLoggerAdapter verbose: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" - testing.go:149: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" -testLoggerAdapter verbose: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" - testing.go:149: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.206Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:10.206Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.206Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering follower state: follower="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Follower]" leader= - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.206Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering follower state: follower="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Follower]" leader= -testLoggerAdapter verbose: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: entering follower state: follower="Node at be21088c-02b5-ae20-1bc8-d13bc655a613 [Follower]" leader= -testLoggerAdapter verbose: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: entering follower state: follower="Node at 09a05e3a-97e3-9542-8e3d-14ac482b9432 [Follower]" leader= - testing.go:149: server-be21088c-02b5-ae20-1bc8-d13bc655a613: 2021-08-25T09:28:10.206Z [INFO] server-be21088c-02b5-ae20-1bc8-d13bc655a613: entering follower state: follower="Node at be21088c-02b5-ae20-1bc8-d13bc655a613 [Follower]" leader= - testing.go:149: server-09a05e3a-97e3-9542-8e3d-14ac482b9432: 2021-08-25T09:28:10.206Z [INFO] server-09a05e3a-97e3-9542-8e3d-14ac482b9432: entering follower state: follower="Node at 09a05e3a-97e3-9542-8e3d-14ac482b9432 [Follower]" leader= -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [WARN] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [WARN] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering candidate state: node="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Candidate]" term=2 - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering candidate state: node="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Candidate]" term=2 -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: election won: tally=2 - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: election won: tally=2 -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering leader state: leader="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Leader]" - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: entering leader state: leader="Node at 4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 [Leader]" -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-be21088c-02b5-ae20-1bc8-d13bc655a613 - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.260Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: added peer, starting replication: peer=server-be21088c-02b5-ae20-1bc8-d13bc655a613 -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.261Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: pipelining replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.366Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.260976684 +0000 UTC m=+10.025192593 (1 nodes), 54.490176ms from start of poll, 55.579355ms from cluster start. Timeout at 2021-08-25 09:28:10.366007922 +0000 UTC m=+10.130223911, 105.031318ms after stability - testing.go:149: cluster: 2021-08-25T09:28:10.366Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.260976684 +0000 UTC m=+10.025192593 (1 nodes), 54.490176ms from start of poll, 55.579355ms from cluster start. Timeout at 2021-08-25 09:28:10.366007922 +0000 UTC m=+10.130223911, 105.031318ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.366Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:10.366Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.471Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.366266798 +0000 UTC m=+10.130482687 (2 nodes), 2.9µs from start of poll, 160.869449ms from cluster start. Timeout at 2021-08-25 09:28:10.471380229 +0000 UTC m=+10.235596198, 105.113511ms after stability - testing.go:149: cluster: 2021-08-25T09:28:10.471Z [INFO] stable state for Follower reached at 2021-08-25 09:28:10.366266798 +0000 UTC m=+10.130482687 (2 nodes), 2.9µs from start of poll, 160.869449ms from cluster start. Timeout at 2021-08-25 09:28:10.471380229 +0000 UTC m=+10.235596198, 105.113511ms after stability -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.471Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: updating configuration: command=AddStaging server-id=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 server-addr=09a05e3a-97e3-9542-8e3d-14ac482b9432 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.471Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: updating configuration: command=AddStaging server-id=server-09a05e3a-97e3-9542-8e3d-14ac482b9432 server-addr=09a05e3a-97e3-9542-8e3d-14ac482b9432 servers="[{Suffrage:Voter ID:server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15 Address:4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15} {Suffrage:Voter ID:server-09a05e3a-97e3-9542-8e3d-14ac482b9432 Address:09a05e3a-97e3-9542-8e3d-14ac482b9432} {Suffrage:Voter ID:server-be21088c-02b5-ae20-1bc8-d13bc655a613 Address:be21088c-02b5-ae20-1bc8-d13bc655a613}]" -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-be21088c-02b5-ae20-1bc8-d13bc655a613 be21088c-02b5-ae20-1bc8-d13bc655a613}" -testLoggerAdapter verbose: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" - testing.go:149: server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: 2021-08-25T09:28:10.472Z [INFO] server-4d80a31f-131f-2b64-4eb9-b9c6c9d0ad15: aborting pipeline replication: peer="{Voter server-09a05e3a-97e3-9542-8e3d-14ac482b9432 09a05e3a-97e3-9542-8e3d-14ac482b9432}" +testLoggerAdapter verbose: server-178aa703-72b6-fd56-e030-6e845a8d3e97: 2021-08-25T09:31:05.738Z [INFO] server-178aa703-72b6-fd56-e030-6e845a8d3e97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" + testing.go:149: server-178aa703-72b6-fd56-e030-6e845a8d3e97: 2021-08-25T09:31:05.738Z [INFO] server-178aa703-72b6-fd56-e030-6e845a8d3e97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" +testLoggerAdapter verbose: server-178aa703-72b6-fd56-e030-6e845a8d3e97: 2021-08-25T09:31:05.739Z [INFO] server-178aa703-72b6-fd56-e030-6e845a8d3e97: entering follower state: follower="Node at 178aa703-72b6-fd56-e030-6e845a8d3e97 [Follower]" leader= + testing.go:149: server-178aa703-72b6-fd56-e030-6e845a8d3e97: 2021-08-25T09:31:05.739Z [INFO] server-178aa703-72b6-fd56-e030-6e845a8d3e97: entering follower state: follower="Node at 178aa703-72b6-fd56-e030-6e845a8d3e97 [Follower]" leader= +testLoggerAdapter verbose: server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: 2021-08-25T09:31:05.739Z [INFO] server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" + testing.go:149: server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: 2021-08-25T09:31:05.739Z [INFO] server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" +testLoggerAdapter verbose: server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: 2021-08-25T09:31:05.739Z [INFO] server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: entering follower state: follower="Node at 931fe7dc-09f5-2a29-e0b1-3e9ac5532433 [Follower]" leader= + testing.go:149: server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: 2021-08-25T09:31:05.739Z [INFO] server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433: entering follower state: follower="Node at 931fe7dc-09f5-2a29-e0b1-3e9ac5532433 [Follower]" leader= +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.739Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.739Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.739Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:05.739Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.739Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: entering follower state: follower="Node at 204c7154-9551-7527-7b60-f402666ad003 [Follower]" leader= + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.739Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: entering follower state: follower="Node at 204c7154-9551-7527-7b60-f402666ad003 [Follower]" leader= +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.794Z [WARN] server-204c7154-9551-7527-7b60-f402666ad003: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.794Z [WARN] server-204c7154-9551-7527-7b60-f402666ad003: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.794Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: entering candidate state: node="Node at 204c7154-9551-7527-7b60-f402666ad003 [Candidate]" term=2 + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.794Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: entering candidate state: node="Node at 204c7154-9551-7527-7b60-f402666ad003 [Candidate]" term=2 +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: election won: tally=2 + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: election won: tally=2 +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: entering leader state: leader="Node at 204c7154-9551-7527-7b60-f402666ad003 [Leader]" + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: entering leader state: leader="Node at 204c7154-9551-7527-7b60-f402666ad003 [Leader]" +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: added peer, starting replication: peer=server-178aa703-72b6-fd56-e030-6e845a8d3e97 + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: added peer, starting replication: peer=server-178aa703-72b6-fd56-e030-6e845a8d3e97 +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: added peer, starting replication: peer=server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: added peer, starting replication: peer=server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: pipelining replication: peer="{Voter server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 931fe7dc-09f5-2a29-e0b1-3e9ac5532433}" + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: pipelining replication: peer="{Voter server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 931fe7dc-09f5-2a29-e0b1-3e9ac5532433}" +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: pipelining replication: peer="{Voter server-178aa703-72b6-fd56-e030-6e845a8d3e97 178aa703-72b6-fd56-e030-6e845a8d3e97}" + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:05.795Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: pipelining replication: peer="{Voter server-178aa703-72b6-fd56-e030-6e845a8d3e97 178aa703-72b6-fd56-e030-6e845a8d3e97}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.900Z [INFO] stable state for Leader reached at 2021-08-25 09:31:05.795240614 +0000 UTC m=+10.061457018 (1 nodes), 55.490998ms from start of poll, 56.560777ms from cluster start. Timeout at 2021-08-25 09:31:05.900372279 +0000 UTC m=+10.166588683, 105.131665ms after stability + testing.go:149: cluster: 2021-08-25T09:31:05.900Z [INFO] stable state for Leader reached at 2021-08-25 09:31:05.795240614 +0000 UTC m=+10.061457018 (1 nodes), 55.490998ms from start of poll, 56.560777ms from cluster start. Timeout at 2021-08-25 09:31:05.900372279 +0000 UTC m=+10.166588683, 105.131665ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:05.900Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:05.900Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.005Z [INFO] stable state for Follower reached at 2021-08-25 09:31:05.900636954 +0000 UTC m=+10.166853338 (2 nodes), 2.74µs from start of poll, 161.957097ms from cluster start. Timeout at 2021-08-25 09:31:06.005687627 +0000 UTC m=+10.271904071, 105.050733ms after stability + testing.go:149: cluster: 2021-08-25T09:31:06.005Z [INFO] stable state for Follower reached at 2021-08-25 09:31:05.900636954 +0000 UTC m=+10.166853338 (2 nodes), 2.74µs from start of poll, 161.957097ms from cluster start. Timeout at 2021-08-25 09:31:06.005687627 +0000 UTC m=+10.271904071, 105.050733ms after stability +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:06.005Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: updating configuration: command=AddStaging server-id=server-178aa703-72b6-fd56-e030-6e845a8d3e97 server-addr=178aa703-72b6-fd56-e030-6e845a8d3e97 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:06.005Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: updating configuration: command=AddStaging server-id=server-178aa703-72b6-fd56-e030-6e845a8d3e97 server-addr=178aa703-72b6-fd56-e030-6e845a8d3e97 servers="[{Suffrage:Voter ID:server-178aa703-72b6-fd56-e030-6e845a8d3e97 Address:178aa703-72b6-fd56-e030-6e845a8d3e97} {Suffrage:Voter ID:server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 Address:931fe7dc-09f5-2a29-e0b1-3e9ac5532433} {Suffrage:Voter ID:server-204c7154-9551-7527-7b60-f402666ad003 Address:204c7154-9551-7527-7b60-f402666ad003}]" +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:06.006Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: aborting pipeline replication: peer="{Voter server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 931fe7dc-09f5-2a29-e0b1-3e9ac5532433}" + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:06.006Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: aborting pipeline replication: peer="{Voter server-931fe7dc-09f5-2a29-e0b1-3e9ac5532433 931fe7dc-09f5-2a29-e0b1-3e9ac5532433}" +testLoggerAdapter verbose: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:06.006Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: aborting pipeline replication: peer="{Voter server-178aa703-72b6-fd56-e030-6e845a8d3e97 178aa703-72b6-fd56-e030-6e845a8d3e97}" + testing.go:149: server-204c7154-9551-7527-7b60-f402666ad003: 2021-08-25T09:31:06.006Z [INFO] server-204c7154-9551-7527-7b60-f402666ad003: aborting pipeline replication: peer="{Voter server-178aa703-72b6-fd56-e030-6e845a8d3e97 178aa703-72b6-fd56-e030-6e845a8d3e97}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer -testLoggerAdapter verbose: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" - testing.go:149: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" -testLoggerAdapter verbose: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: entering follower state: follower="Node at e25a3674-09ad-4651-ba9f-4354641b41fe [Follower]" leader= - testing.go:149: server-e25a3674-09ad-4651-ba9f-4354641b41fe: 2021-08-25T09:28:10.477Z [INFO] server-e25a3674-09ad-4651-ba9f-4354641b41fe: entering follower state: follower="Node at e25a3674-09ad-4651-ba9f-4354641b41fe [Follower]" leader= -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering follower state: follower="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Follower]" leader= - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.477Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering follower state: follower="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Follower]" leader= -testLoggerAdapter verbose: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.477Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" - testing.go:149: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.477Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.478Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.478Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: entering follower state: follower="Node at 0980da88-1e39-d44e-6aa0-522298e67f99 [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:10.478Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-0980da88-1e39-d44e-6aa0-522298e67f99: 2021-08-25T09:28:10.478Z [INFO] server-0980da88-1e39-d44e-6aa0-522298e67f99: entering follower state: follower="Node at 0980da88-1e39-d44e-6aa0-522298e67f99 [Follower]" leader= -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [WARN] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [WARN] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering candidate state: node="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Candidate]" term=2 - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.551Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering candidate state: node="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Candidate]" term=2 -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: election won: tally=2 - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: election won: tally=2 -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering leader state: leader="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Leader]" - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: entering leader state: leader="Node at 2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 [Leader]" -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-e25a3674-09ad-4651-ba9f-4354641b41fe - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-e25a3674-09ad-4651-ba9f-4354641b41fe -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-0980da88-1e39-d44e-6aa0-522298e67f99 - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: added peer, starting replication: peer=server-0980da88-1e39-d44e-6aa0-522298e67f99 -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.552Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: pipelining replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.657Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.552299848 +0000 UTC m=+10.316515737 (1 nodes), 74.098513ms from start of poll, 75.430629ms from cluster start. Timeout at 2021-08-25 09:28:10.657328767 +0000 UTC m=+10.421544736, 105.028999ms after stability - testing.go:149: cluster: 2021-08-25T09:28:10.657Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.552299848 +0000 UTC m=+10.316515737 (1 nodes), 74.098513ms from start of poll, 75.430629ms from cluster start. Timeout at 2021-08-25 09:28:10.657328767 +0000 UTC m=+10.421544736, 105.028999ms after stability -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.657Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: updating configuration: command=RemoveServer server-id=9cb5e7c3-f4fb-cfc9-991c-7296879b73b6 server-addr= servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.657Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: updating configuration: command=RemoveServer server-id=9cb5e7c3-f4fb-cfc9-991c-7296879b73b6 server-addr= servers="[{Suffrage:Voter ID:server-e25a3674-09ad-4651-ba9f-4354641b41fe Address:e25a3674-09ad-4651-ba9f-4354641b41fe} {Suffrage:Voter ID:server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4 Address:2b3b2530-7483-b1f2-759a-5c1ca58e4eb4} {Suffrage:Voter ID:server-0980da88-1e39-d44e-6aa0-522298e67f99 Address:0980da88-1e39-d44e-6aa0-522298e67f99}]" -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-0980da88-1e39-d44e-6aa0-522298e67f99 0980da88-1e39-d44e-6aa0-522298e67f99}" -testLoggerAdapter verbose: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" - testing.go:149: server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: 2021-08-25T09:28:10.658Z [INFO] server-2b3b2530-7483-b1f2-759a-5c1ca58e4eb4: aborting pipeline replication: peer="{Voter server-e25a3674-09ad-4651-ba9f-4354641b41fe e25a3674-09ad-4651-ba9f-4354641b41fe}" +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.010Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.010Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.010Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: entering follower state: follower="Node at 07abca76-14fd-008b-dd28-a98ae32c361b [Follower]" leader= + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.010Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: entering follower state: follower="Node at 07abca76-14fd-008b-dd28-a98ae32c361b [Follower]" leader= +testLoggerAdapter verbose: server-6d78bd77-f691-39d2-7688-e936b6b28d72: 2021-08-25T09:31:06.010Z [INFO] server-6d78bd77-f691-39d2-7688-e936b6b28d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" + testing.go:149: server-6d78bd77-f691-39d2-7688-e936b6b28d72: 2021-08-25T09:31:06.010Z [INFO] server-6d78bd77-f691-39d2-7688-e936b6b28d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" +testLoggerAdapter verbose: server-6d78bd77-f691-39d2-7688-e936b6b28d72: 2021-08-25T09:31:06.010Z [INFO] server-6d78bd77-f691-39d2-7688-e936b6b28d72: entering follower state: follower="Node at 6d78bd77-f691-39d2-7688-e936b6b28d72 [Follower]" leader= + testing.go:149: server-6d78bd77-f691-39d2-7688-e936b6b28d72: 2021-08-25T09:31:06.010Z [INFO] server-6d78bd77-f691-39d2-7688-e936b6b28d72: entering follower state: follower="Node at 6d78bd77-f691-39d2-7688-e936b6b28d72 [Follower]" leader= +testLoggerAdapter verbose: server-fcacfdc3-c29b-a79d-d601-3553c17c5863: 2021-08-25T09:31:06.010Z [INFO] server-fcacfdc3-c29b-a79d-d601-3553c17c5863: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" + testing.go:149: server-fcacfdc3-c29b-a79d-d601-3553c17c5863: 2021-08-25T09:31:06.010Z [INFO] server-fcacfdc3-c29b-a79d-d601-3553c17c5863: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.011Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:06.011Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-fcacfdc3-c29b-a79d-d601-3553c17c5863: 2021-08-25T09:31:06.011Z [INFO] server-fcacfdc3-c29b-a79d-d601-3553c17c5863: entering follower state: follower="Node at fcacfdc3-c29b-a79d-d601-3553c17c5863 [Follower]" leader= + testing.go:149: server-fcacfdc3-c29b-a79d-d601-3553c17c5863: 2021-08-25T09:31:06.011Z [INFO] server-fcacfdc3-c29b-a79d-d601-3553c17c5863: entering follower state: follower="Node at fcacfdc3-c29b-a79d-d601-3553c17c5863 [Follower]" leader= +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [WARN] server-07abca76-14fd-008b-dd28-a98ae32c361b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [WARN] server-07abca76-14fd-008b-dd28-a98ae32c361b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: entering candidate state: node="Node at 07abca76-14fd-008b-dd28-a98ae32c361b [Candidate]" term=2 + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: entering candidate state: node="Node at 07abca76-14fd-008b-dd28-a98ae32c361b [Candidate]" term=2 +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: election won: tally=2 + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: election won: tally=2 +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: entering leader state: leader="Node at 07abca76-14fd-008b-dd28-a98ae32c361b [Leader]" + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.075Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: entering leader state: leader="Node at 07abca76-14fd-008b-dd28-a98ae32c361b [Leader]" +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: added peer, starting replication: peer=server-6d78bd77-f691-39d2-7688-e936b6b28d72 + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: added peer, starting replication: peer=server-6d78bd77-f691-39d2-7688-e936b6b28d72 +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: added peer, starting replication: peer=server-fcacfdc3-c29b-a79d-d601-3553c17c5863 + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: added peer, starting replication: peer=server-fcacfdc3-c29b-a79d-d601-3553c17c5863 +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: pipelining replication: peer="{Voter server-6d78bd77-f691-39d2-7688-e936b6b28d72 6d78bd77-f691-39d2-7688-e936b6b28d72}" + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: pipelining replication: peer="{Voter server-6d78bd77-f691-39d2-7688-e936b6b28d72 6d78bd77-f691-39d2-7688-e936b6b28d72}" +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: pipelining replication: peer="{Voter server-fcacfdc3-c29b-a79d-d601-3553c17c5863 fcacfdc3-c29b-a79d-d601-3553c17c5863}" + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.076Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: pipelining replication: peer="{Voter server-fcacfdc3-c29b-a79d-d601-3553c17c5863 fcacfdc3-c29b-a79d-d601-3553c17c5863}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.181Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.076126157 +0000 UTC m=+10.342342561 (1 nodes), 65.042078ms from start of poll, 66.117857ms from cluster start. Timeout at 2021-08-25 09:31:06.181216026 +0000 UTC m=+10.447432430, 105.089869ms after stability + testing.go:149: cluster: 2021-08-25T09:31:06.181Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.076126157 +0000 UTC m=+10.342342561 (1 nodes), 65.042078ms from start of poll, 66.117857ms from cluster start. Timeout at 2021-08-25 09:31:06.181216026 +0000 UTC m=+10.447432430, 105.089869ms after stability +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.181Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: updating configuration: command=RemoveServer server-id=e0344d06-3ea4-1a5d-8d2d-1dcc88b0af24 server-addr= servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.181Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: updating configuration: command=RemoveServer server-id=e0344d06-3ea4-1a5d-8d2d-1dcc88b0af24 server-addr= servers="[{Suffrage:Voter ID:server-07abca76-14fd-008b-dd28-a98ae32c361b Address:07abca76-14fd-008b-dd28-a98ae32c361b} {Suffrage:Voter ID:server-6d78bd77-f691-39d2-7688-e936b6b28d72 Address:6d78bd77-f691-39d2-7688-e936b6b28d72} {Suffrage:Voter ID:server-fcacfdc3-c29b-a79d-d601-3553c17c5863 Address:fcacfdc3-c29b-a79d-d601-3553c17c5863}]" +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.182Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: aborting pipeline replication: peer="{Voter server-fcacfdc3-c29b-a79d-d601-3553c17c5863 fcacfdc3-c29b-a79d-d601-3553c17c5863}" + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.182Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: aborting pipeline replication: peer="{Voter server-fcacfdc3-c29b-a79d-d601-3553c17c5863 fcacfdc3-c29b-a79d-d601-3553c17c5863}" +testLoggerAdapter verbose: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.182Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: aborting pipeline replication: peer="{Voter server-6d78bd77-f691-39d2-7688-e936b6b28d72 6d78bd77-f691-39d2-7688-e936b6b28d72}" + testing.go:149: server-07abca76-14fd-008b-dd28-a98ae32c361b: 2021-08-25T09:31:06.182Z [INFO] server-07abca76-14fd-008b-dd28-a98ae32c361b: aborting pipeline replication: peer="{Voter server-6d78bd77-f691-39d2-7688-e936b6b28d72 6d78bd77-f691-39d2-7688-e936b6b28d72}" --- PASS: TestRaft_RemoveUnknownPeer (0.18s) === RUN TestRaft_SnapshotRestore -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.660Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:10.660Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering follower state: follower="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Follower]" leader= - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.660Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering follower state: follower="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Follower]" leader= -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [WARN] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [WARN] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering candidate state: node="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Candidate]" term=2 - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering candidate state: node="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Candidate]" term=2 -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: election won: tally=1 - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: election won: tally=1 -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering leader state: leader="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Leader]" - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.718Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: entering leader state: leader="Node at c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 [Leader]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.823Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.718598352 +0000 UTC m=+10.482814261 (1 nodes), 57.789209ms from start of poll, 58.163874ms from cluster start. Timeout at 2021-08-25 09:28:10.823727302 +0000 UTC m=+10.587943331, 105.12907ms after stability - testing.go:149: cluster: 2021-08-25T09:28:10.823Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.718598352 +0000 UTC m=+10.482814261 (1 nodes), 57.789209ms from start of poll, 58.163874ms from cluster start. Timeout at 2021-08-25 09:28:10.823727302 +0000 UTC m=+10.587943331, 105.12907ms after stability -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.828Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: starting snapshot up to: index=102 - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.828Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: starting snapshot up to: index=102 - testing.go:153: 2021-08-25T09:28:10.828Z [INFO] creating new snapshot: path=/tmp/raft457250717/snapshots/2-102-1629883690828.tmp -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.846Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: compacting logs: from=1 to=92 - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.846Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: snapshot complete up to: index=102 - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: restored from snapshot: id=2-102-1629883690828 - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.847Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: restored from snapshot: id=2-102-1629883690828 -testLoggerAdapter verbose: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.848Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" - testing.go:149: server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: 2021-08-25T09:28:10.848Z [INFO] server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c543544b-52e7-7e6d-5e5d-2c31bea1c4e3 Address:c543544b-52e7-7e6d-5e5d-2c31bea1c4e3}]" ---- PASS: TestRaft_SnapshotRestore (0.19s) +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.186Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eee6c360-137e-37a5-657f-f0032536f79e Address:eee6c360-137e-37a5-657f-f0032536f79e}]" + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.186Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eee6c360-137e-37a5-657f-f0032536f79e Address:eee6c360-137e-37a5-657f-f0032536f79e}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.186Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:06.186Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.186Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: entering follower state: follower="Node at eee6c360-137e-37a5-657f-f0032536f79e [Follower]" leader= + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.186Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: entering follower state: follower="Node at eee6c360-137e-37a5-657f-f0032536f79e [Follower]" leader= +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.267Z [WARN] server-eee6c360-137e-37a5-657f-f0032536f79e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.267Z [WARN] server-eee6c360-137e-37a5-657f-f0032536f79e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.267Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: entering candidate state: node="Node at eee6c360-137e-37a5-657f-f0032536f79e [Candidate]" term=2 + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.267Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: entering candidate state: node="Node at eee6c360-137e-37a5-657f-f0032536f79e [Candidate]" term=2 +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.268Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: election won: tally=1 + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.268Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: election won: tally=1 +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.268Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: entering leader state: leader="Node at eee6c360-137e-37a5-657f-f0032536f79e [Leader]" + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.268Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: entering leader state: leader="Node at eee6c360-137e-37a5-657f-f0032536f79e [Leader]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.373Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.268324468 +0000 UTC m=+10.534540912 (1 nodes), 82.02212ms from start of poll, 82.517134ms from cluster start. Timeout at 2021-08-25 09:31:06.373460193 +0000 UTC m=+10.639676697, 105.135785ms after stability + testing.go:149: cluster: 2021-08-25T09:31:06.373Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.268324468 +0000 UTC m=+10.534540912 (1 nodes), 82.02212ms from start of poll, 82.517134ms from cluster start. Timeout at 2021-08-25 09:31:06.373460193 +0000 UTC m=+10.639676697, 105.135785ms after stability +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.376Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: starting snapshot up to: index=102 + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.376Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: starting snapshot up to: index=102 + testing.go:153: 2021-08-25T09:31:06.376Z [INFO] creating new snapshot: path=/tmp/raft490931991/snapshots/2-102-1629883866376.tmp +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.401Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: compacting logs: from=1 to=92 + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.401Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.401Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: snapshot complete up to: index=102 + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.401Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.402Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: restored from snapshot: id=2-102-1629883866376 + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.402Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: restored from snapshot: id=2-102-1629883866376 +testLoggerAdapter verbose: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.402Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eee6c360-137e-37a5-657f-f0032536f79e Address:eee6c360-137e-37a5-657f-f0032536f79e}]" + testing.go:149: server-eee6c360-137e-37a5-657f-f0032536f79e: 2021-08-25T09:31:06.402Z [INFO] server-eee6c360-137e-37a5-657f-f0032536f79e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eee6c360-137e-37a5-657f-f0032536f79e Address:eee6c360-137e-37a5-657f-f0032536f79e}]" +--- PASS: TestRaft_SnapshotRestore (0.22s) === RUN TestRaft_NoRestoreOnStart -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.849Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.849Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:10.849Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:10.849Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.850Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering follower state: follower="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Follower]" leader= - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.850Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering follower state: follower="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Follower]" leader= -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [WARN] server-6af65f82-5092-4322-d891-3af4a67edd70: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [WARN] server-6af65f82-5092-4322-d891-3af4a67edd70: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering candidate state: node="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Candidate]" term=2 - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering candidate state: node="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Candidate]" term=2 -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: election won: tally=1 - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: election won: tally=1 -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering leader state: leader="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Leader]" - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:10.941Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering leader state: leader="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Leader]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.047Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.942114839 +0000 UTC m=+10.706330748 (1 nodes), 92.057145ms from start of poll, 92.485225ms from cluster start. Timeout at 2021-08-25 09:28:11.047146717 +0000 UTC m=+10.811362666, 105.031918ms after stability - testing.go:149: cluster: 2021-08-25T09:28:11.047Z [INFO] stable state for Leader reached at 2021-08-25 09:28:10.942114839 +0000 UTC m=+10.706330748 (1 nodes), 92.057145ms from start of poll, 92.485225ms from cluster start. Timeout at 2021-08-25 09:28:11.047146717 +0000 UTC m=+10.811362666, 105.031918ms after stability -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.049Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: starting snapshot up to: index=102 - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.049Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: starting snapshot up to: index=102 - testing.go:153: 2021-08-25T09:28:11.049Z [INFO] creating new snapshot: path=/tmp/raft819490775/snapshots/2-102-1629883691049.tmp -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: compacting logs: from=1 to=92 - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: snapshot complete up to: index=102 - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" - testing.go:149: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.076Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af65f82-5092-4322-d891-3af4a67edd70 Address:6af65f82-5092-4322-d891-3af4a67edd70}]" ---- PASS: TestRaft_NoRestoreOnStart (0.23s) +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.404Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-243ae5f3-caef-0397-a1cb-b04b55875af8 Address:243ae5f3-caef-0397-a1cb-b04b55875af8}]" + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.404Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-243ae5f3-caef-0397-a1cb-b04b55875af8 Address:243ae5f3-caef-0397-a1cb-b04b55875af8}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.404Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:06.404Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.404Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering follower state: follower="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Follower]" leader= + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.404Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering follower state: follower="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Follower]" leader= +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.462Z [WARN] server-243ae5f3-caef-0397-a1cb-b04b55875af8: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.462Z [WARN] server-243ae5f3-caef-0397-a1cb-b04b55875af8: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.462Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering candidate state: node="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Candidate]" term=2 + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.462Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering candidate state: node="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Candidate]" term=2 +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.463Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: election won: tally=1 + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.463Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: election won: tally=1 +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.463Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering leader state: leader="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Leader]" + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.463Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering leader state: leader="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Leader]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.568Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.463134373 +0000 UTC m=+10.729350797 (1 nodes), 58.430541ms from start of poll, 58.86158ms from cluster start. Timeout at 2021-08-25 09:31:06.568286196 +0000 UTC m=+10.834502620, 105.151823ms after stability + testing.go:149: cluster: 2021-08-25T09:31:06.568Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.463134373 +0000 UTC m=+10.729350797 (1 nodes), 58.430541ms from start of poll, 58.86158ms from cluster start. Timeout at 2021-08-25 09:31:06.568286196 +0000 UTC m=+10.834502620, 105.151823ms after stability +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.570Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: starting snapshot up to: index=102 + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.570Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: starting snapshot up to: index=102 + testing.go:153: 2021-08-25T09:31:06.570Z [INFO] creating new snapshot: path=/tmp/raft294211937/snapshots/2-102-1629883866570.tmp +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.592Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: compacting logs: from=1 to=92 + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.592Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.592Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: snapshot complete up to: index=102 + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.592Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.592Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-243ae5f3-caef-0397-a1cb-b04b55875af8 Address:243ae5f3-caef-0397-a1cb-b04b55875af8}]" + testing.go:149: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.592Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-243ae5f3-caef-0397-a1cb-b04b55875af8 Address:243ae5f3-caef-0397-a1cb-b04b55875af8}]" +--- PASS: TestRaft_NoRestoreOnStart (0.19s) === RUN TestRaft_SnapshotRestore_PeerChange +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.592Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering follower state: follower="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Follower]" leader= raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.077Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering follower state: follower="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Follower]" leader= -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c2ba2ed-a461-73c3-906c-9e95d6b47114 Address:6c2ba2ed-a461-73c3-906c-9e95d6b47114}]" - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c2ba2ed-a461-73c3-906c-9e95d6b47114 Address:6c2ba2ed-a461-73c3-906c-9e95d6b47114}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.078Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:11.078Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering follower state: follower="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Follower]" leader= - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.078Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering follower state: follower="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Follower]" leader= -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.145Z [WARN] server-6af65f82-5092-4322-d891-3af4a67edd70: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.146Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering candidate state: node="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Candidate]" term=3 -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.146Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: election won: tally=1 -testLoggerAdapter verbose: server-6af65f82-5092-4322-d891-3af4a67edd70: 2021-08-25T09:28:11.146Z [INFO] server-6af65f82-5092-4322-d891-3af4a67edd70: entering leader state: leader="Node at 6af65f82-5092-4322-d891-3af4a67edd70 [Leader]" -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [WARN] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [WARN] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering candidate state: node="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Candidate]" term=2 - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering candidate state: node="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Candidate]" term=2 -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: election won: tally=1 - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: election won: tally=1 -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering leader state: leader="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Leader]" - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.148Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: entering leader state: leader="Node at 6c2ba2ed-a461-73c3-906c-9e95d6b47114 [Leader]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.254Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.148721677 +0000 UTC m=+10.912937566 (1 nodes), 70.164119ms from start of poll, 70.70003ms from cluster start. Timeout at 2021-08-25 09:28:11.253905261 +0000 UTC m=+11.018121230, 105.183664ms after stability - testing.go:149: cluster: 2021-08-25T09:28:11.254Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.148721677 +0000 UTC m=+10.912937566 (1 nodes), 70.164119ms from start of poll, 70.70003ms from cluster start. Timeout at 2021-08-25 09:28:11.253905261 +0000 UTC m=+11.018121230, 105.183664ms after stability -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [ERROR] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [ERROR] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: starting snapshot up to: index=12 - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.254Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: starting snapshot up to: index=12 - testing.go:153: 2021-08-25T09:28:11.254Z [INFO] creating new snapshot: path=/tmp/raft077058593/snapshots/2-12-1629883691254.tmp -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: compacting logs: from=1 to=2 - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: compacting logs: from=1 to=2 -testLoggerAdapter verbose: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: snapshot complete up to: index=12 - testing.go:149: server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: 2021-08-25T09:28:11.272Z [INFO] server-6c2ba2ed-a461-73c3-906c-9e95d6b47114: snapshot complete up to: index=12 ---- PASS: TestRaft_UserSnapshot (0.20s) +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.594Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 Address:8cae2db2-e399-b4b5-a1db-4e5a7ce86f73}]" + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.594Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 Address:8cae2db2-e399-b4b5-a1db-4e5a7ce86f73}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.594Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:06.594Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.594Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: entering follower state: follower="Node at 8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 [Follower]" leader= + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.594Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: entering follower state: follower="Node at 8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 [Follower]" leader= +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [WARN] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [WARN] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: entering candidate state: node="Node at 8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 [Candidate]" term=2 + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: entering candidate state: node="Node at 8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 [Candidate]" term=2 +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: election won: tally=1 + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: election won: tally=1 +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: entering leader state: leader="Node at 8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 [Leader]" + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.687Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: entering leader state: leader="Node at 8cae2db2-e399-b4b5-a1db-4e5a7ce86f73 [Leader]" +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.691Z [WARN] server-243ae5f3-caef-0397-a1cb-b04b55875af8: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.691Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering candidate state: node="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Candidate]" term=3 +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.691Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: election won: tally=1 +testLoggerAdapter verbose: server-243ae5f3-caef-0397-a1cb-b04b55875af8: 2021-08-25T09:31:06.691Z [INFO] server-243ae5f3-caef-0397-a1cb-b04b55875af8: entering leader state: leader="Node at 243ae5f3-caef-0397-a1cb-b04b55875af8 [Leader]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.792Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.687588727 +0000 UTC m=+10.953805151 (1 nodes), 93.380711ms from start of poll, 93.81589ms from cluster start. Timeout at 2021-08-25 09:31:06.792759409 +0000 UTC m=+11.058975913, 105.170762ms after stability + testing.go:149: cluster: 2021-08-25T09:31:06.792Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.687588727 +0000 UTC m=+10.953805151 (1 nodes), 93.380711ms from start of poll, 93.81589ms from cluster start. Timeout at 2021-08-25 09:31:06.792759409 +0000 UTC m=+11.058975913, 105.170762ms after stability +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.793Z [ERROR] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.793Z [ERROR] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.793Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: starting snapshot up to: index=12 + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.793Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: starting snapshot up to: index=12 + testing.go:153: 2021-08-25T09:31:06.793Z [INFO] creating new snapshot: path=/tmp/raft795130427/snapshots/2-12-1629883866793.tmp +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.818Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: compacting logs: from=1 to=2 + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.818Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: compacting logs: from=1 to=2 +testLoggerAdapter verbose: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.818Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: snapshot complete up to: index=12 + testing.go:149: server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: 2021-08-25T09:31:06.818Z [INFO] server-8cae2db2-e399-b4b5-a1db-4e5a7ce86f73: snapshot complete up to: index=12 +--- PASS: TestRaft_UserSnapshot (0.23s) === RUN TestRaft_UserRestore -testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" - testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" -testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: entering follower state: follower="Node at 70513784-006e-26f1-77a2-2e72c552214b [Follower]" leader= - testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.275Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: entering follower state: follower="Node at 70513784-006e-26f1-77a2-2e72c552214b [Follower]" leader= -testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.275Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" - testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.275Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" -testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.276Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: entering follower state: follower="Node at ca0d195f-5484-93ff-72b6-e56b017a5c13 [Follower]" leader= - testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.276Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: entering follower state: follower="Node at ca0d195f-5484-93ff-72b6-e56b017a5c13 [Follower]" leader= -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70513784-006e-26f1-77a2-2e72c552214b Address:70513784-006e-26f1-77a2-2e72c552214b} {Suffrage:Voter ID:server-ca0d195f-5484-93ff-72b6-e56b017a5c13 Address:ca0d195f-5484-93ff-72b6-e56b017a5c13} {Suffrage:Voter ID:server-637c2c1c-77d3-88aa-5912-743728615735 Address:637c2c1c-77d3-88aa-5912-743728615735}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.276Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:11.276Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering follower state: follower="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Follower]" leader= - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.276Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering follower state: follower="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Follower]" leader= -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [WARN] server-637c2c1c-77d3-88aa-5912-743728615735: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [WARN] server-637c2c1c-77d3-88aa-5912-743728615735: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering candidate state: node="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Candidate]" term=2 - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering candidate state: node="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Candidate]" term=2 -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: election won: tally=2 - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: election won: tally=2 -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering leader state: leader="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Leader]" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: entering leader state: leader="Node at 637c2c1c-77d3-88aa-5912-743728615735 [Leader]" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-70513784-006e-26f1-77a2-2e72c552214b - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.328Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-70513784-006e-26f1-77a2-2e72c552214b -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-ca0d195f-5484-93ff-72b6-e56b017a5c13 - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: added peer, starting replication: peer=server-ca0d195f-5484-93ff-72b6-e56b017a5c13 -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.329Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.329090293 +0000 UTC m=+11.093306182 (1 nodes), 52.562635ms from start of poll, 53.819078ms from cluster start. Timeout at 2021-08-25 09:28:11.434119592 +0000 UTC m=+11.198335561, 105.029379ms after stability - testing.go:149: cluster: 2021-08-25T09:28:11.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.329090293 +0000 UTC m=+11.093306182 (1 nodes), 52.562635ms from start of poll, 53.819078ms from cluster start. Timeout at 2021-08-25 09:28:11.434119592 +0000 UTC m=+11.198335561, 105.029379ms after stability -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.434Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: starting snapshot up to: index=12 - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.434Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: starting snapshot up to: index=12 - testing.go:153: 2021-08-25T09:28:11.434Z [INFO] creating new snapshot: path=/tmp/raft415421535/snapshots/2-12-1629883691434.tmp -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.453Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: snapshot complete up to: index=12 - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.453Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: snapshot complete up to: index=12 - testing.go:153: 2021-08-25T09:28:11.454Z [INFO] creating new snapshot: path=/tmp/raft415421535/snapshots/2-23-1629883691454.tmp -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: copied to local snapshot: bytes=71 - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: restored user snapshot: index=1 - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: restored user snapshot: index=1 -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.472Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.473Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.479Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.479Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:11.480Z [INFO] creating new snapshot: path=/tmp/raft144830203/snapshots/2-23-1629883691479.tmp -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.482Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.482Z [ERROR] server-637c2c1c-77d3-88aa-5912-743728615735: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:11.483Z [INFO] creating new snapshot: path=/tmp/raft803796197/snapshots/2-23-1629883691483.tmp -testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.508Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: copied to local snapshot: bytes=71 - testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.508Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.509Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: Installed remote snapshot - testing.go:149: server-70513784-006e-26f1-77a2-2e72c552214b: 2021-08-25T09:28:11.509Z [INFO] server-70513784-006e-26f1-77a2-2e72c552214b: Installed remote snapshot -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.509Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.509Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" -testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: copied to local snapshot: bytes=71 - testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: Installed remote snapshot - testing.go:149: server-ca0d195f-5484-93ff-72b6-e56b017a5c13: 2021-08-25T09:28:11.512Z [INFO] server-ca0d195f-5484-93ff-72b6-e56b017a5c13: Installed remote snapshot -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.513Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.513Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: pipelining replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-70513784-006e-26f1-77a2-2e72c552214b 70513784-006e-26f1-77a2-2e72c552214b}" -testLoggerAdapter verbose: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" - testing.go:149: server-637c2c1c-77d3-88aa-5912-743728615735: 2021-08-25T09:28:11.526Z [INFO] server-637c2c1c-77d3-88aa-5912-743728615735: aborting pipeline replication: peer="{Voter server-ca0d195f-5484-93ff-72b6-e56b017a5c13 ca0d195f-5484-93ff-72b6-e56b017a5c13}" -testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.530Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" - testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.530Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" -testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.531Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: entering follower state: follower="Node at f49ced75-b740-fc85-917e-df19e9bc02f4 [Follower]" leader= - testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.531Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: entering follower state: follower="Node at f49ced75-b740-fc85-917e-df19e9bc02f4 [Follower]" leader= -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering follower state: follower="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Follower]" leader= - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.531Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering follower state: follower="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Follower]" leader= -testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" - testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f49ced75-b740-fc85-917e-df19e9bc02f4 Address:f49ced75-b740-fc85-917e-df19e9bc02f4} {Suffrage:Voter ID:server-567bc5cd-c47d-652e-4739-b45ffa36f0d2 Address:567bc5cd-c47d-652e-4739-b45ffa36f0d2} {Suffrage:Voter ID:server-28b1b5fb-9683-9abb-066a-ab7c417f8edc Address:28b1b5fb-9683-9abb-066a-ab7c417f8edc}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.531Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:11.531Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: entering follower state: follower="Node at 28b1b5fb-9683-9abb-066a-ab7c417f8edc [Follower]" leader= - testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.531Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: entering follower state: follower="Node at 28b1b5fb-9683-9abb-066a-ab7c417f8edc [Follower]" leader= -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [WARN] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [WARN] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering candidate state: node="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Candidate]" term=2 - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering candidate state: node="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Candidate]" term=2 -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: election won: tally=2 - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: election won: tally=2 -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering leader state: leader="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Leader]" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: entering leader state: leader="Node at 567bc5cd-c47d-652e-4739-b45ffa36f0d2 [Leader]" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-f49ced75-b740-fc85-917e-df19e9bc02f4 - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-f49ced75-b740-fc85-917e-df19e9bc02f4 -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-28b1b5fb-9683-9abb-066a-ab7c417f8edc - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: added peer, starting replication: peer=server-28b1b5fb-9683-9abb-066a-ab7c417f8edc -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.606Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.607Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.607Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.712Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.606893934 +0000 UTC m=+11.371109823 (1 nodes), 75.161714ms from start of poll, 76.211277ms from cluster start. Timeout at 2021-08-25 09:28:11.712006065 +0000 UTC m=+11.476222034, 105.112211ms after stability - testing.go:149: cluster: 2021-08-25T09:28:11.712Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.606893934 +0000 UTC m=+11.371109823 (1 nodes), 75.161714ms from start of poll, 76.211277ms from cluster start. Timeout at 2021-08-25 09:28:11.712006065 +0000 UTC m=+11.476222034, 105.112211ms after stability -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.712Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: starting snapshot up to: index=12 - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.712Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: starting snapshot up to: index=12 - testing.go:153: 2021-08-25T09:28:11.712Z [INFO] creating new snapshot: path=/tmp/raft920409603/snapshots/2-12-1629883691712.tmp -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.733Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: snapshot complete up to: index=12 - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.733Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: snapshot complete up to: index=12 - testing.go:153: 2021-08-25T09:28:11.734Z [INFO] creating new snapshot: path=/tmp/raft920409603/snapshots/2-23-1629883691734.tmp -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: copied to local snapshot: bytes=71 - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: restored user snapshot: index=1 - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.751Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: restored user snapshot: index=1 -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.752Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.757Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.757Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:11.758Z [INFO] creating new snapshot: path=/tmp/raft718466025/snapshots/2-23-1629883691758.tmp -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.764Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.764Z [ERROR] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:11.765Z [INFO] creating new snapshot: path=/tmp/raft277194541/snapshots/2-23-1629883691765.tmp -testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: copied to local snapshot: bytes=71 - testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: Installed remote snapshot - testing.go:149: server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: 2021-08-25T09:28:11.784Z [INFO] server-28b1b5fb-9683-9abb-066a-ab7c417f8edc: Installed remote snapshot -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.784Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.784Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" -testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: copied to local snapshot: bytes=71 - testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: Installed remote snapshot - testing.go:149: server-f49ced75-b740-fc85-917e-df19e9bc02f4: 2021-08-25T09:28:11.785Z [INFO] server-f49ced75-b740-fc85-917e-df19e9bc02f4: Installed remote snapshot -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.785Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.785Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: pipelining replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-28b1b5fb-9683-9abb-066a-ab7c417f8edc 28b1b5fb-9683-9abb-066a-ab7c417f8edc}" -testLoggerAdapter verbose: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" - testing.go:149: server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: 2021-08-25T09:28:11.801Z [INFO] server-567bc5cd-c47d-652e-4739-b45ffa36f0d2: aborting pipeline replication: peer="{Voter server-f49ced75-b740-fc85-917e-df19e9bc02f4 f49ced75-b740-fc85-917e-df19e9bc02f4}" -testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.805Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" - testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.805Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.805Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.805Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" -testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.806Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: entering follower state: follower="Node at af544a6b-77d8-2d43-79d5-72c50e5769a4 [Follower]" leader= -testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" - testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:11.806Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: entering follower state: follower="Node at af544a6b-77d8-2d43-79d5-72c50e5769a4 [Follower]" leader= -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.806Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering follower state: follower="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Follower]" leader= - testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af544a6b-77d8-2d43-79d5-72c50e5769a4 Address:af544a6b-77d8-2d43-79d5-72c50e5769a4} {Suffrage:Voter ID:server-35dd04bf-32c2-aa68-0421-516c39020c2d Address:35dd04bf-32c2-aa68-0421-516c39020c2d} {Suffrage:Voter ID:server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc Address:f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.806Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:11.806Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.806Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering follower state: follower="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Follower]" leader= -testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: entering follower state: follower="Node at f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc [Follower]" leader= - testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:11.806Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: entering follower state: follower="Node at f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc [Follower]" leader= -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [WARN] server-35dd04bf-32c2-aa68-0421-516c39020c2d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [WARN] server-35dd04bf-32c2-aa68-0421-516c39020c2d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering candidate state: node="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Candidate]" term=2 - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering candidate state: node="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Candidate]" term=2 -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: election won: tally=2 - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: election won: tally=2 -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering leader state: leader="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Leader]" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: entering leader state: leader="Node at 35dd04bf-32c2-aa68-0421-516c39020c2d [Leader]" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-af544a6b-77d8-2d43-79d5-72c50e5769a4 - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.871Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-af544a6b-77d8-2d43-79d5-72c50e5769a4 -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: added peer, starting replication: peer=server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.872Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:11.977Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.872157961 +0000 UTC m=+11.636373850 (1 nodes), 65.657658ms from start of poll, 66.778714ms from cluster start. Timeout at 2021-08-25 09:28:11.977166021 +0000 UTC m=+11.741381990, 105.00814ms after stability - testing.go:149: cluster: 2021-08-25T09:28:11.977Z [INFO] stable state for Leader reached at 2021-08-25 09:28:11.872157961 +0000 UTC m=+11.636373850 (1 nodes), 65.657658ms from start of poll, 66.778714ms from cluster start. Timeout at 2021-08-25 09:28:11.977166021 +0000 UTC m=+11.741381990, 105.00814ms after stability -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.977Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: starting snapshot up to: index=12 - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.977Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: starting snapshot up to: index=12 - testing.go:153: 2021-08-25T09:28:11.978Z [INFO] creating new snapshot: path=/tmp/raft104314033/snapshots/2-12-1629883691978.tmp -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.997Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: snapshot complete up to: index=12 - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:11.997Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: snapshot complete up to: index=12 - testing.go:153: 2021-08-25T09:28:11.998Z [INFO] creating new snapshot: path=/tmp/raft104314033/snapshots/2-23-1629883691998.tmp -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.022Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: copied to local snapshot: bytes=71 - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.022Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: restored user snapshot: index=1 - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: restored user snapshot: index=1 -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.023Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.024Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.024Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.030Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.030Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.031Z [INFO] creating new snapshot: path=/tmp/raft372438503/snapshots/2-23-1629883692031.tmp -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.033Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.033Z [ERROR] server-35dd04bf-32c2-aa68-0421-516c39020c2d: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.034Z [INFO] creating new snapshot: path=/tmp/raft077980171/snapshots/2-23-1629883692034.tmp -testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: copied to local snapshot: bytes=71 - testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: Installed remote snapshot - testing.go:149: server-af544a6b-77d8-2d43-79d5-72c50e5769a4: 2021-08-25T09:28:12.061Z [INFO] server-af544a6b-77d8-2d43-79d5-72c50e5769a4: Installed remote snapshot -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.062Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.062Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" -testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.065Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: copied to local snapshot: bytes=71 - testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.065Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.066Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: Installed remote snapshot - testing.go:149: server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: 2021-08-25T09:28:12.066Z [INFO] server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc: Installed remote snapshot -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.066Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.066Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: pipelining replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc f2d391fb-5f87-e8f0-9d7c-cdbbe10167fc}" -testLoggerAdapter verbose: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" - testing.go:149: server-35dd04bf-32c2-aa68-0421-516c39020c2d: 2021-08-25T09:28:12.083Z [INFO] server-35dd04bf-32c2-aa68-0421-516c39020c2d: aborting pipeline replication: peer="{Voter server-af544a6b-77d8-2d43-79d5-72c50e5769a4 af544a6b-77d8-2d43-79d5-72c50e5769a4}" -testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" - testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" -testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: entering follower state: follower="Node at 1b092822-0ccd-384f-79f0-bd4d6ddb280c [Follower]" leader= - testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.089Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: entering follower state: follower="Node at 1b092822-0ccd-384f-79f0-bd4d6ddb280c [Follower]" leader= -testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" - testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.089Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.089Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b092822-0ccd-384f-79f0-bd4d6ddb280c Address:1b092822-0ccd-384f-79f0-bd4d6ddb280c} {Suffrage:Voter ID:server-fec26d77-9d21-8ed3-d991-338d6313021e Address:fec26d77-9d21-8ed3-d991-338d6313021e} {Suffrage:Voter ID:server-f8ea6912-c680-0e36-82fb-4ced5504d5ea Address:f8ea6912-c680-0e36-82fb-4ced5504d5ea}]" -testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: entering follower state: follower="Node at fec26d77-9d21-8ed3-d991-338d6313021e [Follower]" leader= -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.090Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.089Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: entering follower state: follower="Node at fec26d77-9d21-8ed3-d991-338d6313021e [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:12.090Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.090Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering follower state: follower="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Follower]" leader= - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.090Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering follower state: follower="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Follower]" leader= -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [WARN] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [WARN] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering candidate state: node="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Candidate]" term=2 - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.145Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering candidate state: node="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Candidate]" term=2 -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: election won: tally=2 - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: election won: tally=2 -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering leader state: leader="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Leader]" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: entering leader state: leader="Node at f8ea6912-c680-0e36-82fb-4ced5504d5ea [Leader]" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-1b092822-0ccd-384f-79f0-bd4d6ddb280c - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-1b092822-0ccd-384f-79f0-bd4d6ddb280c -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-fec26d77-9d21-8ed3-d991-338d6313021e - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: added peer, starting replication: peer=server-fec26d77-9d21-8ed3-d991-338d6313021e -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.146Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.252Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.146928903 +0000 UTC m=+11.911144792 (1 nodes), 56.773983ms from start of poll, 58.113558ms from cluster start. Timeout at 2021-08-25 09:28:12.252042954 +0000 UTC m=+12.016258843, 105.114051ms after stability - testing.go:149: cluster: 2021-08-25T09:28:12.252Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.146928903 +0000 UTC m=+11.911144792 (1 nodes), 56.773983ms from start of poll, 58.113558ms from cluster start. Timeout at 2021-08-25 09:28:12.252042954 +0000 UTC m=+12.016258843, 105.114051ms after stability -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.252Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: starting snapshot up to: index=12 - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.252Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: starting snapshot up to: index=12 - testing.go:153: 2021-08-25T09:28:12.252Z [INFO] creating new snapshot: path=/tmp/raft294703737/snapshots/2-12-1629883692252.tmp -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.271Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: snapshot complete up to: index=12 - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.271Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: snapshot complete up to: index=12 - testing.go:153: 2021-08-25T09:28:12.272Z [INFO] creating new snapshot: path=/tmp/raft294703737/snapshots/2-113-1629883692272.tmp -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.290Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: copied to local snapshot: bytes=71 - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.290Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: restored user snapshot: index=1 - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: restored user snapshot: index=1 -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.291Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.296Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.296Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.297Z [INFO] creating new snapshot: path=/tmp/raft104115317/snapshots/2-113-1629883692297.tmp -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.299Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.299Z [ERROR] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.300Z [INFO] creating new snapshot: path=/tmp/raft134093935/snapshots/2-113-1629883692300.tmp -testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: copied to local snapshot: bytes=71 - testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: Installed remote snapshot - testing.go:149: server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: 2021-08-25T09:28:12.324Z [INFO] server-1b092822-0ccd-384f-79f0-bd4d6ddb280c: Installed remote snapshot -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.325Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.325Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" -testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: copied to local snapshot: bytes=71 - testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: Installed remote snapshot - testing.go:149: server-fec26d77-9d21-8ed3-d991-338d6313021e: 2021-08-25T09:28:12.327Z [INFO] server-fec26d77-9d21-8ed3-d991-338d6313021e: Installed remote snapshot -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.328Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.328Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: pipelining replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-1b092822-0ccd-384f-79f0-bd4d6ddb280c 1b092822-0ccd-384f-79f0-bd4d6ddb280c}" -testLoggerAdapter verbose: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" - testing.go:149: server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: 2021-08-25T09:28:12.341Z [INFO] server-f8ea6912-c680-0e36-82fb-4ced5504d5ea: aborting pipeline replication: peer="{Voter server-fec26d77-9d21-8ed3-d991-338d6313021e fec26d77-9d21-8ed3-d991-338d6313021e}" -testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" - testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" -testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: entering follower state: follower="Node at 867ef441-c97f-bde7-5cb1-206ad54eff9b [Follower]" leader= - testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.346Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: entering follower state: follower="Node at 867ef441-c97f-bde7-5cb1-206ad54eff9b [Follower]" leader= -testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" - testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" -testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: entering follower state: follower="Node at ffe8e498-96ff-07ce-8d4f-378576ebda6c [Follower]" leader= - testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.346Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: entering follower state: follower="Node at ffe8e498-96ff-07ce-8d4f-378576ebda6c [Follower]" leader= -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.346Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.346Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867ef441-c97f-bde7-5cb1-206ad54eff9b Address:867ef441-c97f-bde7-5cb1-206ad54eff9b} {Suffrage:Voter ID:server-ffe8e498-96ff-07ce-8d4f-378576ebda6c Address:ffe8e498-96ff-07ce-8d4f-378576ebda6c} {Suffrage:Voter ID:server-a5f12987-1429-775d-a68f-b7499072dd13 Address:a5f12987-1429-775d-a68f-b7499072dd13}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.347Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:12.347Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.347Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering follower state: follower="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Follower]" leader= - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.347Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering follower state: follower="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Follower]" leader= -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [WARN] server-a5f12987-1429-775d-a68f-b7499072dd13: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [WARN] server-a5f12987-1429-775d-a68f-b7499072dd13: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering candidate state: node="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Candidate]" term=2 - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering candidate state: node="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Candidate]" term=2 -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: election won: tally=2 - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.400Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: election won: tally=2 -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering leader state: leader="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Leader]" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: entering leader state: leader="Node at a5f12987-1429-775d-a68f-b7499072dd13 [Leader]" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-867ef441-c97f-bde7-5cb1-206ad54eff9b - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-867ef441-c97f-bde7-5cb1-206ad54eff9b -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-ffe8e498-96ff-07ce-8d4f-378576ebda6c - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: added peer, starting replication: peer=server-ffe8e498-96ff-07ce-8d4f-378576ebda6c -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.401Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.506Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.40121429 +0000 UTC m=+12.165430239 (1 nodes), 54.059755ms from start of poll, 55.270123ms from cluster start. Timeout at 2021-08-25 09:28:12.506354139 +0000 UTC m=+12.270570408, 105.140169ms after stability - testing.go:149: cluster: 2021-08-25T09:28:12.506Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.40121429 +0000 UTC m=+12.165430239 (1 nodes), 54.059755ms from start of poll, 55.270123ms from cluster start. Timeout at 2021-08-25 09:28:12.506354139 +0000 UTC m=+12.270570408, 105.140169ms after stability -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.507Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: starting snapshot up to: index=12 - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.507Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: starting snapshot up to: index=12 - testing.go:153: 2021-08-25T09:28:12.507Z [INFO] creating new snapshot: path=/tmp/raft493305847/snapshots/2-12-1629883692507.tmp -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.524Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: snapshot complete up to: index=12 - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.524Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: snapshot complete up to: index=12 - testing.go:153: 2021-08-25T09:28:12.525Z [INFO] creating new snapshot: path=/tmp/raft493305847/snapshots/2-1013-1629883692525.tmp -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: copied to local snapshot: bytes=71 - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: restored user snapshot: index=1 - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.542Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: restored user snapshot: index=1 -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.543Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.548Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.548Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.549Z [INFO] creating new snapshot: path=/tmp/raft136100029/snapshots/2-1013-1629883692549.tmp -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.553Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.553Z [ERROR] server-a5f12987-1429-775d-a68f-b7499072dd13: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.554Z [INFO] creating new snapshot: path=/tmp/raft573978899/snapshots/2-1013-1629883692554.tmp -testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.576Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: copied to local snapshot: bytes=71 - testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.576Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.577Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: Installed remote snapshot - testing.go:149: server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: 2021-08-25T09:28:12.577Z [INFO] server-ffe8e498-96ff-07ce-8d4f-378576ebda6c: Installed remote snapshot -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.577Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.577Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" -testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: copied to local snapshot: bytes=71 - testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: Installed remote snapshot - testing.go:149: server-867ef441-c97f-bde7-5cb1-206ad54eff9b: 2021-08-25T09:28:12.581Z [INFO] server-867ef441-c97f-bde7-5cb1-206ad54eff9b: Installed remote snapshot -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.581Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.581Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: pipelining replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-ffe8e498-96ff-07ce-8d4f-378576ebda6c ffe8e498-96ff-07ce-8d4f-378576ebda6c}" -testLoggerAdapter verbose: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" - testing.go:149: server-a5f12987-1429-775d-a68f-b7499072dd13: 2021-08-25T09:28:12.593Z [INFO] server-a5f12987-1429-775d-a68f-b7499072dd13: aborting pipeline replication: peer="{Voter server-867ef441-c97f-bde7-5cb1-206ad54eff9b 867ef441-c97f-bde7-5cb1-206ad54eff9b}" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering follower state: follower="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Follower]" leader= - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.600Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering follower state: follower="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Follower]" leader= -testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" - testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" -testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: entering follower state: follower="Node at e461fd51-53cd-29dc-4854-9240d5f8d499 [Follower]" leader= - testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.601Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: entering follower state: follower="Node at e461fd51-53cd-29dc-4854-9240d5f8d499 [Follower]" leader= -testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" - testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29387dcd-909d-fc54-1781-d0ef41cb21f7 Address:29387dcd-909d-fc54-1781-d0ef41cb21f7} {Suffrage:Voter ID:server-e461fd51-53cd-29dc-4854-9240d5f8d499 Address:e461fd51-53cd-29dc-4854-9240d5f8d499} {Suffrage:Voter ID:server-794ef44c-b813-251f-1c50-a891b0f9e742 Address:794ef44c-b813-251f-1c50-a891b0f9e742}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.601Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:12.601Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: entering follower state: follower="Node at 794ef44c-b813-251f-1c50-a891b0f9e742 [Follower]" leader= - testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.601Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: entering follower state: follower="Node at 794ef44c-b813-251f-1c50-a891b0f9e742 [Follower]" leader= -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.662Z [WARN] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.662Z [WARN] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering candidate state: node="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Candidate]" term=2 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering candidate state: node="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Candidate]" term=2 -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: election won: tally=2 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: election won: tally=2 -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering leader state: leader="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Leader]" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: entering leader state: leader="Node at 29387dcd-909d-fc54-1781-d0ef41cb21f7 [Leader]" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-e461fd51-53cd-29dc-4854-9240d5f8d499 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-e461fd51-53cd-29dc-4854-9240d5f8d499 -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-794ef44c-b813-251f-1c50-a891b0f9e742 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: added peer, starting replication: peer=server-794ef44c-b813-251f-1c50-a891b0f9e742 -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.663Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.664Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.664Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.768Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.663517232 +0000 UTC m=+12.427733161 (1 nodes), 61.800656ms from start of poll, 63.153891ms from cluster start. Timeout at 2021-08-25 09:28:12.768538691 +0000 UTC m=+12.532754660, 105.021499ms after stability - testing.go:149: cluster: 2021-08-25T09:28:12.768Z [INFO] stable state for Leader reached at 2021-08-25 09:28:12.663517232 +0000 UTC m=+12.427733161 (1 nodes), 61.800656ms from start of poll, 63.153891ms from cluster start. Timeout at 2021-08-25 09:28:12.768538691 +0000 UTC m=+12.532754660, 105.021499ms after stability -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.769Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: starting snapshot up to: index=12 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.769Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: starting snapshot up to: index=12 - testing.go:153: 2021-08-25T09:28:12.769Z [INFO] creating new snapshot: path=/tmp/raft608534849/snapshots/2-12-1629883692769.tmp -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.788Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: snapshot complete up to: index=12 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.788Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: snapshot complete up to: index=12 - testing.go:153: 2021-08-25T09:28:12.789Z [INFO] creating new snapshot: path=/tmp/raft608534849/snapshots/2-10013-1629883692789.tmp -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.805Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: copied to local snapshot: bytes=71 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.805Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: restored user snapshot: index=1 - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: restored user snapshot: index=1 -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.806Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.814Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.814Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.815Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.815Z [INFO] creating new snapshot: path=/tmp/raft320300549/snapshots/2-10013-1629883692815.tmp - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.815Z [ERROR] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: failed to get log: index=23 error="log not found" - testing.go:153: 2021-08-25T09:28:12.815Z [INFO] creating new snapshot: path=/tmp/raft356716315/snapshots/2-10013-1629883692815.tmp -testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.842Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: copied to local snapshot: bytes=71 - testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.842Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.843Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: Installed remote snapshot - testing.go:149: server-794ef44c-b813-251f-1c50-a891b0f9e742: 2021-08-25T09:28:12.843Z [INFO] server-794ef44c-b813-251f-1c50-a891b0f9e742: Installed remote snapshot -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.843Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.843Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" -testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.845Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: copied to local snapshot: bytes=71 - testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.845Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.846Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: Installed remote snapshot - testing.go:149: server-e461fd51-53cd-29dc-4854-9240d5f8d499: 2021-08-25T09:28:12.846Z [INFO] server-e461fd51-53cd-29dc-4854-9240d5f8d499: Installed remote snapshot -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.846Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.846Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: pipelining replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-794ef44c-b813-251f-1c50-a891b0f9e742 794ef44c-b813-251f-1c50-a891b0f9e742}" -testLoggerAdapter verbose: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" - testing.go:149: server-29387dcd-909d-fc54-1781-d0ef41cb21f7: 2021-08-25T09:28:12.860Z [INFO] server-29387dcd-909d-fc54-1781-d0ef41cb21f7: aborting pipeline replication: peer="{Voter server-e461fd51-53cd-29dc-4854-9240d5f8d499 e461fd51-53cd-29dc-4854-9240d5f8d499}" ---- PASS: TestRaft_UserRestore (1.59s) +testLoggerAdapter verbose: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:06.821Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f Address:d3508785-46c1-1a48-0b5b-cc5b0c34e53f} {Suffrage:Voter ID:server-aae7d584-79fe-9436-96ff-a3ba5b503a79 Address:aae7d584-79fe-9436-96ff-a3ba5b503a79} {Suffrage:Voter ID:server-55987921-1594-d891-edaa-d257d630bb5e Address:55987921-1594-d891-edaa-d257d630bb5e}]" + testing.go:149: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:06.821Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f Address:d3508785-46c1-1a48-0b5b-cc5b0c34e53f} {Suffrage:Voter ID:server-aae7d584-79fe-9436-96ff-a3ba5b503a79 Address:aae7d584-79fe-9436-96ff-a3ba5b503a79} {Suffrage:Voter ID:server-55987921-1594-d891-edaa-d257d630bb5e Address:55987921-1594-d891-edaa-d257d630bb5e}]" +testLoggerAdapter verbose: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:06.822Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: entering follower state: follower="Node at d3508785-46c1-1a48-0b5b-cc5b0c34e53f [Follower]" leader= +testLoggerAdapter verbose: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:06.822Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f Address:d3508785-46c1-1a48-0b5b-cc5b0c34e53f} {Suffrage:Voter ID:server-aae7d584-79fe-9436-96ff-a3ba5b503a79 Address:aae7d584-79fe-9436-96ff-a3ba5b503a79} {Suffrage:Voter ID:server-55987921-1594-d891-edaa-d257d630bb5e Address:55987921-1594-d891-edaa-d257d630bb5e}]" + testing.go:149: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:06.822Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: entering follower state: follower="Node at d3508785-46c1-1a48-0b5b-cc5b0c34e53f [Follower]" leader= + testing.go:149: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:06.822Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f Address:d3508785-46c1-1a48-0b5b-cc5b0c34e53f} {Suffrage:Voter ID:server-aae7d584-79fe-9436-96ff-a3ba5b503a79 Address:aae7d584-79fe-9436-96ff-a3ba5b503a79} {Suffrage:Voter ID:server-55987921-1594-d891-edaa-d257d630bb5e Address:55987921-1594-d891-edaa-d257d630bb5e}]" +testLoggerAdapter verbose: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:06.822Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: entering follower state: follower="Node at aae7d584-79fe-9436-96ff-a3ba5b503a79 [Follower]" leader= + testing.go:149: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:06.822Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: entering follower state: follower="Node at aae7d584-79fe-9436-96ff-a3ba5b503a79 [Follower]" leader= +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.822Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f Address:d3508785-46c1-1a48-0b5b-cc5b0c34e53f} {Suffrage:Voter ID:server-aae7d584-79fe-9436-96ff-a3ba5b503a79 Address:aae7d584-79fe-9436-96ff-a3ba5b503a79} {Suffrage:Voter ID:server-55987921-1594-d891-edaa-d257d630bb5e Address:55987921-1594-d891-edaa-d257d630bb5e}]" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.822Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f Address:d3508785-46c1-1a48-0b5b-cc5b0c34e53f} {Suffrage:Voter ID:server-aae7d584-79fe-9436-96ff-a3ba5b503a79 Address:aae7d584-79fe-9436-96ff-a3ba5b503a79} {Suffrage:Voter ID:server-55987921-1594-d891-edaa-d257d630bb5e Address:55987921-1594-d891-edaa-d257d630bb5e}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.822Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:06.822Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.822Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: entering follower state: follower="Node at 55987921-1594-d891-edaa-d257d630bb5e [Follower]" leader= + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.822Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: entering follower state: follower="Node at 55987921-1594-d891-edaa-d257d630bb5e [Follower]" leader= +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [WARN] server-55987921-1594-d891-edaa-d257d630bb5e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [WARN] server-55987921-1594-d891-edaa-d257d630bb5e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: entering candidate state: node="Node at 55987921-1594-d891-edaa-d257d630bb5e [Candidate]" term=2 + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: entering candidate state: node="Node at 55987921-1594-d891-edaa-d257d630bb5e [Candidate]" term=2 +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: election won: tally=2 + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: election won: tally=2 +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: entering leader state: leader="Node at 55987921-1594-d891-edaa-d257d630bb5e [Leader]" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: entering leader state: leader="Node at 55987921-1594-d891-edaa-d257d630bb5e [Leader]" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: added peer, starting replication: peer=server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: added peer, starting replication: peer=server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: added peer, starting replication: peer=server-aae7d584-79fe-9436-96ff-a3ba5b503a79 + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: added peer, starting replication: peer=server-aae7d584-79fe-9436-96ff-a3ba5b503a79 +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.877Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.878Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.878Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:06.983Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.877802025 +0000 UTC m=+11.144018449 (1 nodes), 54.92649ms from start of poll, 56.173073ms from cluster start. Timeout at 2021-08-25 09:31:06.982952128 +0000 UTC m=+11.249168572, 105.150123ms after stability + testing.go:149: cluster: 2021-08-25T09:31:06.983Z [INFO] stable state for Leader reached at 2021-08-25 09:31:06.877802025 +0000 UTC m=+11.144018449 (1 nodes), 54.92649ms from start of poll, 56.173073ms from cluster start. Timeout at 2021-08-25 09:31:06.982952128 +0000 UTC m=+11.249168572, 105.150123ms after stability +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.983Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: starting snapshot up to: index=12 + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:06.983Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: starting snapshot up to: index=12 + testing.go:153: 2021-08-25T09:31:06.983Z [INFO] creating new snapshot: path=/tmp/raft865700137/snapshots/2-12-1629883866983.tmp +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.011Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: snapshot complete up to: index=12 + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.011Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: snapshot complete up to: index=12 + testing.go:153: 2021-08-25T09:31:07.012Z [INFO] creating new snapshot: path=/tmp/raft865700137/snapshots/2-23-1629883867012.tmp +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.034Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: copied to local snapshot: bytes=71 + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.034Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.034Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: restored user snapshot: index=1 + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.034Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: restored user snapshot: index=1 +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.035Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.042Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.042Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:07.042Z [INFO] creating new snapshot: path=/tmp/raft257921573/snapshots/2-23-1629883867042.tmp +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.044Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.044Z [ERROR] server-55987921-1594-d891-edaa-d257d630bb5e: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:07.045Z [INFO] creating new snapshot: path=/tmp/raft986466719/snapshots/2-23-1629883867045.tmp +testLoggerAdapter verbose: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:07.078Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: copied to local snapshot: bytes=71 + testing.go:149: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:07.078Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:07.078Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: Installed remote snapshot + testing.go:149: server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: 2021-08-25T09:31:07.078Z [INFO] server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f: Installed remote snapshot +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.079Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.079Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" +testLoggerAdapter verbose: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:07.084Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: copied to local snapshot: bytes=71 + testing.go:149: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:07.084Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:07.084Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: Installed remote snapshot + testing.go:149: server-aae7d584-79fe-9436-96ff-a3ba5b503a79: 2021-08-25T09:31:07.084Z [INFO] server-aae7d584-79fe-9436-96ff-a3ba5b503a79: Installed remote snapshot +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.084Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.084Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: pipelining replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.100Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.100Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-aae7d584-79fe-9436-96ff-a3ba5b503a79 aae7d584-79fe-9436-96ff-a3ba5b503a79}" +testLoggerAdapter verbose: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.100Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" + testing.go:149: server-55987921-1594-d891-edaa-d257d630bb5e: 2021-08-25T09:31:07.100Z [INFO] server-55987921-1594-d891-edaa-d257d630bb5e: aborting pipeline replication: peer="{Voter server-d3508785-46c1-1a48-0b5b-cc5b0c34e53f d3508785-46c1-1a48-0b5b-cc5b0c34e53f}" +testLoggerAdapter verbose: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.106Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 Address:99e8c11c-5748-dc3d-1ec8-5917b84ca858} {Suffrage:Voter ID:server-d716f667-3c96-a766-bed3-a5ff266b731c Address:d716f667-3c96-a766-bed3-a5ff266b731c} {Suffrage:Voter ID:server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6 Address:8adc0d77-e42d-0b5a-d11a-6d08377a70d6}]" + testing.go:149: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.106Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 Address:99e8c11c-5748-dc3d-1ec8-5917b84ca858} {Suffrage:Voter ID:server-d716f667-3c96-a766-bed3-a5ff266b731c Address:d716f667-3c96-a766-bed3-a5ff266b731c} {Suffrage:Voter ID:server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6 Address:8adc0d77-e42d-0b5a-d11a-6d08377a70d6}]" +testLoggerAdapter verbose: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.106Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: entering follower state: follower="Node at 99e8c11c-5748-dc3d-1ec8-5917b84ca858 [Follower]" leader= + testing.go:149: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.106Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: entering follower state: follower="Node at 99e8c11c-5748-dc3d-1ec8-5917b84ca858 [Follower]" leader= +testLoggerAdapter verbose: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.106Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 Address:99e8c11c-5748-dc3d-1ec8-5917b84ca858} {Suffrage:Voter ID:server-d716f667-3c96-a766-bed3-a5ff266b731c Address:d716f667-3c96-a766-bed3-a5ff266b731c} {Suffrage:Voter ID:server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6 Address:8adc0d77-e42d-0b5a-d11a-6d08377a70d6}]" + testing.go:149: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.106Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 Address:99e8c11c-5748-dc3d-1ec8-5917b84ca858} {Suffrage:Voter ID:server-d716f667-3c96-a766-bed3-a5ff266b731c Address:d716f667-3c96-a766-bed3-a5ff266b731c} {Suffrage:Voter ID:server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6 Address:8adc0d77-e42d-0b5a-d11a-6d08377a70d6}]" +testLoggerAdapter verbose: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.107Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: entering follower state: follower="Node at d716f667-3c96-a766-bed3-a5ff266b731c [Follower]" leader= + testing.go:149: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.107Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: entering follower state: follower="Node at d716f667-3c96-a766-bed3-a5ff266b731c [Follower]" leader= +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.107Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 Address:99e8c11c-5748-dc3d-1ec8-5917b84ca858} {Suffrage:Voter ID:server-d716f667-3c96-a766-bed3-a5ff266b731c Address:d716f667-3c96-a766-bed3-a5ff266b731c} {Suffrage:Voter ID:server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6 Address:8adc0d77-e42d-0b5a-d11a-6d08377a70d6}]" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.107Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 Address:99e8c11c-5748-dc3d-1ec8-5917b84ca858} {Suffrage:Voter ID:server-d716f667-3c96-a766-bed3-a5ff266b731c Address:d716f667-3c96-a766-bed3-a5ff266b731c} {Suffrage:Voter ID:server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6 Address:8adc0d77-e42d-0b5a-d11a-6d08377a70d6}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:07.107Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:07.107Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.107Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: entering follower state: follower="Node at 8adc0d77-e42d-0b5a-d11a-6d08377a70d6 [Follower]" leader= + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.107Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: entering follower state: follower="Node at 8adc0d77-e42d-0b5a-d11a-6d08377a70d6 [Follower]" leader= +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [WARN] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [WARN] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: entering candidate state: node="Node at 8adc0d77-e42d-0b5a-d11a-6d08377a70d6 [Candidate]" term=2 + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: entering candidate state: node="Node at 8adc0d77-e42d-0b5a-d11a-6d08377a70d6 [Candidate]" term=2 +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: election won: tally=2 + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: election won: tally=2 +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: entering leader state: leader="Node at 8adc0d77-e42d-0b5a-d11a-6d08377a70d6 [Leader]" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: entering leader state: leader="Node at 8adc0d77-e42d-0b5a-d11a-6d08377a70d6 [Leader]" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: added peer, starting replication: peer=server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: added peer, starting replication: peer=server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: added peer, starting replication: peer=server-d716f667-3c96-a766-bed3-a5ff266b731c + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.159Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: added peer, starting replication: peer=server-d716f667-3c96-a766-bed3-a5ff266b731c +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.160Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.160Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.160Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.160Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:07.265Z [INFO] stable state for Leader reached at 2021-08-25 09:31:07.160116074 +0000 UTC m=+11.426332478 (1 nodes), 52.621567ms from start of poll, 53.809515ms from cluster start. Timeout at 2021-08-25 09:31:07.265210603 +0000 UTC m=+11.531427027, 105.094549ms after stability + testing.go:149: cluster: 2021-08-25T09:31:07.265Z [INFO] stable state for Leader reached at 2021-08-25 09:31:07.160116074 +0000 UTC m=+11.426332478 (1 nodes), 52.621567ms from start of poll, 53.809515ms from cluster start. Timeout at 2021-08-25 09:31:07.265210603 +0000 UTC m=+11.531427027, 105.094549ms after stability +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.265Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: starting snapshot up to: index=12 + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.265Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: starting snapshot up to: index=12 + testing.go:153: 2021-08-25T09:31:07.266Z [INFO] creating new snapshot: path=/tmp/raft098219303/snapshots/2-12-1629883867266.tmp +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.293Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: snapshot complete up to: index=12 + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.293Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: snapshot complete up to: index=12 + testing.go:153: 2021-08-25T09:31:07.294Z [INFO] creating new snapshot: path=/tmp/raft098219303/snapshots/2-23-1629883867294.tmp +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: copied to local snapshot: bytes=71 + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: restored user snapshot: index=1 + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: restored user snapshot: index=1 +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.316Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.317Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.317Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.323Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.323Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:07.324Z [INFO] creating new snapshot: path=/tmp/raft877753965/snapshots/2-23-1629883867324.tmp +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.325Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.325Z [ERROR] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:07.326Z [INFO] creating new snapshot: path=/tmp/raft990833475/snapshots/2-23-1629883867326.tmp +testLoggerAdapter verbose: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.360Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: copied to local snapshot: bytes=71 + testing.go:149: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.360Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.360Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: Installed remote snapshot + testing.go:149: server-d716f667-3c96-a766-bed3-a5ff266b731c: 2021-08-25T09:31:07.360Z [INFO] server-d716f667-3c96-a766-bed3-a5ff266b731c: Installed remote snapshot +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.361Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.361Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" +testLoggerAdapter verbose: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.365Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: copied to local snapshot: bytes=71 + testing.go:149: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.365Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.367Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: Installed remote snapshot + testing.go:149: server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: 2021-08-25T09:31:07.367Z [INFO] server-99e8c11c-5748-dc3d-1ec8-5917b84ca858: Installed remote snapshot +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.367Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.367Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: pipelining replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.383Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.383Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-d716f667-3c96-a766-bed3-a5ff266b731c d716f667-3c96-a766-bed3-a5ff266b731c}" +testLoggerAdapter verbose: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.383Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" + testing.go:149: server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: 2021-08-25T09:31:07.383Z [INFO] server-8adc0d77-e42d-0b5a-d11a-6d08377a70d6: aborting pipeline replication: peer="{Voter server-99e8c11c-5748-dc3d-1ec8-5917b84ca858 99e8c11c-5748-dc3d-1ec8-5917b84ca858}" +testLoggerAdapter verbose: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.388Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 Address:f2bf917a-575d-0ae4-37bb-36e38d7840a1} {Suffrage:Voter ID:server-b2dbee26-bcdb-f817-96ea-99c621047f4a Address:b2dbee26-bcdb-f817-96ea-99c621047f4a} {Suffrage:Voter ID:server-6d1d8058-5590-ab98-830f-28dd8ccc8100 Address:6d1d8058-5590-ab98-830f-28dd8ccc8100}]" + testing.go:149: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.388Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 Address:f2bf917a-575d-0ae4-37bb-36e38d7840a1} {Suffrage:Voter ID:server-b2dbee26-bcdb-f817-96ea-99c621047f4a Address:b2dbee26-bcdb-f817-96ea-99c621047f4a} {Suffrage:Voter ID:server-6d1d8058-5590-ab98-830f-28dd8ccc8100 Address:6d1d8058-5590-ab98-830f-28dd8ccc8100}]" +testLoggerAdapter verbose: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.388Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: entering follower state: follower="Node at f2bf917a-575d-0ae4-37bb-36e38d7840a1 [Follower]" leader= + testing.go:149: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.388Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: entering follower state: follower="Node at f2bf917a-575d-0ae4-37bb-36e38d7840a1 [Follower]" leader= +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.388Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 Address:f2bf917a-575d-0ae4-37bb-36e38d7840a1} {Suffrage:Voter ID:server-b2dbee26-bcdb-f817-96ea-99c621047f4a Address:b2dbee26-bcdb-f817-96ea-99c621047f4a} {Suffrage:Voter ID:server-6d1d8058-5590-ab98-830f-28dd8ccc8100 Address:6d1d8058-5590-ab98-830f-28dd8ccc8100}]" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.388Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 Address:f2bf917a-575d-0ae4-37bb-36e38d7840a1} {Suffrage:Voter ID:server-b2dbee26-bcdb-f817-96ea-99c621047f4a Address:b2dbee26-bcdb-f817-96ea-99c621047f4a} {Suffrage:Voter ID:server-6d1d8058-5590-ab98-830f-28dd8ccc8100 Address:6d1d8058-5590-ab98-830f-28dd8ccc8100}]" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.389Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: entering follower state: follower="Node at b2dbee26-bcdb-f817-96ea-99c621047f4a [Follower]" leader= + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.389Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: entering follower state: follower="Node at b2dbee26-bcdb-f817-96ea-99c621047f4a [Follower]" leader= +testLoggerAdapter verbose: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.389Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 Address:f2bf917a-575d-0ae4-37bb-36e38d7840a1} {Suffrage:Voter ID:server-b2dbee26-bcdb-f817-96ea-99c621047f4a Address:b2dbee26-bcdb-f817-96ea-99c621047f4a} {Suffrage:Voter ID:server-6d1d8058-5590-ab98-830f-28dd8ccc8100 Address:6d1d8058-5590-ab98-830f-28dd8ccc8100}]" + testing.go:149: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.389Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 Address:f2bf917a-575d-0ae4-37bb-36e38d7840a1} {Suffrage:Voter ID:server-b2dbee26-bcdb-f817-96ea-99c621047f4a Address:b2dbee26-bcdb-f817-96ea-99c621047f4a} {Suffrage:Voter ID:server-6d1d8058-5590-ab98-830f-28dd8ccc8100 Address:6d1d8058-5590-ab98-830f-28dd8ccc8100}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:07.389Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:07.389Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.389Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: entering follower state: follower="Node at 6d1d8058-5590-ab98-830f-28dd8ccc8100 [Follower]" leader= + testing.go:149: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.389Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: entering follower state: follower="Node at 6d1d8058-5590-ab98-830f-28dd8ccc8100 [Follower]" leader= +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [WARN] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [WARN] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: entering candidate state: node="Node at b2dbee26-bcdb-f817-96ea-99c621047f4a [Candidate]" term=2 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: entering candidate state: node="Node at b2dbee26-bcdb-f817-96ea-99c621047f4a [Candidate]" term=2 +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: election won: tally=2 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: election won: tally=2 +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: entering leader state: leader="Node at b2dbee26-bcdb-f817-96ea-99c621047f4a [Leader]" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: entering leader state: leader="Node at b2dbee26-bcdb-f817-96ea-99c621047f4a [Leader]" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: added peer, starting replication: peer=server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: added peer, starting replication: peer=server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: added peer, starting replication: peer=server-6d1d8058-5590-ab98-830f-28dd8ccc8100 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.443Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: added peer, starting replication: peer=server-6d1d8058-5590-ab98-830f-28dd8ccc8100 +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.444Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.444Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.444Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.444Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:07.548Z [INFO] stable state for Leader reached at 2021-08-25 09:31:07.443841091 +0000 UTC m=+11.710057495 (1 nodes), 54.468514ms from start of poll, 55.541093ms from cluster start. Timeout at 2021-08-25 09:31:07.548848148 +0000 UTC m=+11.815064632, 105.007137ms after stability + testing.go:149: cluster: 2021-08-25T09:31:07.548Z [INFO] stable state for Leader reached at 2021-08-25 09:31:07.443841091 +0000 UTC m=+11.710057495 (1 nodes), 54.468514ms from start of poll, 55.541093ms from cluster start. Timeout at 2021-08-25 09:31:07.548848148 +0000 UTC m=+11.815064632, 105.007137ms after stability +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.549Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: starting snapshot up to: index=12 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.549Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: starting snapshot up to: index=12 + testing.go:153: 2021-08-25T09:31:07.549Z [INFO] creating new snapshot: path=/tmp/raft152936267/snapshots/2-12-1629883867549.tmp +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.577Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: snapshot complete up to: index=12 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.577Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: snapshot complete up to: index=12 + testing.go:153: 2021-08-25T09:31:07.578Z [INFO] creating new snapshot: path=/tmp/raft152936267/snapshots/2-23-1629883867578.tmp +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.598Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: copied to local snapshot: bytes=71 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.598Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: restored user snapshot: index=1 + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: restored user snapshot: index=1 +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.599Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.606Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.606Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.606Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.606Z [ERROR] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:07.606Z [INFO] creating new snapshot: path=/tmp/raft255563761/snapshots/2-23-1629883867606.tmp + testing.go:153: 2021-08-25T09:31:07.607Z [INFO] creating new snapshot: path=/tmp/raft765636021/snapshots/2-23-1629883867607.tmp +testLoggerAdapter verbose: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.643Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: copied to local snapshot: bytes=71 + testing.go:149: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.643Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.643Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: Installed remote snapshot + testing.go:149: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.643Z [INFO] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: Installed remote snapshot +testLoggerAdapter verbose: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.644Z [WARN] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: skipping application of old log: index=23 + testing.go:149: server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: 2021-08-25T09:31:07.644Z [WARN] server-f2bf917a-575d-0ae4-37bb-36e38d7840a1: skipping application of old log: index=23 +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.644Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.644Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" +testLoggerAdapter verbose: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.648Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: copied to local snapshot: bytes=71 + testing.go:149: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.648Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.649Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: Installed remote snapshot + testing.go:149: server-6d1d8058-5590-ab98-830f-28dd8ccc8100: 2021-08-25T09:31:07.649Z [INFO] server-6d1d8058-5590-ab98-830f-28dd8ccc8100: Installed remote snapshot +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.649Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.649Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: pipelining replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.655Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.655Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-6d1d8058-5590-ab98-830f-28dd8ccc8100 6d1d8058-5590-ab98-830f-28dd8ccc8100}" +testLoggerAdapter verbose: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.655Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" + testing.go:149: server-b2dbee26-bcdb-f817-96ea-99c621047f4a: 2021-08-25T09:31:07.655Z [INFO] server-b2dbee26-bcdb-f817-96ea-99c621047f4a: aborting pipeline replication: peer="{Voter server-f2bf917a-575d-0ae4-37bb-36e38d7840a1 f2bf917a-575d-0ae4-37bb-36e38d7840a1}" +testLoggerAdapter verbose: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.659Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 Address:5eeecd91-d2d0-75a6-6b5e-0eb50cc34593} {Suffrage:Voter ID:server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 Address:04723eb6-dbe6-d7cb-b272-204e75aebdd9} {Suffrage:Voter ID:server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 Address:05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12}]" + testing.go:149: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.659Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 Address:5eeecd91-d2d0-75a6-6b5e-0eb50cc34593} {Suffrage:Voter ID:server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 Address:04723eb6-dbe6-d7cb-b272-204e75aebdd9} {Suffrage:Voter ID:server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 Address:05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12}]" +testLoggerAdapter verbose: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.660Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: entering follower state: follower="Node at 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 [Follower]" leader= + testing.go:149: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.660Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: entering follower state: follower="Node at 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 [Follower]" leader= +testLoggerAdapter verbose: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.660Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 Address:5eeecd91-d2d0-75a6-6b5e-0eb50cc34593} {Suffrage:Voter ID:server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 Address:04723eb6-dbe6-d7cb-b272-204e75aebdd9} {Suffrage:Voter ID:server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 Address:05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12}]" + testing.go:149: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.660Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 Address:5eeecd91-d2d0-75a6-6b5e-0eb50cc34593} {Suffrage:Voter ID:server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 Address:04723eb6-dbe6-d7cb-b272-204e75aebdd9} {Suffrage:Voter ID:server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 Address:05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12}]" +testLoggerAdapter verbose: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.660Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: entering follower state: follower="Node at 04723eb6-dbe6-d7cb-b272-204e75aebdd9 [Follower]" leader= + testing.go:149: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.660Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: entering follower state: follower="Node at 04723eb6-dbe6-d7cb-b272-204e75aebdd9 [Follower]" leader= +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.660Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 Address:5eeecd91-d2d0-75a6-6b5e-0eb50cc34593} {Suffrage:Voter ID:server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 Address:04723eb6-dbe6-d7cb-b272-204e75aebdd9} {Suffrage:Voter ID:server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 Address:05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12}]" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.660Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 Address:5eeecd91-d2d0-75a6-6b5e-0eb50cc34593} {Suffrage:Voter ID:server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 Address:04723eb6-dbe6-d7cb-b272-204e75aebdd9} {Suffrage:Voter ID:server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 Address:05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:07.660Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:07.660Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.660Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: entering follower state: follower="Node at 05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 [Follower]" leader= + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.660Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: entering follower state: follower="Node at 05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 [Follower]" leader= +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [WARN] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [WARN] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: entering candidate state: node="Node at 05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 [Candidate]" term=2 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: entering candidate state: node="Node at 05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 [Candidate]" term=2 +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: election won: tally=2 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: election won: tally=2 +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: entering leader state: leader="Node at 05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 [Leader]" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: entering leader state: leader="Node at 05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12 [Leader]" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: added peer, starting replication: peer=server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: added peer, starting replication: peer=server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: added peer, starting replication: peer=server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: added peer, starting replication: peer=server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.718Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:07.823Z [INFO] stable state for Leader reached at 2021-08-25 09:31:07.718675144 +0000 UTC m=+11.984891568 (1 nodes), 58.006241ms from start of poll, 58.986308ms from cluster start. Timeout at 2021-08-25 09:31:07.823791891 +0000 UTC m=+12.090008295, 105.116727ms after stability + testing.go:149: cluster: 2021-08-25T09:31:07.823Z [INFO] stable state for Leader reached at 2021-08-25 09:31:07.718675144 +0000 UTC m=+11.984891568 (1 nodes), 58.006241ms from start of poll, 58.986308ms from cluster start. Timeout at 2021-08-25 09:31:07.823791891 +0000 UTC m=+12.090008295, 105.116727ms after stability +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.824Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: starting snapshot up to: index=12 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.824Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: starting snapshot up to: index=12 + testing.go:153: 2021-08-25T09:31:07.824Z [INFO] creating new snapshot: path=/tmp/raft248361555/snapshots/2-12-1629883867824.tmp +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.851Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: snapshot complete up to: index=12 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.851Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: snapshot complete up to: index=12 + testing.go:153: 2021-08-25T09:31:07.852Z [INFO] creating new snapshot: path=/tmp/raft248361555/snapshots/2-113-1629883867851.tmp +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.872Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: copied to local snapshot: bytes=71 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.872Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: restored user snapshot: index=1 + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: restored user snapshot: index=1 +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.873Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.879Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.879Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:07.880Z [INFO] creating new snapshot: path=/tmp/raft549052847/snapshots/2-113-1629883867880.tmp +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.881Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.881Z [ERROR] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:07.882Z [INFO] creating new snapshot: path=/tmp/raft328772537/snapshots/2-113-1629883867882.tmp +testLoggerAdapter verbose: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.918Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: copied to local snapshot: bytes=71 + testing.go:149: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.918Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.918Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: Installed remote snapshot + testing.go:149: server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: 2021-08-25T09:31:07.918Z [INFO] server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593: Installed remote snapshot +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.919Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.919Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" +testLoggerAdapter verbose: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.924Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: copied to local snapshot: bytes=71 + testing.go:149: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.924Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.924Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: Installed remote snapshot + testing.go:149: server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: 2021-08-25T09:31:07.924Z [INFO] server-04723eb6-dbe6-d7cb-b272-204e75aebdd9: Installed remote snapshot +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.924Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.924Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: pipelining replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.940Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.940Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-04723eb6-dbe6-d7cb-b272-204e75aebdd9 04723eb6-dbe6-d7cb-b272-204e75aebdd9}" +testLoggerAdapter verbose: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.940Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" + testing.go:149: server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: 2021-08-25T09:31:07.940Z [INFO] server-05a2a5ac-a42a-e5b1-b0c0-aec28ec89e12: aborting pipeline replication: peer="{Voter server-5eeecd91-d2d0-75a6-6b5e-0eb50cc34593 5eeecd91-d2d0-75a6-6b5e-0eb50cc34593}" +testLoggerAdapter verbose: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:07.945Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b8d2ed-ee98-699a-0849-2e2d2e99593c Address:20b8d2ed-ee98-699a-0849-2e2d2e99593c} {Suffrage:Voter ID:server-d6a10a2d-8aca-f825-8c38-b610fca47393 Address:d6a10a2d-8aca-f825-8c38-b610fca47393} {Suffrage:Voter ID:server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640 Address:cd97b6fe-27e4-7d70-faf2-cfaa191ae640}]" + testing.go:149: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:07.945Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b8d2ed-ee98-699a-0849-2e2d2e99593c Address:20b8d2ed-ee98-699a-0849-2e2d2e99593c} {Suffrage:Voter ID:server-d6a10a2d-8aca-f825-8c38-b610fca47393 Address:d6a10a2d-8aca-f825-8c38-b610fca47393} {Suffrage:Voter ID:server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640 Address:cd97b6fe-27e4-7d70-faf2-cfaa191ae640}]" +testLoggerAdapter verbose: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:07.946Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: entering follower state: follower="Node at 20b8d2ed-ee98-699a-0849-2e2d2e99593c [Follower]" leader= + testing.go:149: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:07.946Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: entering follower state: follower="Node at 20b8d2ed-ee98-699a-0849-2e2d2e99593c [Follower]" leader= +testLoggerAdapter verbose: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:07.946Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b8d2ed-ee98-699a-0849-2e2d2e99593c Address:20b8d2ed-ee98-699a-0849-2e2d2e99593c} {Suffrage:Voter ID:server-d6a10a2d-8aca-f825-8c38-b610fca47393 Address:d6a10a2d-8aca-f825-8c38-b610fca47393} {Suffrage:Voter ID:server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640 Address:cd97b6fe-27e4-7d70-faf2-cfaa191ae640}]" + testing.go:149: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:07.946Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b8d2ed-ee98-699a-0849-2e2d2e99593c Address:20b8d2ed-ee98-699a-0849-2e2d2e99593c} {Suffrage:Voter ID:server-d6a10a2d-8aca-f825-8c38-b610fca47393 Address:d6a10a2d-8aca-f825-8c38-b610fca47393} {Suffrage:Voter ID:server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640 Address:cd97b6fe-27e4-7d70-faf2-cfaa191ae640}]" +testLoggerAdapter verbose: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:07.946Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: entering follower state: follower="Node at d6a10a2d-8aca-f825-8c38-b610fca47393 [Follower]" leader= + testing.go:149: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:07.946Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: entering follower state: follower="Node at d6a10a2d-8aca-f825-8c38-b610fca47393 [Follower]" leader= +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:07.946Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b8d2ed-ee98-699a-0849-2e2d2e99593c Address:20b8d2ed-ee98-699a-0849-2e2d2e99593c} {Suffrage:Voter ID:server-d6a10a2d-8aca-f825-8c38-b610fca47393 Address:d6a10a2d-8aca-f825-8c38-b610fca47393} {Suffrage:Voter ID:server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640 Address:cd97b6fe-27e4-7d70-faf2-cfaa191ae640}]" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:07.946Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b8d2ed-ee98-699a-0849-2e2d2e99593c Address:20b8d2ed-ee98-699a-0849-2e2d2e99593c} {Suffrage:Voter ID:server-d6a10a2d-8aca-f825-8c38-b610fca47393 Address:d6a10a2d-8aca-f825-8c38-b610fca47393} {Suffrage:Voter ID:server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640 Address:cd97b6fe-27e4-7d70-faf2-cfaa191ae640}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:07.946Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:07.946Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:07.946Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: entering follower state: follower="Node at cd97b6fe-27e4-7d70-faf2-cfaa191ae640 [Follower]" leader= + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:07.946Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: entering follower state: follower="Node at cd97b6fe-27e4-7d70-faf2-cfaa191ae640 [Follower]" leader= +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [WARN] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [WARN] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: entering candidate state: node="Node at cd97b6fe-27e4-7d70-faf2-cfaa191ae640 [Candidate]" term=2 + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: entering candidate state: node="Node at cd97b6fe-27e4-7d70-faf2-cfaa191ae640 [Candidate]" term=2 +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: election won: tally=2 + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: election won: tally=2 +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: entering leader state: leader="Node at cd97b6fe-27e4-7d70-faf2-cfaa191ae640 [Leader]" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.036Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: entering leader state: leader="Node at cd97b6fe-27e4-7d70-faf2-cfaa191ae640 [Leader]" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: added peer, starting replication: peer=server-20b8d2ed-ee98-699a-0849-2e2d2e99593c + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: added peer, starting replication: peer=server-20b8d2ed-ee98-699a-0849-2e2d2e99593c +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: added peer, starting replication: peer=server-d6a10a2d-8aca-f825-8c38-b610fca47393 + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: added peer, starting replication: peer=server-d6a10a2d-8aca-f825-8c38-b610fca47393 +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.037Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.142Z [INFO] stable state for Leader reached at 2021-08-25 09:31:08.037062418 +0000 UTC m=+12.303278822 (1 nodes), 90.334438ms from start of poll, 91.532505ms from cluster start. Timeout at 2021-08-25 09:31:08.142202743 +0000 UTC m=+12.408419267, 105.140445ms after stability + testing.go:149: cluster: 2021-08-25T09:31:08.142Z [INFO] stable state for Leader reached at 2021-08-25 09:31:08.037062418 +0000 UTC m=+12.303278822 (1 nodes), 90.334438ms from start of poll, 91.532505ms from cluster start. Timeout at 2021-08-25 09:31:08.142202743 +0000 UTC m=+12.408419267, 105.140445ms after stability +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.142Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: starting snapshot up to: index=12 + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.142Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: starting snapshot up to: index=12 + testing.go:153: 2021-08-25T09:31:08.143Z [INFO] creating new snapshot: path=/tmp/raft237964417/snapshots/2-12-1629883868143.tmp +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.170Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: snapshot complete up to: index=12 + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.170Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: snapshot complete up to: index=12 + testing.go:153: 2021-08-25T09:31:08.171Z [INFO] creating new snapshot: path=/tmp/raft237964417/snapshots/2-1013-1629883868171.tmp +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.199Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: copied to local snapshot: bytes=71 + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.199Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.199Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: restored user snapshot: index=1 + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.199Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: restored user snapshot: index=1 +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.200Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.208Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.208Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.209Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.209Z [ERROR] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:08.211Z [INFO] creating new snapshot: path=/tmp/raft987653629/snapshots/2-1013-1629883868211.tmp + testing.go:153: 2021-08-25T09:31:08.211Z [INFO] creating new snapshot: path=/tmp/raft557900599/snapshots/2-1013-1629883868211.tmp +testLoggerAdapter verbose: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:08.239Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:08.239Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: copied to local snapshot: bytes=71 + testing.go:149: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:08.239Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: copied to local snapshot: bytes=71 + testing.go:149: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:08.239Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:08.239Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: Installed remote snapshot + testing.go:149: server-d6a10a2d-8aca-f825-8c38-b610fca47393: 2021-08-25T09:31:08.239Z [INFO] server-d6a10a2d-8aca-f825-8c38-b610fca47393: Installed remote snapshot +testLoggerAdapter verbose: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:08.239Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: Installed remote snapshot + testing.go:149: server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: 2021-08-25T09:31:08.239Z [INFO] server-20b8d2ed-ee98-699a-0849-2e2d2e99593c: Installed remote snapshot +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.239Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.239Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.239Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.239Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: pipelining replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.250Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.250Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-d6a10a2d-8aca-f825-8c38-b610fca47393 d6a10a2d-8aca-f825-8c38-b610fca47393}" +testLoggerAdapter verbose: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.250Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" + testing.go:149: server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: 2021-08-25T09:31:08.250Z [INFO] server-cd97b6fe-27e4-7d70-faf2-cfaa191ae640: aborting pipeline replication: peer="{Voter server-20b8d2ed-ee98-699a-0849-2e2d2e99593c 20b8d2ed-ee98-699a-0849-2e2d2e99593c}" +testLoggerAdapter verbose: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.255Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5622461-0627-5f6c-fe99-a4ba5a632a91 Address:d5622461-0627-5f6c-fe99-a4ba5a632a91} {Suffrage:Voter ID:server-d586e47d-e0a6-65fe-9392-2370d6d507e2 Address:d586e47d-e0a6-65fe-9392-2370d6d507e2} {Suffrage:Voter ID:server-c817f651-f295-15fe-623e-659e32c18c2c Address:c817f651-f295-15fe-623e-659e32c18c2c}]" + testing.go:149: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.255Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5622461-0627-5f6c-fe99-a4ba5a632a91 Address:d5622461-0627-5f6c-fe99-a4ba5a632a91} {Suffrage:Voter ID:server-d586e47d-e0a6-65fe-9392-2370d6d507e2 Address:d586e47d-e0a6-65fe-9392-2370d6d507e2} {Suffrage:Voter ID:server-c817f651-f295-15fe-623e-659e32c18c2c Address:c817f651-f295-15fe-623e-659e32c18c2c}]" +testLoggerAdapter verbose: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.255Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: entering follower state: follower="Node at d5622461-0627-5f6c-fe99-a4ba5a632a91 [Follower]" leader= + testing.go:149: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.255Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: entering follower state: follower="Node at d5622461-0627-5f6c-fe99-a4ba5a632a91 [Follower]" leader= +testLoggerAdapter verbose: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.256Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5622461-0627-5f6c-fe99-a4ba5a632a91 Address:d5622461-0627-5f6c-fe99-a4ba5a632a91} {Suffrage:Voter ID:server-d586e47d-e0a6-65fe-9392-2370d6d507e2 Address:d586e47d-e0a6-65fe-9392-2370d6d507e2} {Suffrage:Voter ID:server-c817f651-f295-15fe-623e-659e32c18c2c Address:c817f651-f295-15fe-623e-659e32c18c2c}]" + testing.go:149: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.256Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5622461-0627-5f6c-fe99-a4ba5a632a91 Address:d5622461-0627-5f6c-fe99-a4ba5a632a91} {Suffrage:Voter ID:server-d586e47d-e0a6-65fe-9392-2370d6d507e2 Address:d586e47d-e0a6-65fe-9392-2370d6d507e2} {Suffrage:Voter ID:server-c817f651-f295-15fe-623e-659e32c18c2c Address:c817f651-f295-15fe-623e-659e32c18c2c}]" +testLoggerAdapter verbose: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.256Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: entering follower state: follower="Node at d586e47d-e0a6-65fe-9392-2370d6d507e2 [Follower]" leader= + testing.go:149: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.256Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: entering follower state: follower="Node at d586e47d-e0a6-65fe-9392-2370d6d507e2 [Follower]" leader= +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.256Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5622461-0627-5f6c-fe99-a4ba5a632a91 Address:d5622461-0627-5f6c-fe99-a4ba5a632a91} {Suffrage:Voter ID:server-d586e47d-e0a6-65fe-9392-2370d6d507e2 Address:d586e47d-e0a6-65fe-9392-2370d6d507e2} {Suffrage:Voter ID:server-c817f651-f295-15fe-623e-659e32c18c2c Address:c817f651-f295-15fe-623e-659e32c18c2c}]" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.256Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5622461-0627-5f6c-fe99-a4ba5a632a91 Address:d5622461-0627-5f6c-fe99-a4ba5a632a91} {Suffrage:Voter ID:server-d586e47d-e0a6-65fe-9392-2370d6d507e2 Address:d586e47d-e0a6-65fe-9392-2370d6d507e2} {Suffrage:Voter ID:server-c817f651-f295-15fe-623e-659e32c18c2c Address:c817f651-f295-15fe-623e-659e32c18c2c}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.256Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:08.256Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.256Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: entering follower state: follower="Node at c817f651-f295-15fe-623e-659e32c18c2c [Follower]" leader= + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.256Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: entering follower state: follower="Node at c817f651-f295-15fe-623e-659e32c18c2c [Follower]" leader= +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [WARN] server-c817f651-f295-15fe-623e-659e32c18c2c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [WARN] server-c817f651-f295-15fe-623e-659e32c18c2c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: entering candidate state: node="Node at c817f651-f295-15fe-623e-659e32c18c2c [Candidate]" term=2 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: entering candidate state: node="Node at c817f651-f295-15fe-623e-659e32c18c2c [Candidate]" term=2 +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: election won: tally=2 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: election won: tally=2 +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: entering leader state: leader="Node at c817f651-f295-15fe-623e-659e32c18c2c [Leader]" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: entering leader state: leader="Node at c817f651-f295-15fe-623e-659e32c18c2c [Leader]" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: added peer, starting replication: peer=server-d5622461-0627-5f6c-fe99-a4ba5a632a91 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: added peer, starting replication: peer=server-d5622461-0627-5f6c-fe99-a4ba5a632a91 +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: added peer, starting replication: peer=server-d586e47d-e0a6-65fe-9392-2370d6d507e2 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: added peer, starting replication: peer=server-d586e47d-e0a6-65fe-9392-2370d6d507e2 +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.320Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.425Z [INFO] stable state for Leader reached at 2021-08-25 09:31:08.320608152 +0000 UTC m=+12.586824536 (1 nodes), 64.116326ms from start of poll, 65.208043ms from cluster start. Timeout at 2021-08-25 09:31:08.425724898 +0000 UTC m=+12.691941282, 105.116746ms after stability + testing.go:149: cluster: 2021-08-25T09:31:08.425Z [INFO] stable state for Leader reached at 2021-08-25 09:31:08.320608152 +0000 UTC m=+12.586824536 (1 nodes), 64.116326ms from start of poll, 65.208043ms from cluster start. Timeout at 2021-08-25 09:31:08.425724898 +0000 UTC m=+12.691941282, 105.116746ms after stability +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.426Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: starting snapshot up to: index=12 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.426Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: starting snapshot up to: index=12 + testing.go:153: 2021-08-25T09:31:08.426Z [INFO] creating new snapshot: path=/tmp/raft569570239/snapshots/2-12-1629883868426.tmp +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.454Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: snapshot complete up to: index=12 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.454Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: snapshot complete up to: index=12 + testing.go:153: 2021-08-25T09:31:08.455Z [INFO] creating new snapshot: path=/tmp/raft569570239/snapshots/2-10013-1629883868455.tmp +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.481Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: copied to local snapshot: bytes=71 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.481Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: restored user snapshot: index=1 + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: restored user snapshot: index=1 +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.482Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.488Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.488Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:08.489Z [INFO] creating new snapshot: path=/tmp/raft808114501/snapshots/2-10013-1629883868489.tmp +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.490Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.490Z [ERROR] server-c817f651-f295-15fe-623e-659e32c18c2c: failed to get log: index=23 error="log not found" + testing.go:153: 2021-08-25T09:31:08.491Z [INFO] creating new snapshot: path=/tmp/raft866417243/snapshots/2-10013-1629883868491.tmp +testLoggerAdapter verbose: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.532Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: copied to local snapshot: bytes=71 + testing.go:149: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.532Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.532Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: Installed remote snapshot + testing.go:149: server-d586e47d-e0a6-65fe-9392-2370d6d507e2: 2021-08-25T09:31:08.532Z [INFO] server-d586e47d-e0a6-65fe-9392-2370d6d507e2: Installed remote snapshot +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.532Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.532Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" +testLoggerAdapter verbose: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.537Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: copied to local snapshot: bytes=71 + testing.go:149: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.537Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.538Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: Installed remote snapshot + testing.go:149: server-d5622461-0627-5f6c-fe99-a4ba5a632a91: 2021-08-25T09:31:08.538Z [INFO] server-d5622461-0627-5f6c-fe99-a4ba5a632a91: Installed remote snapshot +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.538Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.538Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: pipelining replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.554Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.554Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d586e47d-e0a6-65fe-9392-2370d6d507e2 d586e47d-e0a6-65fe-9392-2370d6d507e2}" +testLoggerAdapter verbose: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.554Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" + testing.go:149: server-c817f651-f295-15fe-623e-659e32c18c2c: 2021-08-25T09:31:08.554Z [INFO] server-c817f651-f295-15fe-623e-659e32c18c2c: aborting pipeline replication: peer="{Voter server-d5622461-0627-5f6c-fe99-a4ba5a632a91 d5622461-0627-5f6c-fe99-a4ba5a632a91}" +--- PASS: TestRaft_UserRestore (1.74s) === RUN TestRaft_SendSnapshotFollower -testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" - testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" -testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: entering follower state: follower="Node at a242f697-00d2-d8bf-2bcf-137be5f882fc [Follower]" leader= - testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:12.865Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: entering follower state: follower="Node at a242f697-00d2-d8bf-2bcf-137be5f882fc [Follower]" leader= -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering follower state: follower="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Follower]" leader= - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.865Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering follower state: follower="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Follower]" leader= -testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.865Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" - testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.865Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a242f697-00d2-d8bf-2bcf-137be5f882fc Address:a242f697-00d2-d8bf-2bcf-137be5f882fc} {Suffrage:Voter ID:server-90ea17d4-ad59-3540-df78-af734963bf9f Address:90ea17d4-ad59-3540-df78-af734963bf9f} {Suffrage:Voter ID:server-1956d413-47c5-00aa-5727-c278aeaeb89b Address:1956d413-47c5-00aa-5727-c278aeaeb89b}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:12.866Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:12.866Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.866Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: entering follower state: follower="Node at 1956d413-47c5-00aa-5727-c278aeaeb89b [Follower]" leader= - testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:12.866Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: entering follower state: follower="Node at 1956d413-47c5-00aa-5727-c278aeaeb89b [Follower]" leader= -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering candidate state: node="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Candidate]" term=2 - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering candidate state: node="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Candidate]" term=2 -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: election won: tally=2 - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: election won: tally=2 -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering leader state: leader="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Leader]" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: entering leader state: leader="Node at 90ea17d4-ad59-3540-df78-af734963bf9f [Leader]" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-a242f697-00d2-d8bf-2bcf-137be5f882fc - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-a242f697-00d2-d8bf-2bcf-137be5f882fc -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-1956d413-47c5-00aa-5727-c278aeaeb89b - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.926Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: added peer, starting replication: peer=server-1956d413-47c5-00aa-5727-c278aeaeb89b -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:12.927Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: pipelining replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.032Z [INFO] stable state for Follower reached at 2021-08-25 09:28:12.926919251 +0000 UTC m=+12.691135160 (2 nodes), 60.803149ms from start of poll, 62.069531ms from cluster start. Timeout at 2021-08-25 09:28:13.032024243 +0000 UTC m=+12.796240172, 105.105012ms after stability - testing.go:149: cluster: 2021-08-25T09:28:13.032Z [INFO] stable state for Follower reached at 2021-08-25 09:28:12.926919251 +0000 UTC m=+12.691135160 (2 nodes), 60.803149ms from start of poll, 62.069531ms from cluster start. Timeout at 2021-08-25 09:28:13.032024243 +0000 UTC m=+12.796240172, 105.105012ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.032Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:13.032Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.137Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.032299937 +0000 UTC m=+12.796515826 (1 nodes), 1.979µs from start of poll, 167.450197ms from cluster start. Timeout at 2021-08-25 09:28:13.137444865 +0000 UTC m=+12.901660834, 105.145008ms after stability - testing.go:149: cluster: 2021-08-25T09:28:13.137Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.032299937 +0000 UTC m=+12.796515826 (1 nodes), 1.979µs from start of poll, 167.450197ms from cluster start. Timeout at 2021-08-25 09:28:13.137444865 +0000 UTC m=+12.901660834, 105.145008ms after stability -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.137Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to pipeline appendEntries: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="append pipeline closed" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.137Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to pipeline appendEntries: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="append pipeline closed" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.138Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.558Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37119388-362e-973d-92b2-51d33844c523 Address:37119388-362e-973d-92b2-51d33844c523} {Suffrage:Voter ID:server-43fb39b5-4607-03b8-990a-01483cd126f6 Address:43fb39b5-4607-03b8-990a-01483cd126f6} {Suffrage:Voter ID:server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 Address:2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}]" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.558Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37119388-362e-973d-92b2-51d33844c523 Address:37119388-362e-973d-92b2-51d33844c523} {Suffrage:Voter ID:server-43fb39b5-4607-03b8-990a-01483cd126f6 Address:43fb39b5-4607-03b8-990a-01483cd126f6} {Suffrage:Voter ID:server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 Address:2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}]" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.559Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: entering follower state: follower="Node at 37119388-362e-973d-92b2-51d33844c523 [Follower]" leader= + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.559Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: entering follower state: follower="Node at 37119388-362e-973d-92b2-51d33844c523 [Follower]" leader= +testLoggerAdapter verbose: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.559Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37119388-362e-973d-92b2-51d33844c523 Address:37119388-362e-973d-92b2-51d33844c523} {Suffrage:Voter ID:server-43fb39b5-4607-03b8-990a-01483cd126f6 Address:43fb39b5-4607-03b8-990a-01483cd126f6} {Suffrage:Voter ID:server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 Address:2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}]" + testing.go:149: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.559Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37119388-362e-973d-92b2-51d33844c523 Address:37119388-362e-973d-92b2-51d33844c523} {Suffrage:Voter ID:server-43fb39b5-4607-03b8-990a-01483cd126f6 Address:43fb39b5-4607-03b8-990a-01483cd126f6} {Suffrage:Voter ID:server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 Address:2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}]" +testLoggerAdapter verbose: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.559Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: entering follower state: follower="Node at 43fb39b5-4607-03b8-990a-01483cd126f6 [Follower]" leader= + testing.go:149: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.559Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: entering follower state: follower="Node at 43fb39b5-4607-03b8-990a-01483cd126f6 [Follower]" leader= +testLoggerAdapter verbose: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.559Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37119388-362e-973d-92b2-51d33844c523 Address:37119388-362e-973d-92b2-51d33844c523} {Suffrage:Voter ID:server-43fb39b5-4607-03b8-990a-01483cd126f6 Address:43fb39b5-4607-03b8-990a-01483cd126f6} {Suffrage:Voter ID:server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 Address:2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}]" + testing.go:149: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.559Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37119388-362e-973d-92b2-51d33844c523 Address:37119388-362e-973d-92b2-51d33844c523} {Suffrage:Voter ID:server-43fb39b5-4607-03b8-990a-01483cd126f6 Address:43fb39b5-4607-03b8-990a-01483cd126f6} {Suffrage:Voter ID:server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 Address:2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.559Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:08.559Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.559Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: entering follower state: follower="Node at 2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 [Follower]" leader= + testing.go:149: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.559Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: entering follower state: follower="Node at 2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 [Follower]" leader= +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.611Z [WARN] server-37119388-362e-973d-92b2-51d33844c523: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.611Z [WARN] server-37119388-362e-973d-92b2-51d33844c523: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: entering candidate state: node="Node at 37119388-362e-973d-92b2-51d33844c523 [Candidate]" term=2 + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: entering candidate state: node="Node at 37119388-362e-973d-92b2-51d33844c523 [Candidate]" term=2 +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: election won: tally=2 + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: election won: tally=2 +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: entering leader state: leader="Node at 37119388-362e-973d-92b2-51d33844c523 [Leader]" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: entering leader state: leader="Node at 37119388-362e-973d-92b2-51d33844c523 [Leader]" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: added peer, starting replication: peer=server-43fb39b5-4607-03b8-990a-01483cd126f6 + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: added peer, starting replication: peer=server-43fb39b5-4607-03b8-990a-01483cd126f6 +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: added peer, starting replication: peer=server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: added peer, starting replication: peer=server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: pipelining replication: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: pipelining replication: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: pipelining replication: peer="{Voter server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.612Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: pipelining replication: peer="{Voter server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.717Z [INFO] stable state for Follower reached at 2021-08-25 09:31:08.612529877 +0000 UTC m=+12.878746301 (2 nodes), 52.804091ms from start of poll, 53.86739ms from cluster start. Timeout at 2021-08-25 09:31:08.717688099 +0000 UTC m=+12.983904543, 105.158242ms after stability + testing.go:149: cluster: 2021-08-25T09:31:08.717Z [INFO] stable state for Follower reached at 2021-08-25 09:31:08.612529877 +0000 UTC m=+12.878746301 (2 nodes), 52.804091ms from start of poll, 53.86739ms from cluster start. Timeout at 2021-08-25 09:31:08.717688099 +0000 UTC m=+12.983904543, 105.158242ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.717Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:08.717Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.823Z [INFO] stable state for Leader reached at 2021-08-25 09:31:08.717968653 +0000 UTC m=+12.984185057 (1 nodes), 2.6µs from start of poll, 159.306146ms from cluster start. Timeout at 2021-08-25 09:31:08.823140535 +0000 UTC m=+13.089357019, 105.171962ms after stability + testing.go:149: cluster: 2021-08-25T09:31:08.823Z [INFO] stable state for Leader reached at 2021-08-25 09:31:08.717968653 +0000 UTC m=+12.984185057 (1 nodes), 2.6µs from start of poll, 159.306146ms from cluster start. Timeout at 2021-08-25 09:31:08.823140535 +0000 UTC m=+13.089357019, 105.171962ms after stability +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.823Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to pipeline appendEntries: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="append pipeline closed" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.823Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to pipeline appendEntries: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="append pipeline closed" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.823Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: aborting pipeline replication: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.823Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: aborting pipeline replication: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.823Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to appendEntries to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.823Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to appendEntries to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.826Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to heartbeat to: peer=43fb39b5-4607-03b8-990a-01483cd126f6 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.826Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to heartbeat to: peer=43fb39b5-4607-03b8-990a-01483cd126f6 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" raft_test.go:1320: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.141Z [ERROR] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.141Z [ERROR] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.142Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: starting snapshot up to: index=102 - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.142Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: starting snapshot up to: index=102 - testing.go:153: 2021-08-25T09:28:13.142Z [INFO] creating new snapshot: path=/tmp/raft868723465/snapshots/2-102-1629883693142.tmp -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.149Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.149Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.154Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.154Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to appendEntries to: peer="{Voter server-a242f697-00d2-d8bf-2bcf-137be5f882fc a242f697-00d2-d8bf-2bcf-137be5f882fc}" error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: compacting logs: from=1 to=92 - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.160Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.161Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: snapshot complete up to: index=102 - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.161Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.161Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: starting snapshot up to: index=102 - testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.161Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: starting snapshot up to: index=102 - testing.go:153: 2021-08-25T09:28:13.161Z [INFO] creating new snapshot: path=/tmp/raft991231523/snapshots/2-102-1629883693161.tmp -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.173Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.173Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to heartbeat to: peer=a242f697-00d2-d8bf-2bcf-137be5f882fc error="failed to connect to peer: a242f697-00d2-d8bf-2bcf-137be5f882fc" -testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.177Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: compacting logs: from=1 to=92 - testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.177Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.178Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: snapshot complete up to: index=102 - testing.go:149: server-1956d413-47c5-00aa-5727-c278aeaeb89b: 2021-08-25T09:28:13.178Z [INFO] server-1956d413-47c5-00aa-5727-c278aeaeb89b: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.181Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to contact: server-id=server-a242f697-00d2-d8bf-2bcf-137be5f882fc time=50.180976ms - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.181Z [WARN] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to contact: server-id=server-a242f697-00d2-d8bf-2bcf-137be5f882fc time=50.180976ms -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.190Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to get log: index=2 error="log not found" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.190Z [ERROR] server-90ea17d4-ad59-3540-df78-af734963bf9f: failed to get log: index=2 error="log not found" - testing.go:153: 2021-08-25T09:28:13.191Z [INFO] creating new snapshot: path=/tmp/raft944482431/snapshots/2-102-1629883693191.tmp -testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.210Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: copied to local snapshot: bytes=693 - testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.210Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.211Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: Installed remote snapshot - testing.go:149: server-a242f697-00d2-d8bf-2bcf-137be5f882fc: 2021-08-25T09:28:13.211Z [INFO] server-a242f697-00d2-d8bf-2bcf-137be5f882fc: Installed remote snapshot -testLoggerAdapter verbose: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.215Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" - testing.go:149: server-90ea17d4-ad59-3540-df78-af734963bf9f: 2021-08-25T09:28:13.215Z [INFO] server-90ea17d4-ad59-3540-df78-af734963bf9f: aborting pipeline replication: peer="{Voter server-1956d413-47c5-00aa-5727-c278aeaeb89b 1956d413-47c5-00aa-5727-c278aeaeb89b}" ---- PASS: TestRaft_SendSnapshotFollower (0.35s) +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.826Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: starting snapshot up to: index=102 + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.826Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: starting snapshot up to: index=102 + testing.go:153: 2021-08-25T09:31:08.827Z [INFO] creating new snapshot: path=/tmp/raft765819977/snapshots/2-102-1629883868827.tmp +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.834Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to appendEntries to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.834Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to appendEntries to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.844Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to heartbeat to: peer=43fb39b5-4607-03b8-990a-01483cd126f6 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.844Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to heartbeat to: peer=43fb39b5-4607-03b8-990a-01483cd126f6 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.844Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to appendEntries to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.844Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to appendEntries to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.855Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: compacting logs: from=1 to=92 + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.855Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.855Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: snapshot complete up to: index=102 + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.855Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.855Z [ERROR] server-43fb39b5-4607-03b8-990a-01483cd126f6: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.855Z [ERROR] server-43fb39b5-4607-03b8-990a-01483cd126f6: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.855Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: starting snapshot up to: index=102 + testing.go:149: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.855Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: starting snapshot up to: index=102 + testing.go:153: 2021-08-25T09:31:08.855Z [INFO] creating new snapshot: path=/tmp/raft198525837/snapshots/2-102-1629883868855.tmp +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.861Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to heartbeat to: peer=43fb39b5-4607-03b8-990a-01483cd126f6 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.861Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to heartbeat to: peer=43fb39b5-4607-03b8-990a-01483cd126f6 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.870Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to get log: index=2 error="log not found" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.870Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to get log: index=2 error="log not found" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.871Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to install snapshot: id=2-102-1629883868827 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.871Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to install snapshot: id=2-102-1629883868827 error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.871Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to send snapshot to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.871Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to send snapshot to: peer="{Voter server-43fb39b5-4607-03b8-990a-01483cd126f6 43fb39b5-4607-03b8-990a-01483cd126f6}" error="failed to connect to peer: 43fb39b5-4607-03b8-990a-01483cd126f6" +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.872Z [WARN] server-37119388-362e-973d-92b2-51d33844c523: failed to contact: server-id=server-43fb39b5-4607-03b8-990a-01483cd126f6 time=50.136182ms + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.872Z [WARN] server-37119388-362e-973d-92b2-51d33844c523: failed to contact: server-id=server-43fb39b5-4607-03b8-990a-01483cd126f6 time=50.136182ms +testLoggerAdapter verbose: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.882Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: compacting logs: from=1 to=92 + testing.go:149: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.882Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.882Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: snapshot complete up to: index=102 + testing.go:149: server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: 2021-08-25T09:31:08.882Z [INFO] server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.920Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to get log: index=2 error="log not found" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.920Z [ERROR] server-37119388-362e-973d-92b2-51d33844c523: failed to get log: index=2 error="log not found" + testing.go:153: 2021-08-25T09:31:08.920Z [INFO] creating new snapshot: path=/tmp/raft106233187/snapshots/2-102-1629883868920.tmp +testLoggerAdapter verbose: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.948Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: copied to local snapshot: bytes=693 + testing.go:149: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.948Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.948Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: Installed remote snapshot + testing.go:149: server-43fb39b5-4607-03b8-990a-01483cd126f6: 2021-08-25T09:31:08.948Z [INFO] server-43fb39b5-4607-03b8-990a-01483cd126f6: Installed remote snapshot +testLoggerAdapter verbose: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.950Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: aborting pipeline replication: peer="{Voter server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}" + testing.go:149: server-37119388-362e-973d-92b2-51d33844c523: 2021-08-25T09:31:08.950Z [INFO] server-37119388-362e-973d-92b2-51d33844c523: aborting pipeline replication: peer="{Voter server-2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6 2d1ecf8b-587d-b06c-8ad7-83c880ebd5c6}" +--- PASS: TestRaft_SendSnapshotFollower (0.40s) === RUN TestRaft_SendSnapshotAndLogsFollower -testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" - testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" -testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: entering follower state: follower="Node at d6fdd995-74de-79cc-3395-f2acc9b7508a [Follower]" leader= - testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.219Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: entering follower state: follower="Node at d6fdd995-74de-79cc-3395-f2acc9b7508a [Follower]" leader= -testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" - testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" -testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: entering follower state: follower="Node at d879deee-1498-c8d0-09f8-225d22a28293 [Follower]" leader= - testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.220Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: entering follower state: follower="Node at d879deee-1498-c8d0-09f8-225d22a28293 [Follower]" leader= -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6fdd995-74de-79cc-3395-f2acc9b7508a Address:d6fdd995-74de-79cc-3395-f2acc9b7508a} {Suffrage:Voter ID:server-d879deee-1498-c8d0-09f8-225d22a28293 Address:d879deee-1498-c8d0-09f8-225d22a28293} {Suffrage:Voter ID:server-371e187f-c0a7-77bf-207e-183b35ce8bc9 Address:371e187f-c0a7-77bf-207e-183b35ce8bc9}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.220Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering follower state: follower="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:13.220Z [INFO] starting stability test: raft-state=Follower - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.220Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering follower state: follower="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Follower]" leader= -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.272Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.272Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering candidate state: node="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Candidate]" term=2 - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering candidate state: node="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Candidate]" term=2 -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: election won: tally=2 - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: election won: tally=2 -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering leader state: leader="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Leader]" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: entering leader state: leader="Node at 371e187f-c0a7-77bf-207e-183b35ce8bc9 [Leader]" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d6fdd995-74de-79cc-3395-f2acc9b7508a - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d6fdd995-74de-79cc-3395-f2acc9b7508a -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d879deee-1498-c8d0-09f8-225d22a28293 - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: added peer, starting replication: peer=server-d879deee-1498-c8d0-09f8-225d22a28293 -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.273Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.378Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.273443206 +0000 UTC m=+13.037659135 (2 nodes), 52.775875ms from start of poll, 53.980263ms from cluster start. Timeout at 2021-08-25 09:28:13.37852422 +0000 UTC m=+13.142740188, 105.081053ms after stability - testing.go:149: cluster: 2021-08-25T09:28:13.378Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.273443206 +0000 UTC m=+13.037659135 (2 nodes), 52.775875ms from start of poll, 53.980263ms from cluster start. Timeout at 2021-08-25 09:28:13.37852422 +0000 UTC m=+13.142740188, 105.081053ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.378Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:13.378Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.484Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.378781656 +0000 UTC m=+13.142997525 (1 nodes), 1.8µs from start of poll, 159.318653ms from cluster start. Timeout at 2021-08-25 09:28:13.483931323 +0000 UTC m=+13.248147272, 105.149747ms after stability - testing.go:149: cluster: 2021-08-25T09:28:13.484Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.378781656 +0000 UTC m=+13.142997525 (1 nodes), 1.8µs from start of poll, 159.318653ms from cluster start. Timeout at 2021-08-25 09:28:13.483931323 +0000 UTC m=+13.248147272, 105.149747ms after stability -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to pipeline appendEntries: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="append pipeline closed" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to pipeline appendEntries: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="append pipeline closed" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.484Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:08.955Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e Address:dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e} {Suffrage:Voter ID:server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 Address:e677fbf9-c92d-40a1-2d87-ba39ab43b829} {Suffrage:Voter ID:server-540075b3-bb51-75bb-89b6-177b8d6d1b82 Address:540075b3-bb51-75bb-89b6-177b8d6d1b82}]" + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:08.955Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e Address:dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e} {Suffrage:Voter ID:server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 Address:e677fbf9-c92d-40a1-2d87-ba39ab43b829} {Suffrage:Voter ID:server-540075b3-bb51-75bb-89b6-177b8d6d1b82 Address:540075b3-bb51-75bb-89b6-177b8d6d1b82}]" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:08.955Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering follower state: follower="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Follower]" leader= + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:08.955Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering follower state: follower="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Follower]" leader= +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:08.955Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e Address:dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e} {Suffrage:Voter ID:server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 Address:e677fbf9-c92d-40a1-2d87-ba39ab43b829} {Suffrage:Voter ID:server-540075b3-bb51-75bb-89b6-177b8d6d1b82 Address:540075b3-bb51-75bb-89b6-177b8d6d1b82}]" + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:08.955Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e Address:dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e} {Suffrage:Voter ID:server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 Address:e677fbf9-c92d-40a1-2d87-ba39ab43b829} {Suffrage:Voter ID:server-540075b3-bb51-75bb-89b6-177b8d6d1b82 Address:540075b3-bb51-75bb-89b6-177b8d6d1b82}]" +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:08.956Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: entering follower state: follower="Node at e677fbf9-c92d-40a1-2d87-ba39ab43b829 [Follower]" leader= + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:08.956Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: entering follower state: follower="Node at e677fbf9-c92d-40a1-2d87-ba39ab43b829 [Follower]" leader= +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:08.956Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e Address:dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e} {Suffrage:Voter ID:server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 Address:e677fbf9-c92d-40a1-2d87-ba39ab43b829} {Suffrage:Voter ID:server-540075b3-bb51-75bb-89b6-177b8d6d1b82 Address:540075b3-bb51-75bb-89b6-177b8d6d1b82}]" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:08.956Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e Address:dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e} {Suffrage:Voter ID:server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 Address:e677fbf9-c92d-40a1-2d87-ba39ab43b829} {Suffrage:Voter ID:server-540075b3-bb51-75bb-89b6-177b8d6d1b82 Address:540075b3-bb51-75bb-89b6-177b8d6d1b82}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:08.956Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:08.956Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:08.956Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering follower state: follower="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Follower]" leader= + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:08.956Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering follower state: follower="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Follower]" leader= +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.009Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.009Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering candidate state: node="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Candidate]" term=2 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering candidate state: node="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Candidate]" term=2 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: election won: tally=2 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: election won: tally=2 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering leader state: leader="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Leader]" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering leader state: leader="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Leader]" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.010Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.115Z [INFO] stable state for Follower reached at 2021-08-25 09:31:09.010321778 +0000 UTC m=+13.276538182 (2 nodes), 53.907907ms from start of poll, 55.070697ms from cluster start. Timeout at 2021-08-25 09:31:09.115422526 +0000 UTC m=+13.381638950, 105.100768ms after stability + testing.go:149: cluster: 2021-08-25T09:31:09.115Z [INFO] stable state for Follower reached at 2021-08-25 09:31:09.010321778 +0000 UTC m=+13.276538182 (2 nodes), 53.907907ms from start of poll, 55.070697ms from cluster start. Timeout at 2021-08-25 09:31:09.115422526 +0000 UTC m=+13.381638950, 105.100768ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.115Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:09.115Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.220Z [INFO] stable state for Leader reached at 2021-08-25 09:31:09.115687281 +0000 UTC m=+13.381903665 (1 nodes), 2.22µs from start of poll, 160.43618ms from cluster start. Timeout at 2021-08-25 09:31:09.220813007 +0000 UTC m=+13.487029491, 105.125826ms after stability + testing.go:149: cluster: 2021-08-25T09:31:09.220Z [INFO] stable state for Leader reached at 2021-08-25 09:31:09.115687281 +0000 UTC m=+13.381903665 (1 nodes), 2.22µs from start of poll, 160.43618ms from cluster start. Timeout at 2021-08-25 09:31:09.220813007 +0000 UTC m=+13.487029491, 105.125826ms after stability +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.221Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to pipeline appendEntries: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="append pipeline closed" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.221Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to pipeline appendEntries: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="append pipeline closed" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.221Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.221Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.221Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.221Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.224Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to heartbeat to: peer=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.224Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to heartbeat to: peer=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" raft_test.go:1362: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.487Z [ERROR] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.487Z [ERROR] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.487Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: starting snapshot up to: index=101 - testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.487Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: starting snapshot up to: index=101 - testing.go:153: 2021-08-25T09:28:13.487Z [INFO] creating new snapshot: path=/tmp/raft663820295/snapshots/2-101-1629883693487.tmp -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.488Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.488Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.494Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.494Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.505Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.505Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to appendEntries to: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" -testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: compacting logs: from=1 to=92 - testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: snapshot complete up to: index=101 - testing.go:149: server-d879deee-1498-c8d0-09f8-225d22a28293: 2021-08-25T09:28:13.506Z [INFO] server-d879deee-1498-c8d0-09f8-225d22a28293: snapshot complete up to: index=101 -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: starting snapshot up to: index=102 - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: starting snapshot up to: index=102 - testing.go:153: 2021-08-25T09:28:13.506Z [INFO] creating new snapshot: path=/tmp/raft833142737/snapshots/2-102-1629883693506.tmp -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.506Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: compacting logs: from=1 to=92 - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: snapshot complete up to: index=102 - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.524Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.528Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.528Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="failed to connect to peer: d6fdd995-74de-79cc-3395-f2acc9b7508a" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.224Z [ERROR] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.224Z [ERROR] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.224Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: starting snapshot up to: index=101 + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.224Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: starting snapshot up to: index=101 + testing.go:153: 2021-08-25T09:31:09.224Z [INFO] creating new snapshot: path=/tmp/raft428650257/snapshots/2-101-1629883869224.tmp +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.231Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.231Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.240Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to heartbeat to: peer=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.240Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to heartbeat to: peer=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.242Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.242Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.252Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: compacting logs: from=1 to=92 + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.252Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.253Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: snapshot complete up to: index=101 + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.253Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: snapshot complete up to: index=101 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.253Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: starting snapshot up to: index=102 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.253Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: starting snapshot up to: index=102 + testing.go:153: 2021-08-25T09:31:09.253Z [INFO] creating new snapshot: path=/tmp/raft962829675/snapshots/2-102-1629883869253.tmp +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.260Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to heartbeat to: peer=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.260Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to heartbeat to: peer=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.271Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to contact: server-id=server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e time=50.116143ms + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.271Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to contact: server-id=server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e time=50.116143ms +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.271Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.271Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to appendEntries to: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" error="failed to connect to peer: dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.275Z [WARN] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: heartbeat timeout reached, starting election: last-leader=540075b3-bb51-75bb-89b6-177b8d6d1b82 + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.275Z [WARN] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: heartbeat timeout reached, starting election: last-leader=540075b3-bb51-75bb-89b6-177b8d6d1b82 +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.275Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering candidate state: node="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Candidate]" term=3 + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.275Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering candidate state: node="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Candidate]" term=3 +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.276Z [ERROR] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to make requestVote RPC: target="{Voter server-540075b3-bb51-75bb-89b6-177b8d6d1b82 540075b3-bb51-75bb-89b6-177b8d6d1b82}" error="failed to connect to peer: 540075b3-bb51-75bb-89b6-177b8d6d1b82" + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.276Z [ERROR] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to make requestVote RPC: target="{Voter server-540075b3-bb51-75bb-89b6-177b8d6d1b82 540075b3-bb51-75bb-89b6-177b8d6d1b82}" error="failed to connect to peer: 540075b3-bb51-75bb-89b6-177b8d6d1b82" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.276Z [ERROR] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to make requestVote RPC: target="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" error="failed to connect to peer: e677fbf9-c92d-40a1-2d87-ba39ab43b829" + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.276Z [ERROR] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to make requestVote RPC: target="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" error="failed to connect to peer: e677fbf9-c92d-40a1-2d87-ba39ab43b829" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.280Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: compacting logs: from=1 to=92 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.280Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.280Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: snapshot complete up to: index=102 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.280Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: snapshot complete up to: index=102 raft_test.go:1383: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to contact: server-id=server-d6fdd995-74de-79cc-3395-f2acc9b7508a time=50.192335ms - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [WARN] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to contact: server-id=server-d6fdd995-74de-79cc-3395-f2acc9b7508a time=50.192335ms -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to get log: index=2 error="log not found" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.532Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to get log: index=2 error="log not found" - testing.go:153: 2021-08-25T09:28:13.533Z [INFO] creating new snapshot: path=/tmp/raft226398285/snapshots/2-102-1629883693533.tmp -testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.550Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: copied to local snapshot: bytes=693 - testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.550Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.551Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: Installed remote snapshot - testing.go:149: server-d6fdd995-74de-79cc-3395-f2acc9b7508a: 2021-08-25T09:28:13.551Z [INFO] server-d6fdd995-74de-79cc-3395-f2acc9b7508a: Installed remote snapshot -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.551Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.551Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: pipelining replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d879deee-1498-c8d0-09f8-225d22a28293 d879deee-1498-c8d0-09f8-225d22a28293}" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.556Z [INFO] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: aborting pipeline replication: peer="{Voter server-d6fdd995-74de-79cc-3395-f2acc9b7508a d6fdd995-74de-79cc-3395-f2acc9b7508a}" -testLoggerAdapter verbose: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.606Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="command timed out" - testing.go:149: server-371e187f-c0a7-77bf-207e-183b35ce8bc9: 2021-08-25T09:28:13.606Z [ERROR] server-371e187f-c0a7-77bf-207e-183b35ce8bc9: failed to heartbeat to: peer=d6fdd995-74de-79cc-3395-f2acc9b7508a error="command timed out" ---- PASS: TestRaft_SendSnapshotAndLogsFollower (0.39s) +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.320Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to get log: index=2 error="log not found" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.320Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to get log: index=2 error="log not found" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.321Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: ignoring installSnapshot request with older term than current term: request-term=2 current-term=3 + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.321Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: ignoring installSnapshot request with older term than current term: request-term=2 current-term=3 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.321Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: peer has newer term, stopping replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.321Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: peer has newer term, stopping replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.321Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering follower state: follower="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Follower]" leader= + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.321Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering follower state: follower="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Follower]" leader= +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.321Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.321Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.374Z [WARN] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: Election timeout reached, restarting election + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.374Z [WARN] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: Election timeout reached, restarting election +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.374Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering candidate state: node="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Candidate]" term=4 + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.374Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering candidate state: node="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Candidate]" term=4 +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.374Z [WARN] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: rejecting vote request since we have a leader: from=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e leader=540075b3-bb51-75bb-89b6-177b8d6d1b82 + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.374Z [WARN] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: rejecting vote request since we have a leader: from=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e leader=540075b3-bb51-75bb-89b6-177b8d6d1b82 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.374Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: rejecting vote request since our last index is greater: candidate=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e last-index=202 last-candidate-index=2 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.374Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: rejecting vote request since our last index is greater: candidate=dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e last-index=202 last-candidate-index=2 +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.382Z [WARN] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: heartbeat timeout reached, starting election: last-leader=540075b3-bb51-75bb-89b6-177b8d6d1b82 + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.382Z [WARN] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: heartbeat timeout reached, starting election: last-leader=540075b3-bb51-75bb-89b6-177b8d6d1b82 +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.382Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: entering candidate state: node="Node at e677fbf9-c92d-40a1-2d87-ba39ab43b829 [Candidate]" term=3 + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.382Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: entering candidate state: node="Node at e677fbf9-c92d-40a1-2d87-ba39ab43b829 [Candidate]" term=3 +testLoggerAdapter verbose: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.382Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: entering follower state: follower="Node at e677fbf9-c92d-40a1-2d87-ba39ab43b829 [Follower]" leader= + testing.go:149: server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: 2021-08-25T09:31:09.382Z [INFO] server-e677fbf9-c92d-40a1-2d87-ba39ab43b829: entering follower state: follower="Node at e677fbf9-c92d-40a1-2d87-ba39ab43b829 [Follower]" leader= +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering candidate state: node="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Candidate]" term=5 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering candidate state: node="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Candidate]" term=5 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: election won: tally=2 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: election won: tally=2 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering leader state: leader="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Leader]" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.399Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering follower state: follower="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Follower]" leader= + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: entering leader state: leader="Node at 540075b3-bb51-75bb-89b6-177b8d6d1b82 [Leader]" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: added peer, starting replication: peer=server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.399Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: entering follower state: follower="Node at dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e [Follower]" leader= +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.399Z [WARN] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to get previous log: previous-index=202 last-index=2 error="log not found" + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.399Z [WARN] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: failed to get previous log: previous-index=202 last-index=2 error="log not found" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: appendEntries rejected, sending older logs: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" next=3 + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.399Z [WARN] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: appendEntries rejected, sending older logs: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" next=3 +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.400Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to get log: index=2 error="log not found" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.400Z [ERROR] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: failed to get log: index=2 error="log not found" + testing.go:153: 2021-08-25T09:31:09.400Z [INFO] creating new snapshot: path=/tmp/raft562394951/snapshots/2-102-1629883869400.tmp +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.428Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: copied to local snapshot: bytes=693 + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.428Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.429Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: Installed remote snapshot + testing.go:149: server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: 2021-08-25T09:31:09.429Z [INFO] server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e: Installed remote snapshot +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.429Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.429Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: pipelining replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.432Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.432Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-e677fbf9-c92d-40a1-2d87-ba39ab43b829 e677fbf9-c92d-40a1-2d87-ba39ab43b829}" +testLoggerAdapter verbose: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.432Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" + testing.go:149: server-540075b3-bb51-75bb-89b6-177b8d6d1b82: 2021-08-25T09:31:09.432Z [INFO] server-540075b3-bb51-75bb-89b6-177b8d6d1b82: aborting pipeline replication: peer="{Voter server-dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e dbc5d3cd-12df-fe6d-a292-6a7bb5590a2e}" +--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.48s) === RUN TestRaft_ReJoinFollower -testLoggerAdapter verbose: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.610Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" - testing.go:149: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.610Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" -testLoggerAdapter verbose: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.611Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: entering follower state: follower="Node at ef4138ce-d4e7-30c7-623a-85f9f00f8740 [Follower]" leader= - testing.go:149: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:13.611Z [INFO] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: entering follower state: follower="Node at ef4138ce-d4e7-30c7-623a-85f9f00f8740 [Follower]" leader= -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering follower state: follower="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Follower]" leader= - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.611Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering follower state: follower="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Follower]" leader= -testLoggerAdapter verbose: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" - testing.go:149: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740} {Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.611Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:13.611Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: entering follower state: follower="Node at fd0ad873-2e1b-a020-190e-ee6147ed4eb8 [Follower]" leader= - testing.go:149: server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: 2021-08-25T09:28:13.611Z [INFO] server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8: entering follower state: follower="Node at fd0ad873-2e1b-a020-190e-ee6147ed4eb8 [Follower]" leader= -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering candidate state: node="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Candidate]" term=2 - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.691Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering candidate state: node="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Candidate]" term=2 -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: election won: tally=2 - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: election won: tally=2 -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering leader state: leader="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Leader]" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: entering leader state: leader="Node at 08a48afc-4e7f-e0ea-889d-60485e66883c [Leader]" -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.692Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.798Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.69281025 +0000 UTC m=+13.457026139 (1 nodes), 81.02523ms from start of poll, 82.085131ms from cluster start. Timeout at 2021-08-25 09:28:13.7979282 +0000 UTC m=+13.562144149, 105.11801ms after stability - testing.go:149: cluster: 2021-08-25T09:28:13.798Z [INFO] stable state for Leader reached at 2021-08-25 09:28:13.69281025 +0000 UTC m=+13.457026139 (1 nodes), 81.02523ms from start of poll, 82.085131ms from cluster start. Timeout at 2021-08-25 09:28:13.7979282 +0000 UTC m=+13.562144149, 105.11801ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.803Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:13.803Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:13.908Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.803380214 +0000 UTC m=+13.567596123 (2 nodes), 3.32µs from start of poll, 192.655115ms from cluster start. Timeout at 2021-08-25 09:28:13.908528881 +0000 UTC m=+13.672744850, 105.148727ms after stability - testing.go:149: cluster: 2021-08-25T09:28:13.908Z [INFO] stable state for Follower reached at 2021-08-25 09:28:13.803380214 +0000 UTC m=+13.567596123 (2 nodes), 3.32µs from start of poll, 192.655115ms from cluster start. Timeout at 2021-08-25 09:28:13.908528881 +0000 UTC m=+13.672744850, 105.148727ms after stability -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.908Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=RemoveServer server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr= servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.908Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=RemoveServer server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr= servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8}]" -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: removed peer, stopping replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 last-index=3 - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: removed peer, stopping replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 last-index=3 -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:13.909Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.019Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=AddStaging server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr=ef4138ce-d4e7-30c7-623a-85f9f00f8740 servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8} {Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740}]" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.019Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: updating configuration: command=AddStaging server-id=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 server-addr=ef4138ce-d4e7-30c7-623a-85f9f00f8740 servers="[{Suffrage:Voter ID:server-08a48afc-4e7f-e0ea-889d-60485e66883c Address:08a48afc-4e7f-e0ea-889d-60485e66883c} {Suffrage:Voter ID:server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 Address:fd0ad873-2e1b-a020-190e-ee6147ed4eb8} {Suffrage:Voter ID:server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 Address:ef4138ce-d4e7-30c7-623a-85f9f00f8740}]" -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: added peer, starting replication: peer=server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 -testLoggerAdapter verbose: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:14.020Z [WARN] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: failed to get previous log: previous-index=4 last-index=3 error="log not found" - testing.go:149: server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: 2021-08-25T09:28:14.020Z [WARN] server-ef4138ce-d4e7-30c7-623a-85f9f00f8740: failed to get previous log: previous-index=4 last-index=3 error="log not found" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.020Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:14.020Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: appendEntries rejected, sending older logs: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" next=4 - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [WARN] server-08a48afc-4e7f-e0ea-889d-60485e66883c: appendEntries rejected, sending older logs: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" next=4 -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.020Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: pipelining replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.125Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.020561449 +0000 UTC m=+13.784777338 (1 nodes), 1.52µs from start of poll, 409.83633ms from cluster start. Timeout at 2021-08-25 09:28:14.125733834 +0000 UTC m=+13.889949763, 105.172425ms after stability - testing.go:149: cluster: 2021-08-25T09:28:14.125Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.020561449 +0000 UTC m=+13.784777338 (1 nodes), 1.52µs from start of poll, 409.83633ms from cluster start. Timeout at 2021-08-25 09:28:14.125733834 +0000 UTC m=+13.889949763, 105.172425ms after stability -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-fd0ad873-2e1b-a020-190e-ee6147ed4eb8 fd0ad873-2e1b-a020-190e-ee6147ed4eb8}" -testLoggerAdapter verbose: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" - testing.go:149: server-08a48afc-4e7f-e0ea-889d-60485e66883c: 2021-08-25T09:28:14.126Z [INFO] server-08a48afc-4e7f-e0ea-889d-60485e66883c: aborting pipeline replication: peer="{Voter server-ef4138ce-d4e7-30c7-623a-85f9f00f8740 ef4138ce-d4e7-30c7-623a-85f9f00f8740}" ---- PASS: TestRaft_ReJoinFollower (0.52s) +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.438Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3} {Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.438Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3} {Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.438Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering follower state: follower="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Follower]" leader= + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.438Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering follower state: follower="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Follower]" leader= +testLoggerAdapter verbose: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.438Z [INFO] server-081c4e6a-550d-289e-5f10-021ed3b72622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3} {Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" + testing.go:149: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.438Z [INFO] server-081c4e6a-550d-289e-5f10-021ed3b72622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3} {Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" +testLoggerAdapter verbose: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.438Z [INFO] server-081c4e6a-550d-289e-5f10-021ed3b72622: entering follower state: follower="Node at 081c4e6a-550d-289e-5f10-021ed3b72622 [Follower]" leader= + testing.go:149: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.438Z [INFO] server-081c4e6a-550d-289e-5f10-021ed3b72622: entering follower state: follower="Node at 081c4e6a-550d-289e-5f10-021ed3b72622 [Follower]" leader= +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.438Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3} {Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.438Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3} {Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.439Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:09.439Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.439Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering follower state: follower="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Follower]" leader= + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.439Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering follower state: follower="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Follower]" leader= +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering candidate state: node="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Candidate]" term=2 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering candidate state: node="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Candidate]" term=2 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: election won: tally=2 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: election won: tally=2 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering leader state: leader="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Leader]" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering leader state: leader="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Leader]" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-081c4e6a-550d-289e-5f10-021ed3b72622 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-081c4e6a-550d-289e-5f10-021ed3b72622 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.507Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.508Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.508Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.613Z [INFO] stable state for Leader reached at 2021-08-25 09:31:09.507786178 +0000 UTC m=+13.774002582 (1 nodes), 68.635161ms from start of poll, 69.834288ms from cluster start. Timeout at 2021-08-25 09:31:09.612911584 +0000 UTC m=+13.879128048, 105.125466ms after stability + testing.go:149: cluster: 2021-08-25T09:31:09.613Z [INFO] stable state for Leader reached at 2021-08-25 09:31:09.507786178 +0000 UTC m=+13.774002582 (1 nodes), 68.635161ms from start of poll, 69.834288ms from cluster start. Timeout at 2021-08-25 09:31:09.612911584 +0000 UTC m=+13.879128048, 105.125466ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.618Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:09.618Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.723Z [INFO] stable state for Follower reached at 2021-08-25 09:31:09.618437124 +0000 UTC m=+13.884653508 (2 nodes), 3.06µs from start of poll, 180.485214ms from cluster start. Timeout at 2021-08-25 09:31:09.723511754 +0000 UTC m=+13.989728238, 105.07473ms after stability + testing.go:149: cluster: 2021-08-25T09:31:09.723Z [INFO] stable state for Follower reached at 2021-08-25 09:31:09.618437124 +0000 UTC m=+13.884653508 (2 nodes), 3.06µs from start of poll, 180.485214ms from cluster start. Timeout at 2021-08-25 09:31:09.723511754 +0000 UTC m=+13.989728238, 105.07473ms after stability +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.723Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: updating configuration: command=RemoveServer server-id=server-ded9c8bd-6279-0276-15e8-13f587309de3 server-addr= servers="[{Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.723Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: updating configuration: command=RemoveServer server-id=server-ded9c8bd-6279-0276-15e8-13f587309de3 server-addr= servers="[{Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881}]" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.724Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: removed peer, stopping replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 last-index=3 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.724Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: removed peer, stopping replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 last-index=3 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.724Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.724Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.786Z [WARN] server-ded9c8bd-6279-0276-15e8-13f587309de3: heartbeat timeout reached, starting election: last-leader=bc1116fe-ae2a-50da-2733-45068030f881 + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.786Z [WARN] server-ded9c8bd-6279-0276-15e8-13f587309de3: heartbeat timeout reached, starting election: last-leader=bc1116fe-ae2a-50da-2733-45068030f881 +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.787Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering candidate state: node="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Candidate]" term=3 + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.787Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering candidate state: node="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Candidate]" term=3 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.787Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: rejecting vote request since we have a leader: from=ded9c8bd-6279-0276-15e8-13f587309de3 leader=bc1116fe-ae2a-50da-2733-45068030f881 +testLoggerAdapter verbose: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.787Z [WARN] server-081c4e6a-550d-289e-5f10-021ed3b72622: rejecting vote request since we have a leader: from=ded9c8bd-6279-0276-15e8-13f587309de3 leader=bc1116fe-ae2a-50da-2733-45068030f881 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.787Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: rejecting vote request since we have a leader: from=ded9c8bd-6279-0276-15e8-13f587309de3 leader=bc1116fe-ae2a-50da-2733-45068030f881 + testing.go:149: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.787Z [WARN] server-081c4e6a-550d-289e-5f10-021ed3b72622: rejecting vote request since we have a leader: from=ded9c8bd-6279-0276-15e8-13f587309de3 leader=bc1116fe-ae2a-50da-2733-45068030f881 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.829Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: updating configuration: command=AddStaging server-id=server-ded9c8bd-6279-0276-15e8-13f587309de3 server-addr=ded9c8bd-6279-0276-15e8-13f587309de3 servers="[{Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881} {Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3}]" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.829Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: updating configuration: command=AddStaging server-id=server-ded9c8bd-6279-0276-15e8-13f587309de3 server-addr=ded9c8bd-6279-0276-15e8-13f587309de3 servers="[{Suffrage:Voter ID:server-081c4e6a-550d-289e-5f10-021ed3b72622 Address:081c4e6a-550d-289e-5f10-021ed3b72622} {Suffrage:Voter ID:server-bc1116fe-ae2a-50da-2733-45068030f881 Address:bc1116fe-ae2a-50da-2733-45068030f881} {Suffrage:Voter ID:server-ded9c8bd-6279-0276-15e8-13f587309de3 Address:ded9c8bd-6279-0276-15e8-13f587309de3}]" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.829Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.829Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.829Z [ERROR] server-bc1116fe-ae2a-50da-2733-45068030f881: peer has newer term, stopping replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.829Z [ERROR] server-bc1116fe-ae2a-50da-2733-45068030f881: peer has newer term, stopping replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.830Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering follower state: follower="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Follower]" leader= + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.830Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering follower state: follower="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Follower]" leader= +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.830Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.830Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.830Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" + testing.go:149: cluster: 2021-08-25T09:31:09.830Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.842Z [WARN] server-ded9c8bd-6279-0276-15e8-13f587309de3: Election timeout reached, restarting election + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.842Z [WARN] server-ded9c8bd-6279-0276-15e8-13f587309de3: Election timeout reached, restarting election +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.842Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering candidate state: node="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Candidate]" term=4 + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.842Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering candidate state: node="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Candidate]" term=4 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.842Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: rejecting vote request since our last index is greater: candidate=ded9c8bd-6279-0276-15e8-13f587309de3 last-index=4 last-candidate-index=3 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.842Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: rejecting vote request since our last index is greater: candidate=ded9c8bd-6279-0276-15e8-13f587309de3 last-index=4 last-candidate-index=3 +testLoggerAdapter verbose: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.842Z [WARN] server-081c4e6a-550d-289e-5f10-021ed3b72622: rejecting vote request since we have a leader: from=ded9c8bd-6279-0276-15e8-13f587309de3 leader=bc1116fe-ae2a-50da-2733-45068030f881 + testing.go:149: server-081c4e6a-550d-289e-5f10-021ed3b72622: 2021-08-25T09:31:09.842Z [WARN] server-081c4e6a-550d-289e-5f10-021ed3b72622: rejecting vote request since we have a leader: from=ded9c8bd-6279-0276-15e8-13f587309de3 leader=bc1116fe-ae2a-50da-2733-45068030f881 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering candidate state: node="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Candidate]" term=5 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering candidate state: node="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Candidate]" term=5 +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.889Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering follower state: follower="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Follower]" leader= + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.889Z [INFO] server-ded9c8bd-6279-0276-15e8-13f587309de3: entering follower state: follower="Node at ded9c8bd-6279-0276-15e8-13f587309de3 [Follower]" leader= +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: election won: tally=2 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: election won: tally=2 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering leader state: leader="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Leader]" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: entering leader state: leader="Node at bc1116fe-ae2a-50da-2733-45068030f881 [Leader]" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-081c4e6a-550d-289e-5f10-021ed3b72622 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-081c4e6a-550d-289e-5f10-021ed3b72622 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: added peer, starting replication: peer=server-ded9c8bd-6279-0276-15e8-13f587309de3 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.889Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" +testLoggerAdapter verbose: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.889Z [WARN] server-ded9c8bd-6279-0276-15e8-13f587309de3: failed to get previous log: previous-index=4 last-index=3 error="log not found" + testing.go:149: server-ded9c8bd-6279-0276-15e8-13f587309de3: 2021-08-25T09:31:09.889Z [WARN] server-ded9c8bd-6279-0276-15e8-13f587309de3: failed to get previous log: previous-index=4 last-index=3 error="log not found" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.890Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: appendEntries rejected, sending older logs: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" next=4 + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.890Z [WARN] server-bc1116fe-ae2a-50da-2733-45068030f881: appendEntries rejected, sending older logs: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" next=4 +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.890Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.890Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: pipelining replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.994Z [INFO] stable state for Leader reached at 2021-08-25 09:31:09.88973413 +0000 UTC m=+14.155950554 (1 nodes), 59.518618ms from start of poll, 451.78226ms from cluster start. Timeout at 2021-08-25 09:31:09.994760565 +0000 UTC m=+14.260976969, 105.026415ms after stability + testing.go:149: cluster: 2021-08-25T09:31:09.994Z [INFO] stable state for Leader reached at 2021-08-25 09:31:09.88973413 +0000 UTC m=+14.155950554 (1 nodes), 59.518618ms from start of poll, 451.78226ms from cluster start. Timeout at 2021-08-25 09:31:09.994760565 +0000 UTC m=+14.260976969, 105.026415ms after stability +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.995Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.995Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-ded9c8bd-6279-0276-15e8-13f587309de3 ded9c8bd-6279-0276-15e8-13f587309de3}" +testLoggerAdapter verbose: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.995Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" + testing.go:149: server-bc1116fe-ae2a-50da-2733-45068030f881: 2021-08-25T09:31:09.995Z [INFO] server-bc1116fe-ae2a-50da-2733-45068030f881: aborting pipeline replication: peer="{Voter server-081c4e6a-550d-289e-5f10-021ed3b72622 081c4e6a-550d-289e-5f10-021ed3b72622}" +--- PASS: TestRaft_ReJoinFollower (0.56s) === RUN TestRaft_LeaderLeaseExpire -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering follower state: follower="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Follower]" leader= - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.130Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering follower state: follower="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Follower]" leader= -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-056c5c14-3305-18be-9824-f917cef0ab0e Address:056c5c14-3305-18be-9824-f917cef0ab0e} {Suffrage:Voter ID:server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 Address:0f74db2c-ded7-6807-fe92-1bb9c00da2f8}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.130Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:14.130Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.130Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=2 - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.220Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=2 -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: election won: tally=2 - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: election won: tally=2 -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering leader state: leader="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Leader]" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering leader state: leader="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Leader]" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: added peer, starting replication: peer=server-056c5c14-3305-18be-9824-f917cef0ab0e - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: added peer, starting replication: peer=server-056c5c14-3305-18be-9824-f917cef0ab0e -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: pipelining replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.221Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: pipelining replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.326Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.221477996 +0000 UTC m=+13.985693925 (1 nodes), 90.740347ms from start of poll, 91.704797ms from cluster start. Timeout at 2021-08-25 09:28:14.326609765 +0000 UTC m=+14.090825834, 105.131909ms after stability - testing.go:149: cluster: 2021-08-25T09:28:14.326Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.221477996 +0000 UTC m=+13.985693925 (1 nodes), 90.740347ms from start of poll, 91.704797ms from cluster start. Timeout at 2021-08-25 09:28:14.326609765 +0000 UTC m=+14.090825834, 105.131909ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.332Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:14.332Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.437Z [INFO] stable state for Follower reached at 2021-08-25 09:28:14.332183087 +0000 UTC m=+14.096398996 (1 nodes), 2.58µs from start of poll, 202.409868ms from cluster start. Timeout at 2021-08-25 09:28:14.437336614 +0000 UTC m=+14.201552543, 105.153547ms after stability - testing.go:149: cluster: 2021-08-25T09:28:14.437Z [INFO] stable state for Follower reached at 2021-08-25 09:28:14.332183087 +0000 UTC m=+14.096398996 (1 nodes), 2.58µs from start of poll, 202.409868ms from cluster start. Timeout at 2021-08-25 09:28:14.437336614 +0000 UTC m=+14.201552543, 105.153547ms after stability - raft_test.go:1492: [INFO] Disconnecting Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Follower] -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to pipeline appendEntries: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="append pipeline closed" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.441Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to pipeline appendEntries: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="append pipeline closed" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.442Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: aborting pipeline replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.442Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: aborting pipeline replication: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.450Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.450Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.457Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.457Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.467Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.467Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.475Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.475Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to heartbeat to: peer=056c5c14-3305-18be-9824-f917cef0ab0e error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.482Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.482Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to appendEntries to: peer="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact: server-id=server-056c5c14-3305-18be-9824-f917cef0ab0e time=50.164197ms - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact: server-id=server-056c5c14-3305-18be-9824-f917cef0ab0e time=50.164197ms -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact quorum of nodes, stepping down - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.486Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.487Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.487Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering follower state: follower="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.487Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:14.487Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: heartbeat timeout reached, starting election: last-leader=0f74db2c-ded7-6807-fe92-1bb9c00da2f8 - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: heartbeat timeout reached, starting election: last-leader=0f74db2c-ded7-6807-fe92-1bb9c00da2f8 -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=3 - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=3 -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.497Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.540Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.540Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=3 - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=3 -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.541Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.591Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.591Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=4 - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=4 -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.592Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.600Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.600Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=4 - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=4 -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.601Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.602Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.497597213 +0000 UTC m=+14.261813122 (0 nodes), 10.358597ms from start of poll, 367.823994ms from cluster start. Timeout at 2021-08-25 09:28:14.602643671 +0000 UTC m=+14.366859640, 105.046518ms after stability - testing.go:149: cluster: 2021-08-25T09:28:14.602Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.497597213 +0000 UTC m=+14.261813122 (0 nodes), 10.358597ms from start of poll, 367.823994ms from cluster start. Timeout at 2021-08-25 09:28:14.602643671 +0000 UTC m=+14.366859640, 105.046518ms after stability -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.653Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.653Z [WARN] server-056c5c14-3305-18be-9824-f917cef0ab0e: Election timeout reached, restarting election -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=5 - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [INFO] server-056c5c14-3305-18be-9824-f917cef0ab0e: entering candidate state: node="Node at 056c5c14-3305-18be-9824-f917cef0ab0e [Candidate]" term=5 -testLoggerAdapter verbose: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" - testing.go:149: server-056c5c14-3305-18be-9824-f917cef0ab0e: 2021-08-25T09:28:14.654Z [ERROR] server-056c5c14-3305-18be-9824-f917cef0ab0e: failed to make requestVote RPC: target="{Voter server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8 0f74db2c-ded7-6807-fe92-1bb9c00da2f8}" error="failed to connect to peer: 0f74db2c-ded7-6807-fe92-1bb9c00da2f8" -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [WARN] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: Election timeout reached, restarting election -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=5 - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [INFO] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: entering candidate state: node="Node at 0f74db2c-ded7-6807-fe92-1bb9c00da2f8 [Candidate]" term=5 -testLoggerAdapter verbose: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" - testing.go:149: server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: 2021-08-25T09:28:14.675Z [ERROR] server-0f74db2c-ded7-6807-fe92-1bb9c00da2f8: failed to make requestVote RPC: target="{Voter server-056c5c14-3305-18be-9824-f917cef0ab0e 056c5c14-3305-18be-9824-f917cef0ab0e}" error="failed to connect to peer: 056c5c14-3305-18be-9824-f917cef0ab0e" ---- PASS: TestRaft_LeaderLeaseExpire (0.58s) +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:09.998Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88830c53-3729-34c0-46cc-7675dcddee4a Address:88830c53-3729-34c0-46cc-7675dcddee4a} {Suffrage:Voter ID:server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a Address:aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}]" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:09.998Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88830c53-3729-34c0-46cc-7675dcddee4a Address:88830c53-3729-34c0-46cc-7675dcddee4a} {Suffrage:Voter ID:server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a Address:aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}]" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:09.998Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering follower state: follower="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Follower]" leader= + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:09.998Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering follower state: follower="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Follower]" leader= +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:09.998Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88830c53-3729-34c0-46cc-7675dcddee4a Address:88830c53-3729-34c0-46cc-7675dcddee4a} {Suffrage:Voter ID:server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a Address:aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}]" + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:09.998Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88830c53-3729-34c0-46cc-7675dcddee4a Address:88830c53-3729-34c0-46cc-7675dcddee4a} {Suffrage:Voter ID:server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a Address:aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:09.998Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:09.998Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:09.998Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering follower state: follower="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Follower]" leader= + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:09.998Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering follower state: follower="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Follower]" leader= +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.064Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.064Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering candidate state: node="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Candidate]" term=2 + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering candidate state: node="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Candidate]" term=2 +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: election won: tally=2 + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: election won: tally=2 +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering leader state: leader="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Leader]" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering leader state: leader="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Leader]" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: added peer, starting replication: peer=server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: added peer, starting replication: peer=server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: pipelining replication: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.065Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: pipelining replication: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.170Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.06557974 +0000 UTC m=+14.331796164 (1 nodes), 66.7231ms from start of poll, 67.446332ms from cluster start. Timeout at 2021-08-25 09:31:10.170704706 +0000 UTC m=+14.436921190, 105.125026ms after stability + testing.go:149: cluster: 2021-08-25T09:31:10.170Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.06557974 +0000 UTC m=+14.331796164 (1 nodes), 66.7231ms from start of poll, 67.446332ms from cluster start. Timeout at 2021-08-25 09:31:10.170704706 +0000 UTC m=+14.436921190, 105.125026ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.176Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:10.176Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.281Z [INFO] stable state for Follower reached at 2021-08-25 09:31:10.176285781 +0000 UTC m=+14.442502165 (1 nodes), 2.04µs from start of poll, 178.152333ms from cluster start. Timeout at 2021-08-25 09:31:10.281413786 +0000 UTC m=+14.547630270, 105.128105ms after stability + testing.go:149: cluster: 2021-08-25T09:31:10.281Z [INFO] stable state for Follower reached at 2021-08-25 09:31:10.176285781 +0000 UTC m=+14.442502165 (1 nodes), 2.04µs from start of poll, 178.152333ms from cluster start. Timeout at 2021-08-25 09:31:10.281413786 +0000 UTC m=+14.547630270, 105.128105ms after stability + raft_test.go:1492: [INFO] Disconnecting Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Follower] +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.287Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to pipeline appendEntries: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="append pipeline closed" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.287Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to pipeline appendEntries: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="append pipeline closed" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.288Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: aborting pipeline replication: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.288Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: aborting pipeline replication: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.291Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to heartbeat to: peer=aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.291Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to heartbeat to: peer=aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.295Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to appendEntries to: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.295Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to appendEntries to: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.310Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to heartbeat to: peer=aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.310Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to heartbeat to: peer=aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.312Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to appendEntries to: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.312Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to appendEntries to: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.327Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to heartbeat to: peer=aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.327Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to heartbeat to: peer=aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.328Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to appendEntries to: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.328Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to appendEntries to: peer="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.331Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to contact: server-id=server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a time=50.056129ms + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.331Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to contact: server-id=server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a time=50.056129ms +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.331Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to contact quorum of nodes, stepping down + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.331Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.331Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering follower state: follower="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Follower]" leader= + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.331Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering follower state: follower="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Follower]" leader= +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.331Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:10.331Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.378Z [WARN] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: heartbeat timeout reached, starting election: last-leader=88830c53-3729-34c0-46cc-7675dcddee4a + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.378Z [WARN] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: heartbeat timeout reached, starting election: last-leader=88830c53-3729-34c0-46cc-7675dcddee4a +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.378Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering candidate state: node="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Candidate]" term=3 + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.378Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering candidate state: node="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Candidate]" term=3 +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.378Z [ERROR] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: failed to make requestVote RPC: target="{Voter server-88830c53-3729-34c0-46cc-7675dcddee4a 88830c53-3729-34c0-46cc-7675dcddee4a}" error="failed to connect to peer: 88830c53-3729-34c0-46cc-7675dcddee4a" + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.378Z [ERROR] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: failed to make requestVote RPC: target="{Voter server-88830c53-3729-34c0-46cc-7675dcddee4a 88830c53-3729-34c0-46cc-7675dcddee4a}" error="failed to connect to peer: 88830c53-3729-34c0-46cc-7675dcddee4a" +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.416Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.416Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.416Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering candidate state: node="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Candidate]" term=3 + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.416Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering candidate state: node="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Candidate]" term=3 +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.416Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to make requestVote RPC: target="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.416Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to make requestVote RPC: target="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.468Z [WARN] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: Election timeout reached, restarting election + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.468Z [WARN] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: Election timeout reached, restarting election +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.468Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering candidate state: node="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Candidate]" term=4 + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.468Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering candidate state: node="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Candidate]" term=4 +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.468Z [ERROR] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: failed to make requestVote RPC: target="{Voter server-88830c53-3729-34c0-46cc-7675dcddee4a 88830c53-3729-34c0-46cc-7675dcddee4a}" error="failed to connect to peer: 88830c53-3729-34c0-46cc-7675dcddee4a" + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.468Z [ERROR] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: failed to make requestVote RPC: target="{Voter server-88830c53-3729-34c0-46cc-7675dcddee4a 88830c53-3729-34c0-46cc-7675dcddee4a}" error="failed to connect to peer: 88830c53-3729-34c0-46cc-7675dcddee4a" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.483Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.37870431 +0000 UTC m=+14.644920714 (0 nodes), 46.739141ms from start of poll, 380.570882ms from cluster start. Timeout at 2021-08-25 09:31:10.483868512 +0000 UTC m=+14.750084996, 105.164282ms after stability + testing.go:149: cluster: 2021-08-25T09:31:10.483Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.37870431 +0000 UTC m=+14.644920714 (0 nodes), 46.739141ms from start of poll, 380.570882ms from cluster start. Timeout at 2021-08-25 09:31:10.483868512 +0000 UTC m=+14.750084996, 105.164282ms after stability +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.510Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: Election timeout reached, restarting election + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.510Z [WARN] server-88830c53-3729-34c0-46cc-7675dcddee4a: Election timeout reached, restarting election +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.510Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering candidate state: node="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Candidate]" term=4 + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.510Z [INFO] server-88830c53-3729-34c0-46cc-7675dcddee4a: entering candidate state: node="Node at 88830c53-3729-34c0-46cc-7675dcddee4a [Candidate]" term=4 +testLoggerAdapter verbose: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.511Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to make requestVote RPC: target="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" + testing.go:149: server-88830c53-3729-34c0-46cc-7675dcddee4a: 2021-08-25T09:31:10.511Z [ERROR] server-88830c53-3729-34c0-46cc-7675dcddee4a: failed to make requestVote RPC: target="{Voter server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a}" error="failed to connect to peer: aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a" +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.550Z [WARN] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: Election timeout reached, restarting election + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.550Z [WARN] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: Election timeout reached, restarting election +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.550Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering candidate state: node="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Candidate]" term=5 + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.550Z [INFO] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: entering candidate state: node="Node at aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a [Candidate]" term=5 +testLoggerAdapter verbose: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.551Z [ERROR] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: failed to make requestVote RPC: target="{Voter server-88830c53-3729-34c0-46cc-7675dcddee4a 88830c53-3729-34c0-46cc-7675dcddee4a}" error="failed to connect to peer: 88830c53-3729-34c0-46cc-7675dcddee4a" + testing.go:149: server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: 2021-08-25T09:31:10.551Z [ERROR] server-aa99f1cd-8408-ba94-fb30-1eb87a4f3c2a: failed to make requestVote RPC: target="{Voter server-88830c53-3729-34c0-46cc-7675dcddee4a 88830c53-3729-34c0-46cc-7675dcddee4a}" error="failed to connect to peer: 88830c53-3729-34c0-46cc-7675dcddee4a" +--- PASS: TestRaft_LeaderLeaseExpire (0.59s) === RUN TestRaft_Barrier -testLoggerAdapter verbose: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.711Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" - testing.go:149: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.711Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" -testLoggerAdapter verbose: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.712Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: entering follower state: follower="Node at 441b9051-90cb-b2b4-5a24-a088a275ab3e [Follower]" leader= - testing.go:149: server-441b9051-90cb-b2b4-5a24-a088a275ab3e: 2021-08-25T09:28:14.712Z [INFO] server-441b9051-90cb-b2b4-5a24-a088a275ab3e: entering follower state: follower="Node at 441b9051-90cb-b2b4-5a24-a088a275ab3e [Follower]" leader= -testLoggerAdapter verbose: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" - testing.go:149: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" -testLoggerAdapter verbose: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: entering follower state: follower="Node at 4598815a-3bc5-63dc-d49a-21e462306ae0 [Follower]" leader= - testing.go:149: server-4598815a-3bc5-63dc-d49a-21e462306ae0: 2021-08-25T09:28:14.712Z [INFO] server-4598815a-3bc5-63dc-d49a-21e462306ae0: entering follower state: follower="Node at 4598815a-3bc5-63dc-d49a-21e462306ae0 [Follower]" leader= -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.712Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.712Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-441b9051-90cb-b2b4-5a24-a088a275ab3e Address:441b9051-90cb-b2b4-5a24-a088a275ab3e} {Suffrage:Voter ID:server-4598815a-3bc5-63dc-d49a-21e462306ae0 Address:4598815a-3bc5-63dc-d49a-21e462306ae0} {Suffrage:Voter ID:server-c5e140b4-a14d-3987-b3b9-79f8927f619c Address:c5e140b4-a14d-3987-b3b9-79f8927f619c}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.712Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.713Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering follower state: follower="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:14.712Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.713Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering follower state: follower="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Follower]" leader= -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [WARN] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [WARN] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering candidate state: node="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Candidate]" term=2 - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering candidate state: node="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Candidate]" term=2 -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: election won: tally=2 - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: election won: tally=2 -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering leader state: leader="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Leader]" - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: entering leader state: leader="Node at c5e140b4-a14d-3987-b3b9-79f8927f619c [Leader]" -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-441b9051-90cb-b2b4-5a24-a088a275ab3e - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-441b9051-90cb-b2b4-5a24-a088a275ab3e -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-4598815a-3bc5-63dc-d49a-21e462306ae0 - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: added peer, starting replication: peer=server-4598815a-3bc5-63dc-d49a-21e462306ae0 -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.774Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: pipelining replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.879Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.774521856 +0000 UTC m=+14.538737745 (1 nodes), 61.43865ms from start of poll, 62.847699ms from cluster start. Timeout at 2021-08-25 09:28:14.879544316 +0000 UTC m=+14.643760245, 105.0225ms after stability - testing.go:149: cluster: 2021-08-25T09:28:14.879Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.774521856 +0000 UTC m=+14.538737745 (1 nodes), 61.43865ms from start of poll, 62.847699ms from cluster start. Timeout at 2021-08-25 09:28:14.879544316 +0000 UTC m=+14.643760245, 105.0225ms after stability -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-4598815a-3bc5-63dc-d49a-21e462306ae0 4598815a-3bc5-63dc-d49a-21e462306ae0}" -testLoggerAdapter verbose: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" - testing.go:149: server-c5e140b4-a14d-3987-b3b9-79f8927f619c: 2021-08-25T09:28:14.894Z [INFO] server-c5e140b4-a14d-3987-b3b9-79f8927f619c: aborting pipeline replication: peer="{Voter server-441b9051-90cb-b2b4-5a24-a088a275ab3e 441b9051-90cb-b2b4-5a24-a088a275ab3e}" +testLoggerAdapter verbose: server-154e89a4-3aab-2684-8a41-72b8848dc531: 2021-08-25T09:31:10.592Z [INFO] server-154e89a4-3aab-2684-8a41-72b8848dc531: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-154e89a4-3aab-2684-8a41-72b8848dc531 Address:154e89a4-3aab-2684-8a41-72b8848dc531} {Suffrage:Voter ID:server-e878387b-410a-e289-4e3a-06aac34f1966 Address:e878387b-410a-e289-4e3a-06aac34f1966} {Suffrage:Voter ID:server-82e396a7-fc2a-4c62-9a44-6e2b16874476 Address:82e396a7-fc2a-4c62-9a44-6e2b16874476}]" + testing.go:149: server-154e89a4-3aab-2684-8a41-72b8848dc531: 2021-08-25T09:31:10.592Z [INFO] server-154e89a4-3aab-2684-8a41-72b8848dc531: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-154e89a4-3aab-2684-8a41-72b8848dc531 Address:154e89a4-3aab-2684-8a41-72b8848dc531} {Suffrage:Voter ID:server-e878387b-410a-e289-4e3a-06aac34f1966 Address:e878387b-410a-e289-4e3a-06aac34f1966} {Suffrage:Voter ID:server-82e396a7-fc2a-4c62-9a44-6e2b16874476 Address:82e396a7-fc2a-4c62-9a44-6e2b16874476}]" +testLoggerAdapter verbose: server-154e89a4-3aab-2684-8a41-72b8848dc531: 2021-08-25T09:31:10.592Z [INFO] server-154e89a4-3aab-2684-8a41-72b8848dc531: entering follower state: follower="Node at 154e89a4-3aab-2684-8a41-72b8848dc531 [Follower]" leader= + testing.go:149: server-154e89a4-3aab-2684-8a41-72b8848dc531: 2021-08-25T09:31:10.592Z [INFO] server-154e89a4-3aab-2684-8a41-72b8848dc531: entering follower state: follower="Node at 154e89a4-3aab-2684-8a41-72b8848dc531 [Follower]" leader= +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.593Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-154e89a4-3aab-2684-8a41-72b8848dc531 Address:154e89a4-3aab-2684-8a41-72b8848dc531} {Suffrage:Voter ID:server-e878387b-410a-e289-4e3a-06aac34f1966 Address:e878387b-410a-e289-4e3a-06aac34f1966} {Suffrage:Voter ID:server-82e396a7-fc2a-4c62-9a44-6e2b16874476 Address:82e396a7-fc2a-4c62-9a44-6e2b16874476}]" + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.593Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-154e89a4-3aab-2684-8a41-72b8848dc531 Address:154e89a4-3aab-2684-8a41-72b8848dc531} {Suffrage:Voter ID:server-e878387b-410a-e289-4e3a-06aac34f1966 Address:e878387b-410a-e289-4e3a-06aac34f1966} {Suffrage:Voter ID:server-82e396a7-fc2a-4c62-9a44-6e2b16874476 Address:82e396a7-fc2a-4c62-9a44-6e2b16874476}]" +testLoggerAdapter verbose: server-82e396a7-fc2a-4c62-9a44-6e2b16874476: 2021-08-25T09:31:10.593Z [INFO] server-82e396a7-fc2a-4c62-9a44-6e2b16874476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-154e89a4-3aab-2684-8a41-72b8848dc531 Address:154e89a4-3aab-2684-8a41-72b8848dc531} {Suffrage:Voter ID:server-e878387b-410a-e289-4e3a-06aac34f1966 Address:e878387b-410a-e289-4e3a-06aac34f1966} {Suffrage:Voter ID:server-82e396a7-fc2a-4c62-9a44-6e2b16874476 Address:82e396a7-fc2a-4c62-9a44-6e2b16874476}]" + testing.go:149: server-82e396a7-fc2a-4c62-9a44-6e2b16874476: 2021-08-25T09:31:10.593Z [INFO] server-82e396a7-fc2a-4c62-9a44-6e2b16874476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-154e89a4-3aab-2684-8a41-72b8848dc531 Address:154e89a4-3aab-2684-8a41-72b8848dc531} {Suffrage:Voter ID:server-e878387b-410a-e289-4e3a-06aac34f1966 Address:e878387b-410a-e289-4e3a-06aac34f1966} {Suffrage:Voter ID:server-82e396a7-fc2a-4c62-9a44-6e2b16874476 Address:82e396a7-fc2a-4c62-9a44-6e2b16874476}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.593Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:10.593Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.593Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: entering follower state: follower="Node at e878387b-410a-e289-4e3a-06aac34f1966 [Follower]" leader= +testLoggerAdapter verbose: server-82e396a7-fc2a-4c62-9a44-6e2b16874476: 2021-08-25T09:31:10.593Z [INFO] server-82e396a7-fc2a-4c62-9a44-6e2b16874476: entering follower state: follower="Node at 82e396a7-fc2a-4c62-9a44-6e2b16874476 [Follower]" leader= + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.593Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: entering follower state: follower="Node at e878387b-410a-e289-4e3a-06aac34f1966 [Follower]" leader= + testing.go:149: server-82e396a7-fc2a-4c62-9a44-6e2b16874476: 2021-08-25T09:31:10.593Z [INFO] server-82e396a7-fc2a-4c62-9a44-6e2b16874476: entering follower state: follower="Node at 82e396a7-fc2a-4c62-9a44-6e2b16874476 [Follower]" leader= +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [WARN] server-e878387b-410a-e289-4e3a-06aac34f1966: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [WARN] server-e878387b-410a-e289-4e3a-06aac34f1966: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: entering candidate state: node="Node at e878387b-410a-e289-4e3a-06aac34f1966 [Candidate]" term=2 + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: entering candidate state: node="Node at e878387b-410a-e289-4e3a-06aac34f1966 [Candidate]" term=2 +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: election won: tally=2 + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: election won: tally=2 +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: entering leader state: leader="Node at e878387b-410a-e289-4e3a-06aac34f1966 [Leader]" + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: entering leader state: leader="Node at e878387b-410a-e289-4e3a-06aac34f1966 [Leader]" +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: added peer, starting replication: peer=server-154e89a4-3aab-2684-8a41-72b8848dc531 + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.659Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: added peer, starting replication: peer=server-154e89a4-3aab-2684-8a41-72b8848dc531 +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.660Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: added peer, starting replication: peer=server-82e396a7-fc2a-4c62-9a44-6e2b16874476 + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.660Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: added peer, starting replication: peer=server-82e396a7-fc2a-4c62-9a44-6e2b16874476 +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.660Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: pipelining replication: peer="{Voter server-82e396a7-fc2a-4c62-9a44-6e2b16874476 82e396a7-fc2a-4c62-9a44-6e2b16874476}" + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.660Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: pipelining replication: peer="{Voter server-82e396a7-fc2a-4c62-9a44-6e2b16874476 82e396a7-fc2a-4c62-9a44-6e2b16874476}" +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.660Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: pipelining replication: peer="{Voter server-154e89a4-3aab-2684-8a41-72b8848dc531 154e89a4-3aab-2684-8a41-72b8848dc531}" + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.660Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: pipelining replication: peer="{Voter server-154e89a4-3aab-2684-8a41-72b8848dc531 154e89a4-3aab-2684-8a41-72b8848dc531}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.765Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.660110845 +0000 UTC m=+14.926327269 (1 nodes), 66.588093ms from start of poll, 67.743005ms from cluster start. Timeout at 2021-08-25 09:31:10.765230411 +0000 UTC m=+15.031446895, 105.119626ms after stability + testing.go:149: cluster: 2021-08-25T09:31:10.765Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.660110845 +0000 UTC m=+14.926327269 (1 nodes), 66.588093ms from start of poll, 67.743005ms from cluster start. Timeout at 2021-08-25 09:31:10.765230411 +0000 UTC m=+15.031446895, 105.119626ms after stability +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.778Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: aborting pipeline replication: peer="{Voter server-82e396a7-fc2a-4c62-9a44-6e2b16874476 82e396a7-fc2a-4c62-9a44-6e2b16874476}" + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.778Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: aborting pipeline replication: peer="{Voter server-82e396a7-fc2a-4c62-9a44-6e2b16874476 82e396a7-fc2a-4c62-9a44-6e2b16874476}" +testLoggerAdapter verbose: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.778Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: aborting pipeline replication: peer="{Voter server-154e89a4-3aab-2684-8a41-72b8848dc531 154e89a4-3aab-2684-8a41-72b8848dc531}" + testing.go:149: server-e878387b-410a-e289-4e3a-06aac34f1966: 2021-08-25T09:31:10.778Z [INFO] server-e878387b-410a-e289-4e3a-06aac34f1966: aborting pipeline replication: peer="{Voter server-154e89a4-3aab-2684-8a41-72b8848dc531 154e89a4-3aab-2684-8a41-72b8848dc531}" --- PASS: TestRaft_Barrier (0.19s) === RUN TestRaft_VerifyLeader -testLoggerAdapter verbose: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" - testing.go:149: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" -testLoggerAdapter verbose: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: entering follower state: follower="Node at d77979d5-3e0c-27a6-5587-5e3136457df4 [Follower]" leader= - testing.go:149: server-d77979d5-3e0c-27a6-5587-5e3136457df4: 2021-08-25T09:28:14.898Z [INFO] server-d77979d5-3e0c-27a6-5587-5e3136457df4: entering follower state: follower="Node at d77979d5-3e0c-27a6-5587-5e3136457df4 [Follower]" leader= -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering follower state: follower="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Follower]" leader= - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.899Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering follower state: follower="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Follower]" leader= -testLoggerAdapter verbose: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" - testing.go:149: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d77979d5-3e0c-27a6-5587-5e3136457df4 Address:d77979d5-3e0c-27a6-5587-5e3136457df4} {Suffrage:Voter ID:server-736cb78a-a94c-bf71-c6c9-9b33575daa09 Address:736cb78a-a94c-bf71-c6c9-9b33575daa09} {Suffrage:Voter ID:server-bf526228-a556-793d-c54e-8a62b64cc54b Address:bf526228-a556-793d-c54e-8a62b64cc54b}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:14.899Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:14.899Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: entering follower state: follower="Node at bf526228-a556-793d-c54e-8a62b64cc54b [Follower]" leader= - testing.go:149: server-bf526228-a556-793d-c54e-8a62b64cc54b: 2021-08-25T09:28:14.899Z [INFO] server-bf526228-a556-793d-c54e-8a62b64cc54b: entering follower state: follower="Node at bf526228-a556-793d-c54e-8a62b64cc54b [Follower]" leader= -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [WARN] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [WARN] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering candidate state: node="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Candidate]" term=2 - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering candidate state: node="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Candidate]" term=2 -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: election won: tally=2 - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: election won: tally=2 -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering leader state: leader="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Leader]" - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: entering leader state: leader="Node at 736cb78a-a94c-bf71-c6c9-9b33575daa09 [Leader]" -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-d77979d5-3e0c-27a6-5587-5e3136457df4 - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-d77979d5-3e0c-27a6-5587-5e3136457df4 -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-bf526228-a556-793d-c54e-8a62b64cc54b - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: added peer, starting replication: peer=server-bf526228-a556-793d-c54e-8a62b64cc54b -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:14.951Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: pipelining replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.056Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.951724287 +0000 UTC m=+14.715940156 (1 nodes), 51.959671ms from start of poll, 53.233292ms from cluster start. Timeout at 2021-08-25 09:28:15.056841698 +0000 UTC m=+14.821057647, 105.117491ms after stability - testing.go:149: cluster: 2021-08-25T09:28:15.056Z [INFO] stable state for Leader reached at 2021-08-25 09:28:14.951724287 +0000 UTC m=+14.715940156 (1 nodes), 51.959671ms from start of poll, 53.233292ms from cluster start. Timeout at 2021-08-25 09:28:15.056841698 +0000 UTC m=+14.821057647, 105.117491ms after stability -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-bf526228-a556-793d-c54e-8a62b64cc54b bf526228-a556-793d-c54e-8a62b64cc54b}" -testLoggerAdapter verbose: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" - testing.go:149: server-736cb78a-a94c-bf71-c6c9-9b33575daa09: 2021-08-25T09:28:15.057Z [INFO] server-736cb78a-a94c-bf71-c6c9-9b33575daa09: aborting pipeline replication: peer="{Voter server-d77979d5-3e0c-27a6-5587-5e3136457df4 d77979d5-3e0c-27a6-5587-5e3136457df4}" ---- PASS: TestRaft_VerifyLeader (0.16s) +testLoggerAdapter verbose: server-a608c8f4-ef70-7540-bd39-ebb870cca270: 2021-08-25T09:31:10.782Z [INFO] server-a608c8f4-ef70-7540-bd39-ebb870cca270: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c8f4-ef70-7540-bd39-ebb870cca270 Address:a608c8f4-ef70-7540-bd39-ebb870cca270} {Suffrage:Voter ID:server-6b40de83-1845-9d67-1b6c-7d3ce573a402 Address:6b40de83-1845-9d67-1b6c-7d3ce573a402} {Suffrage:Voter ID:server-78c42101-5b4f-cfb9-118b-cf5560acad66 Address:78c42101-5b4f-cfb9-118b-cf5560acad66}]" + testing.go:149: server-a608c8f4-ef70-7540-bd39-ebb870cca270: 2021-08-25T09:31:10.782Z [INFO] server-a608c8f4-ef70-7540-bd39-ebb870cca270: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c8f4-ef70-7540-bd39-ebb870cca270 Address:a608c8f4-ef70-7540-bd39-ebb870cca270} {Suffrage:Voter ID:server-6b40de83-1845-9d67-1b6c-7d3ce573a402 Address:6b40de83-1845-9d67-1b6c-7d3ce573a402} {Suffrage:Voter ID:server-78c42101-5b4f-cfb9-118b-cf5560acad66 Address:78c42101-5b4f-cfb9-118b-cf5560acad66}]" +testLoggerAdapter verbose: server-a608c8f4-ef70-7540-bd39-ebb870cca270: 2021-08-25T09:31:10.782Z [INFO] server-a608c8f4-ef70-7540-bd39-ebb870cca270: entering follower state: follower="Node at a608c8f4-ef70-7540-bd39-ebb870cca270 [Follower]" leader= + testing.go:149: server-a608c8f4-ef70-7540-bd39-ebb870cca270: 2021-08-25T09:31:10.782Z [INFO] server-a608c8f4-ef70-7540-bd39-ebb870cca270: entering follower state: follower="Node at a608c8f4-ef70-7540-bd39-ebb870cca270 [Follower]" leader= +testLoggerAdapter verbose: server-6b40de83-1845-9d67-1b6c-7d3ce573a402: 2021-08-25T09:31:10.782Z [INFO] server-6b40de83-1845-9d67-1b6c-7d3ce573a402: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c8f4-ef70-7540-bd39-ebb870cca270 Address:a608c8f4-ef70-7540-bd39-ebb870cca270} {Suffrage:Voter ID:server-6b40de83-1845-9d67-1b6c-7d3ce573a402 Address:6b40de83-1845-9d67-1b6c-7d3ce573a402} {Suffrage:Voter ID:server-78c42101-5b4f-cfb9-118b-cf5560acad66 Address:78c42101-5b4f-cfb9-118b-cf5560acad66}]" + testing.go:149: server-6b40de83-1845-9d67-1b6c-7d3ce573a402: 2021-08-25T09:31:10.782Z [INFO] server-6b40de83-1845-9d67-1b6c-7d3ce573a402: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c8f4-ef70-7540-bd39-ebb870cca270 Address:a608c8f4-ef70-7540-bd39-ebb870cca270} {Suffrage:Voter ID:server-6b40de83-1845-9d67-1b6c-7d3ce573a402 Address:6b40de83-1845-9d67-1b6c-7d3ce573a402} {Suffrage:Voter ID:server-78c42101-5b4f-cfb9-118b-cf5560acad66 Address:78c42101-5b4f-cfb9-118b-cf5560acad66}]" +testLoggerAdapter verbose: server-6b40de83-1845-9d67-1b6c-7d3ce573a402: 2021-08-25T09:31:10.782Z [INFO] server-6b40de83-1845-9d67-1b6c-7d3ce573a402: entering follower state: follower="Node at 6b40de83-1845-9d67-1b6c-7d3ce573a402 [Follower]" leader= + testing.go:149: server-6b40de83-1845-9d67-1b6c-7d3ce573a402: 2021-08-25T09:31:10.782Z [INFO] server-6b40de83-1845-9d67-1b6c-7d3ce573a402: entering follower state: follower="Node at 6b40de83-1845-9d67-1b6c-7d3ce573a402 [Follower]" leader= +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.783Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c8f4-ef70-7540-bd39-ebb870cca270 Address:a608c8f4-ef70-7540-bd39-ebb870cca270} {Suffrage:Voter ID:server-6b40de83-1845-9d67-1b6c-7d3ce573a402 Address:6b40de83-1845-9d67-1b6c-7d3ce573a402} {Suffrage:Voter ID:server-78c42101-5b4f-cfb9-118b-cf5560acad66 Address:78c42101-5b4f-cfb9-118b-cf5560acad66}]" + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.783Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c8f4-ef70-7540-bd39-ebb870cca270 Address:a608c8f4-ef70-7540-bd39-ebb870cca270} {Suffrage:Voter ID:server-6b40de83-1845-9d67-1b6c-7d3ce573a402 Address:6b40de83-1845-9d67-1b6c-7d3ce573a402} {Suffrage:Voter ID:server-78c42101-5b4f-cfb9-118b-cf5560acad66 Address:78c42101-5b4f-cfb9-118b-cf5560acad66}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.783Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:10.783Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.783Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: entering follower state: follower="Node at 78c42101-5b4f-cfb9-118b-cf5560acad66 [Follower]" leader= + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.783Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: entering follower state: follower="Node at 78c42101-5b4f-cfb9-118b-cf5560acad66 [Follower]" leader= +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [WARN] server-78c42101-5b4f-cfb9-118b-cf5560acad66: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [WARN] server-78c42101-5b4f-cfb9-118b-cf5560acad66: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: entering candidate state: node="Node at 78c42101-5b4f-cfb9-118b-cf5560acad66 [Candidate]" term=2 + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: entering candidate state: node="Node at 78c42101-5b4f-cfb9-118b-cf5560acad66 [Candidate]" term=2 +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: election won: tally=2 + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: election won: tally=2 +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: entering leader state: leader="Node at 78c42101-5b4f-cfb9-118b-cf5560acad66 [Leader]" + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: entering leader state: leader="Node at 78c42101-5b4f-cfb9-118b-cf5560acad66 [Leader]" +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: added peer, starting replication: peer=server-a608c8f4-ef70-7540-bd39-ebb870cca270 + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.860Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: added peer, starting replication: peer=server-a608c8f4-ef70-7540-bd39-ebb870cca270 +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.861Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: added peer, starting replication: peer=server-6b40de83-1845-9d67-1b6c-7d3ce573a402 + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.861Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: added peer, starting replication: peer=server-6b40de83-1845-9d67-1b6c-7d3ce573a402 +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.861Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: pipelining replication: peer="{Voter server-6b40de83-1845-9d67-1b6c-7d3ce573a402 6b40de83-1845-9d67-1b6c-7d3ce573a402}" + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.861Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: pipelining replication: peer="{Voter server-6b40de83-1845-9d67-1b6c-7d3ce573a402 6b40de83-1845-9d67-1b6c-7d3ce573a402}" +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.861Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: pipelining replication: peer="{Voter server-a608c8f4-ef70-7540-bd39-ebb870cca270 a608c8f4-ef70-7540-bd39-ebb870cca270}" + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.861Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: pipelining replication: peer="{Voter server-a608c8f4-ef70-7540-bd39-ebb870cca270 a608c8f4-ef70-7540-bd39-ebb870cca270}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.966Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.86107815 +0000 UTC m=+15.127294594 (1 nodes), 77.842993ms from start of poll, 78.890255ms from cluster start. Timeout at 2021-08-25 09:31:10.966182378 +0000 UTC m=+15.232398822, 105.104228ms after stability + testing.go:149: cluster: 2021-08-25T09:31:10.966Z [INFO] stable state for Leader reached at 2021-08-25 09:31:10.86107815 +0000 UTC m=+15.127294594 (1 nodes), 77.842993ms from start of poll, 78.890255ms from cluster start. Timeout at 2021-08-25 09:31:10.966182378 +0000 UTC m=+15.232398822, 105.104228ms after stability +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.966Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: aborting pipeline replication: peer="{Voter server-6b40de83-1845-9d67-1b6c-7d3ce573a402 6b40de83-1845-9d67-1b6c-7d3ce573a402}" + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.966Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: aborting pipeline replication: peer="{Voter server-6b40de83-1845-9d67-1b6c-7d3ce573a402 6b40de83-1845-9d67-1b6c-7d3ce573a402}" +testLoggerAdapter verbose: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.966Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: aborting pipeline replication: peer="{Voter server-a608c8f4-ef70-7540-bd39-ebb870cca270 a608c8f4-ef70-7540-bd39-ebb870cca270}" + testing.go:149: server-78c42101-5b4f-cfb9-118b-cf5560acad66: 2021-08-25T09:31:10.966Z [INFO] server-78c42101-5b4f-cfb9-118b-cf5560acad66: aborting pipeline replication: peer="{Voter server-a608c8f4-ef70-7540-bd39-ebb870cca270 a608c8f4-ef70-7540-bd39-ebb870cca270}" +--- PASS: TestRaft_VerifyLeader (0.19s) === RUN TestRaft_VerifyLeader_Single -testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-466af8da-ea3c-48eb-2e8c-965c4b963578 Address:466af8da-ea3c-48eb-2e8c-965c4b963578}]" - testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-466af8da-ea3c-48eb-2e8c-965c4b963578 Address:466af8da-ea3c-48eb-2e8c-965c4b963578}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.060Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:15.060Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering follower state: follower="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Follower]" leader= - testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.060Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering follower state: follower="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Follower]" leader= -testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [WARN] server-466af8da-ea3c-48eb-2e8c-965c4b963578: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [WARN] server-466af8da-ea3c-48eb-2e8c-965c4b963578: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering candidate state: node="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Candidate]" term=2 - testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering candidate state: node="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Candidate]" term=2 -testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: election won: tally=1 - testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: election won: tally=1 -testLoggerAdapter verbose: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering leader state: leader="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Leader]" - testing.go:149: server-466af8da-ea3c-48eb-2e8c-965c4b963578: 2021-08-25T09:28:15.134Z [INFO] server-466af8da-ea3c-48eb-2e8c-965c4b963578: entering leader state: leader="Node at 466af8da-ea3c-48eb-2e8c-965c4b963578 [Leader]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.240Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.134869726 +0000 UTC m=+14.899085615 (1 nodes), 74.666941ms from start of poll, 75.095361ms from cluster start. Timeout at 2021-08-25 09:28:15.240013934 +0000 UTC m=+15.004229863, 105.144248ms after stability - testing.go:149: cluster: 2021-08-25T09:28:15.240Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.134869726 +0000 UTC m=+14.899085615 (1 nodes), 74.666941ms from start of poll, 75.095361ms from cluster start. Timeout at 2021-08-25 09:28:15.240013934 +0000 UTC m=+15.004229863, 105.144248ms after stability ---- PASS: TestRaft_VerifyLeader_Single (0.18s) +testLoggerAdapter verbose: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:10.969Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f60b581-6195-885e-c190-3adde8d05dfe Address:9f60b581-6195-885e-c190-3adde8d05dfe}]" + testing.go:149: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:10.969Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f60b581-6195-885e-c190-3adde8d05dfe Address:9f60b581-6195-885e-c190-3adde8d05dfe}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:10.969Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:10.969Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:10.969Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: entering follower state: follower="Node at 9f60b581-6195-885e-c190-3adde8d05dfe [Follower]" leader= + testing.go:149: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:10.969Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: entering follower state: follower="Node at 9f60b581-6195-885e-c190-3adde8d05dfe [Follower]" leader= +testLoggerAdapter verbose: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.046Z [WARN] server-9f60b581-6195-885e-c190-3adde8d05dfe: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.046Z [WARN] server-9f60b581-6195-885e-c190-3adde8d05dfe: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.046Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: entering candidate state: node="Node at 9f60b581-6195-885e-c190-3adde8d05dfe [Candidate]" term=2 + testing.go:149: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.046Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: entering candidate state: node="Node at 9f60b581-6195-885e-c190-3adde8d05dfe [Candidate]" term=2 +testLoggerAdapter verbose: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.047Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: election won: tally=1 + testing.go:149: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.047Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: election won: tally=1 +testLoggerAdapter verbose: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.047Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: entering leader state: leader="Node at 9f60b581-6195-885e-c190-3adde8d05dfe [Leader]" + testing.go:149: server-9f60b581-6195-885e-c190-3adde8d05dfe: 2021-08-25T09:31:11.047Z [INFO] server-9f60b581-6195-885e-c190-3adde8d05dfe: entering leader state: leader="Node at 9f60b581-6195-885e-c190-3adde8d05dfe [Leader]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.152Z [INFO] stable state for Leader reached at 2021-08-25 09:31:11.04723531 +0000 UTC m=+15.313451714 (1 nodes), 77.616135ms from start of poll, 78.089351ms from cluster start. Timeout at 2021-08-25 09:31:11.152368415 +0000 UTC m=+15.418584899, 105.133185ms after stability + testing.go:149: cluster: 2021-08-25T09:31:11.152Z [INFO] stable state for Leader reached at 2021-08-25 09:31:11.04723531 +0000 UTC m=+15.313451714 (1 nodes), 77.616135ms from start of poll, 78.089351ms from cluster start. Timeout at 2021-08-25 09:31:11.152368415 +0000 UTC m=+15.418584899, 105.133185ms after stability +--- PASS: TestRaft_VerifyLeader_Single (0.19s) === RUN TestRaft_VerifyLeader_Fail -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.242Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= -testLoggerAdapter verbose: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" - testing.go:149: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869a35fe-e12f-49f8-8547-bdfd07dfd703 Address:869a35fe-e12f-49f8-8547-bdfd07dfd703} {Suffrage:Voter ID:server-cc036e16-8377-4c65-99df-f407746bc6a7 Address:cc036e16-8377-4c65-99df-f407746bc6a7}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.243Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:15.243Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: entering follower state: follower="Node at cc036e16-8377-4c65-99df-f407746bc6a7 [Follower]" leader= - testing.go:149: server-cc036e16-8377-4c65-99df-f407746bc6a7: 2021-08-25T09:28:15.243Z [INFO] server-cc036e16-8377-4c65-99df-f407746bc6a7: entering follower state: follower="Node at cc036e16-8377-4c65-99df-f407746bc6a7 [Follower]" leader= -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering candidate state: node="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Candidate]" term=2 - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.299Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering candidate state: node="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Candidate]" term=2 -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: election won: tally=2 - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: election won: tally=2 -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering leader state: leader="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Leader]" - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering leader state: leader="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Leader]" -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: added peer, starting replication: peer=server-cc036e16-8377-4c65-99df-f407746bc6a7 - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: added peer, starting replication: peer=server-cc036e16-8377-4c65-99df-f407746bc6a7 -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: pipelining replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.300Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: pipelining replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.405Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.300281893 +0000 UTC m=+15.064497802 (1 nodes), 57.071696ms from start of poll, 57.906239ms from cluster start. Timeout at 2021-08-25 09:28:15.405409982 +0000 UTC m=+15.169625911, 105.128109ms after stability - testing.go:149: cluster: 2021-08-25T09:28:15.405Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.300281893 +0000 UTC m=+15.064497802 (1 nodes), 57.071696ms from start of poll, 57.906239ms from cluster start. Timeout at 2021-08-25 09:28:15.405409982 +0000 UTC m=+15.169625911, 105.128109ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.405Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:15.405Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.510Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.405696135 +0000 UTC m=+15.169912024 (1 nodes), 2.159µs from start of poll, 163.320461ms from cluster start. Timeout at 2021-08-25 09:28:15.510831224 +0000 UTC m=+15.275047253, 105.135229ms after stability - testing.go:149: cluster: 2021-08-25T09:28:15.510Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.405696135 +0000 UTC m=+15.169912024 (1 nodes), 2.159µs from start of poll, 163.320461ms from cluster start. Timeout at 2021-08-25 09:28:15.510831224 +0000 UTC m=+15.275047253, 105.135229ms after stability -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: new leader elected, stepping down - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [WARN] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: new leader elected, stepping down -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: entering follower state: follower="Node at 869a35fe-e12f-49f8-8547-bdfd07dfd703 [Follower]" leader= -testLoggerAdapter verbose: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: aborting pipeline replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" - testing.go:149: server-869a35fe-e12f-49f8-8547-bdfd07dfd703: 2021-08-25T09:28:15.511Z [INFO] server-869a35fe-e12f-49f8-8547-bdfd07dfd703: aborting pipeline replication: peer="{Voter server-cc036e16-8377-4c65-99df-f407746bc6a7 cc036e16-8377-4c65-99df-f407746bc6a7}" ---- PASS: TestRaft_VerifyLeader_Fail (0.27s) +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.156Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-069231f4-d283-a901-d9f9-5a87c397b0be Address:069231f4-d283-a901-d9f9-5a87c397b0be} {Suffrage:Voter ID:server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 Address:4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}]" + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.156Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-069231f4-d283-a901-d9f9-5a87c397b0be Address:069231f4-d283-a901-d9f9-5a87c397b0be} {Suffrage:Voter ID:server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 Address:4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}]" +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.156Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering follower state: follower="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Follower]" leader= + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.156Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering follower state: follower="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Follower]" leader= +testLoggerAdapter verbose: server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: 2021-08-25T09:31:11.156Z [INFO] server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-069231f4-d283-a901-d9f9-5a87c397b0be Address:069231f4-d283-a901-d9f9-5a87c397b0be} {Suffrage:Voter ID:server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 Address:4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}]" + testing.go:149: server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: 2021-08-25T09:31:11.156Z [INFO] server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-069231f4-d283-a901-d9f9-5a87c397b0be Address:069231f4-d283-a901-d9f9-5a87c397b0be} {Suffrage:Voter ID:server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 Address:4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.156Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:11.156Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: 2021-08-25T09:31:11.156Z [INFO] server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: entering follower state: follower="Node at 4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 [Follower]" leader= + testing.go:149: server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: 2021-08-25T09:31:11.156Z [INFO] server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9: entering follower state: follower="Node at 4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 [Follower]" leader= +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [WARN] server-069231f4-d283-a901-d9f9-5a87c397b0be: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [WARN] server-069231f4-d283-a901-d9f9-5a87c397b0be: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering candidate state: node="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Candidate]" term=2 + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering candidate state: node="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Candidate]" term=2 +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: election won: tally=2 + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: election won: tally=2 +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering leader state: leader="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Leader]" + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.228Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering leader state: leader="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Leader]" +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.229Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: added peer, starting replication: peer=server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.229Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: added peer, starting replication: peer=server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.229Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: pipelining replication: peer="{Voter server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}" + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.229Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: pipelining replication: peer="{Voter server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.334Z [INFO] stable state for Leader reached at 2021-08-25 09:31:11.229073616 +0000 UTC m=+15.495290040 (1 nodes), 72.441942ms from start of poll, 73.239367ms from cluster start. Timeout at 2021-08-25 09:31:11.334252137 +0000 UTC m=+15.600468601, 105.178561ms after stability + testing.go:149: cluster: 2021-08-25T09:31:11.334Z [INFO] stable state for Leader reached at 2021-08-25 09:31:11.229073616 +0000 UTC m=+15.495290040 (1 nodes), 72.441942ms from start of poll, 73.239367ms from cluster start. Timeout at 2021-08-25 09:31:11.334252137 +0000 UTC m=+15.600468601, 105.178561ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.334Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:11.334Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.439Z [INFO] stable state for Follower reached at 2021-08-25 09:31:11.334634061 +0000 UTC m=+15.600850505 (1 nodes), 2.44µs from start of poll, 178.799832ms from cluster start. Timeout at 2021-08-25 09:31:11.439746928 +0000 UTC m=+15.705963352, 105.112847ms after stability + testing.go:149: cluster: 2021-08-25T09:31:11.439Z [INFO] stable state for Follower reached at 2021-08-25 09:31:11.334634061 +0000 UTC m=+15.600850505 (1 nodes), 2.44µs from start of poll, 178.799832ms from cluster start. Timeout at 2021-08-25 09:31:11.439746928 +0000 UTC m=+15.705963352, 105.112847ms after stability +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.440Z [WARN] server-069231f4-d283-a901-d9f9-5a87c397b0be: new leader elected, stepping down + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.440Z [WARN] server-069231f4-d283-a901-d9f9-5a87c397b0be: new leader elected, stepping down +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.440Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering follower state: follower="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Follower]" leader= + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.440Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: entering follower state: follower="Node at 069231f4-d283-a901-d9f9-5a87c397b0be [Follower]" leader= +testLoggerAdapter verbose: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.440Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: aborting pipeline replication: peer="{Voter server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}" + testing.go:149: server-069231f4-d283-a901-d9f9-5a87c397b0be: 2021-08-25T09:31:11.440Z [INFO] server-069231f4-d283-a901-d9f9-5a87c397b0be: aborting pipeline replication: peer="{Voter server-4c8c4a3b-e938-1e01-5360-9e6bdbc562e9 4c8c4a3b-e938-1e01-5360-9e6bdbc562e9}" +--- PASS: TestRaft_VerifyLeader_Fail (0.29s) === RUN TestRaft_VerifyLeader_PartialConnect -testLoggerAdapter verbose: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" - testing.go:149: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" -testLoggerAdapter verbose: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: entering follower state: follower="Node at 9ce58e74-9aac-7f51-6be4-deecc99ceacd [Follower]" leader= - testing.go:149: server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: 2021-08-25T09:28:15.514Z [INFO] server-9ce58e74-9aac-7f51-6be4-deecc99ceacd: entering follower state: follower="Node at 9ce58e74-9aac-7f51-6be4-deecc99ceacd [Follower]" leader= -testLoggerAdapter verbose: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" - testing.go:149: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" -testLoggerAdapter verbose: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: entering follower state: follower="Node at 507aad92-91f2-0b08-b12f-363dec5e687a [Follower]" leader= - testing.go:149: server-507aad92-91f2-0b08-b12f-363dec5e687a: 2021-08-25T09:28:15.514Z [INFO] server-507aad92-91f2-0b08-b12f-363dec5e687a: entering follower state: follower="Node at 507aad92-91f2-0b08-b12f-363dec5e687a [Follower]" leader= -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ce58e74-9aac-7f51-6be4-deecc99ceacd Address:9ce58e74-9aac-7f51-6be4-deecc99ceacd} {Suffrage:Voter ID:server-507aad92-91f2-0b08-b12f-363dec5e687a Address:507aad92-91f2-0b08-b12f-363dec5e687a} {Suffrage:Voter ID:server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7 Address:1ef90d23-f193-2c79-49e9-a0c7a27be6d7}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.515Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering follower state: follower="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:15.515Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.515Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering follower state: follower="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Follower]" leader= -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [WARN] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [WARN] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering candidate state: node="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Candidate]" term=2 - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering candidate state: node="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Candidate]" term=2 -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: election won: tally=2 - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: election won: tally=2 -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering leader state: leader="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Leader]" - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: entering leader state: leader="Node at 1ef90d23-f193-2c79-49e9-a0c7a27be6d7 [Leader]" -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-9ce58e74-9aac-7f51-6be4-deecc99ceacd - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-9ce58e74-9aac-7f51-6be4-deecc99ceacd -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-507aad92-91f2-0b08-b12f-363dec5e687a - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: added peer, starting replication: peer=server-507aad92-91f2-0b08-b12f-363dec5e687a -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.580Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.581Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.581Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: pipelining replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.686Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.581369988 +0000 UTC m=+15.345585897 (1 nodes), 65.95033ms from start of poll, 67.217292ms from cluster start. Timeout at 2021-08-25 09:28:15.686557272 +0000 UTC m=+15.450773261, 105.187364ms after stability - testing.go:149: cluster: 2021-08-25T09:28:15.686Z [INFO] stable state for Leader reached at 2021-08-25 09:28:15.581369988 +0000 UTC m=+15.345585897 (1 nodes), 65.95033ms from start of poll, 67.217292ms from cluster start. Timeout at 2021-08-25 09:28:15.686557272 +0000 UTC m=+15.450773261, 105.187364ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.692Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:15.692Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.797Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.692155332 +0000 UTC m=+15.456371221 (2 nodes), 2.86µs from start of poll, 178.002616ms from cluster start. Timeout at 2021-08-25 09:28:15.797314099 +0000 UTC m=+15.561530048, 105.158827ms after stability - testing.go:149: cluster: 2021-08-25T09:28:15.797Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.692155332 +0000 UTC m=+15.456371221 (2 nodes), 2.86µs from start of poll, 178.002616ms from cluster start. Timeout at 2021-08-25 09:28:15.797314099 +0000 UTC m=+15.561530048, 105.158827ms after stability - raft_test.go:1647: [INFO] Disconnecting Node at 9ce58e74-9aac-7f51-6be4-deecc99ceacd [Follower] -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [ERROR] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: failed to heartbeat to: peer=9ce58e74-9aac-7f51-6be4-deecc99ceacd error="failed to connect to peer: 9ce58e74-9aac-7f51-6be4-deecc99ceacd" - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [ERROR] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: failed to heartbeat to: peer=9ce58e74-9aac-7f51-6be4-deecc99ceacd error="failed to connect to peer: 9ce58e74-9aac-7f51-6be4-deecc99ceacd" -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-507aad92-91f2-0b08-b12f-363dec5e687a 507aad92-91f2-0b08-b12f-363dec5e687a}" -testLoggerAdapter verbose: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" - testing.go:149: server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: 2021-08-25T09:28:15.797Z [INFO] server-1ef90d23-f193-2c79-49e9-a0c7a27be6d7: aborting pipeline replication: peer="{Voter server-9ce58e74-9aac-7f51-6be4-deecc99ceacd 9ce58e74-9aac-7f51-6be4-deecc99ceacd}" ---- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s) +testLoggerAdapter verbose: server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: 2021-08-25T09:31:11.443Z [INFO] server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 Address:ce5fdc15-67fd-fed9-48c0-67c7f16df928} {Suffrage:Voter ID:server-44e3b96b-e662-8caf-28f6-a369ff3ef97b Address:44e3b96b-e662-8caf-28f6-a369ff3ef97b} {Suffrage:Voter ID:server-185c6ebc-ee71-0646-9494-9c3280d22dff Address:185c6ebc-ee71-0646-9494-9c3280d22dff}]" + testing.go:149: server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: 2021-08-25T09:31:11.443Z [INFO] server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 Address:ce5fdc15-67fd-fed9-48c0-67c7f16df928} {Suffrage:Voter ID:server-44e3b96b-e662-8caf-28f6-a369ff3ef97b Address:44e3b96b-e662-8caf-28f6-a369ff3ef97b} {Suffrage:Voter ID:server-185c6ebc-ee71-0646-9494-9c3280d22dff Address:185c6ebc-ee71-0646-9494-9c3280d22dff}]" +testLoggerAdapter verbose: server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: 2021-08-25T09:31:11.443Z [INFO] server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: entering follower state: follower="Node at ce5fdc15-67fd-fed9-48c0-67c7f16df928 [Follower]" leader= + testing.go:149: server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: 2021-08-25T09:31:11.443Z [INFO] server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: entering follower state: follower="Node at ce5fdc15-67fd-fed9-48c0-67c7f16df928 [Follower]" leader= +testLoggerAdapter verbose: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.443Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 Address:ce5fdc15-67fd-fed9-48c0-67c7f16df928} {Suffrage:Voter ID:server-44e3b96b-e662-8caf-28f6-a369ff3ef97b Address:44e3b96b-e662-8caf-28f6-a369ff3ef97b} {Suffrage:Voter ID:server-185c6ebc-ee71-0646-9494-9c3280d22dff Address:185c6ebc-ee71-0646-9494-9c3280d22dff}]" + testing.go:149: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.443Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 Address:ce5fdc15-67fd-fed9-48c0-67c7f16df928} {Suffrage:Voter ID:server-44e3b96b-e662-8caf-28f6-a369ff3ef97b Address:44e3b96b-e662-8caf-28f6-a369ff3ef97b} {Suffrage:Voter ID:server-185c6ebc-ee71-0646-9494-9c3280d22dff Address:185c6ebc-ee71-0646-9494-9c3280d22dff}]" +testLoggerAdapter verbose: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.443Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: entering follower state: follower="Node at 44e3b96b-e662-8caf-28f6-a369ff3ef97b [Follower]" leader= + testing.go:149: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.443Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: entering follower state: follower="Node at 44e3b96b-e662-8caf-28f6-a369ff3ef97b [Follower]" leader= +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.444Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 Address:ce5fdc15-67fd-fed9-48c0-67c7f16df928} {Suffrage:Voter ID:server-44e3b96b-e662-8caf-28f6-a369ff3ef97b Address:44e3b96b-e662-8caf-28f6-a369ff3ef97b} {Suffrage:Voter ID:server-185c6ebc-ee71-0646-9494-9c3280d22dff Address:185c6ebc-ee71-0646-9494-9c3280d22dff}]" + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.444Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 Address:ce5fdc15-67fd-fed9-48c0-67c7f16df928} {Suffrage:Voter ID:server-44e3b96b-e662-8caf-28f6-a369ff3ef97b Address:44e3b96b-e662-8caf-28f6-a369ff3ef97b} {Suffrage:Voter ID:server-185c6ebc-ee71-0646-9494-9c3280d22dff Address:185c6ebc-ee71-0646-9494-9c3280d22dff}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.444Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:11.444Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.444Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: entering follower state: follower="Node at 185c6ebc-ee71-0646-9494-9c3280d22dff [Follower]" leader= + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.444Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: entering follower state: follower="Node at 185c6ebc-ee71-0646-9494-9c3280d22dff [Follower]" leader= +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.538Z [WARN] server-185c6ebc-ee71-0646-9494-9c3280d22dff: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.538Z [WARN] server-185c6ebc-ee71-0646-9494-9c3280d22dff: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: entering candidate state: node="Node at 185c6ebc-ee71-0646-9494-9c3280d22dff [Candidate]" term=2 +testLoggerAdapter verbose: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.539Z [WARN] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: entering candidate state: node="Node at 185c6ebc-ee71-0646-9494-9c3280d22dff [Candidate]" term=2 + testing.go:149: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.539Z [WARN] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.539Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: entering candidate state: node="Node at 44e3b96b-e662-8caf-28f6-a369ff3ef97b [Candidate]" term=2 + testing.go:149: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.539Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: entering candidate state: node="Node at 44e3b96b-e662-8caf-28f6-a369ff3ef97b [Candidate]" term=2 +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: election won: tally=2 +testLoggerAdapter verbose: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.539Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: duplicate requestVote for same term: term=2 + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: election won: tally=2 +testLoggerAdapter verbose: server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: 2021-08-25T09:31:11.539Z [INFO] server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: duplicate requestVote for same term: term=2 + testing.go:149: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.539Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: entering leader state: leader="Node at 185c6ebc-ee71-0646-9494-9c3280d22dff [Leader]" + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: entering leader state: leader="Node at 185c6ebc-ee71-0646-9494-9c3280d22dff [Leader]" +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: added peer, starting replication: peer=server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: added peer, starting replication: peer=server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: added peer, starting replication: peer=server-44e3b96b-e662-8caf-28f6-a369ff3ef97b + testing.go:149: server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: 2021-08-25T09:31:11.539Z [INFO] server-ce5fdc15-67fd-fed9-48c0-67c7f16df928: duplicate requestVote for same term: term=2 + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: added peer, starting replication: peer=server-44e3b96b-e662-8caf-28f6-a369ff3ef97b +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: pipelining replication: peer="{Voter server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 ce5fdc15-67fd-fed9-48c0-67c7f16df928}" + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.539Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: pipelining replication: peer="{Voter server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 ce5fdc15-67fd-fed9-48c0-67c7f16df928}" +testLoggerAdapter verbose: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.540Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: entering follower state: follower="Node at 44e3b96b-e662-8caf-28f6-a369ff3ef97b [Follower]" leader=185c6ebc-ee71-0646-9494-9c3280d22dff + testing.go:149: server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: 2021-08-25T09:31:11.540Z [INFO] server-44e3b96b-e662-8caf-28f6-a369ff3ef97b: entering follower state: follower="Node at 44e3b96b-e662-8caf-28f6-a369ff3ef97b [Follower]" leader=185c6ebc-ee71-0646-9494-9c3280d22dff +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.540Z [WARN] server-185c6ebc-ee71-0646-9494-9c3280d22dff: rejecting vote request since we have a leader: from=44e3b96b-e662-8caf-28f6-a369ff3ef97b leader=185c6ebc-ee71-0646-9494-9c3280d22dff + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.540Z [WARN] server-185c6ebc-ee71-0646-9494-9c3280d22dff: rejecting vote request since we have a leader: from=44e3b96b-e662-8caf-28f6-a369ff3ef97b leader=185c6ebc-ee71-0646-9494-9c3280d22dff +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.540Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: pipelining replication: peer="{Voter server-44e3b96b-e662-8caf-28f6-a369ff3ef97b 44e3b96b-e662-8caf-28f6-a369ff3ef97b}" + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.540Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: pipelining replication: peer="{Voter server-44e3b96b-e662-8caf-28f6-a369ff3ef97b 44e3b96b-e662-8caf-28f6-a369ff3ef97b}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.645Z [INFO] stable state for Leader reached at 2021-08-25 09:31:11.540143219 +0000 UTC m=+15.806359603 (1 nodes), 95.887675ms from start of poll, 96.993671ms from cluster start. Timeout at 2021-08-25 09:31:11.645270205 +0000 UTC m=+15.911486629, 105.127026ms after stability + testing.go:149: cluster: 2021-08-25T09:31:11.645Z [INFO] stable state for Leader reached at 2021-08-25 09:31:11.540143219 +0000 UTC m=+15.806359603 (1 nodes), 95.887675ms from start of poll, 96.993671ms from cluster start. Timeout at 2021-08-25 09:31:11.645270205 +0000 UTC m=+15.911486629, 105.127026ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.650Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:11.650Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.756Z [INFO] stable state for Follower reached at 2021-08-25 09:31:11.650882297 +0000 UTC m=+15.917098701 (2 nodes), 3.16µs from start of poll, 207.732769ms from cluster start. Timeout at 2021-08-25 09:31:11.756050859 +0000 UTC m=+16.022267343, 105.168642ms after stability + testing.go:149: cluster: 2021-08-25T09:31:11.756Z [INFO] stable state for Follower reached at 2021-08-25 09:31:11.650882297 +0000 UTC m=+15.917098701 (2 nodes), 3.16µs from start of poll, 207.732769ms from cluster start. Timeout at 2021-08-25 09:31:11.756050859 +0000 UTC m=+16.022267343, 105.168642ms after stability + raft_test.go:1647: [INFO] Disconnecting Node at ce5fdc15-67fd-fed9-48c0-67c7f16df928 [Follower] +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.756Z [ERROR] server-185c6ebc-ee71-0646-9494-9c3280d22dff: failed to heartbeat to: peer=ce5fdc15-67fd-fed9-48c0-67c7f16df928 error="failed to connect to peer: ce5fdc15-67fd-fed9-48c0-67c7f16df928" + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.756Z [ERROR] server-185c6ebc-ee71-0646-9494-9c3280d22dff: failed to heartbeat to: peer=ce5fdc15-67fd-fed9-48c0-67c7f16df928 error="failed to connect to peer: ce5fdc15-67fd-fed9-48c0-67c7f16df928" +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.756Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: aborting pipeline replication: peer="{Voter server-44e3b96b-e662-8caf-28f6-a369ff3ef97b 44e3b96b-e662-8caf-28f6-a369ff3ef97b}" + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.756Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: aborting pipeline replication: peer="{Voter server-44e3b96b-e662-8caf-28f6-a369ff3ef97b 44e3b96b-e662-8caf-28f6-a369ff3ef97b}" +testLoggerAdapter verbose: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.756Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: aborting pipeline replication: peer="{Voter server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 ce5fdc15-67fd-fed9-48c0-67c7f16df928}" + testing.go:149: server-185c6ebc-ee71-0646-9494-9c3280d22dff: 2021-08-25T09:31:11.756Z [INFO] server-185c6ebc-ee71-0646-9494-9c3280d22dff: aborting pipeline replication: peer="{Voter server-ce5fdc15-67fd-fed9-48c0-67c7f16df928 ce5fdc15-67fd-fed9-48c0-67c7f16df928}" +--- PASS: TestRaft_VerifyLeader_PartialConnect (0.32s) === RUN TestRaft_StartAsLeader -testLoggerAdapter verbose: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.800Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4459b3a1-3046-e4a6-ea59-ba718b18f246 Address:4459b3a1-3046-e4a6-ea59-ba718b18f246}]" - testing.go:149: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.800Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4459b3a1-3046-e4a6-ea59-ba718b18f246 Address:4459b3a1-3046-e4a6-ea59-ba718b18f246}]" -testLoggerAdapter verbose: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.801Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: entering leader state: leader="Node at 4459b3a1-3046-e4a6-ea59-ba718b18f246 [Leader]" - testing.go:149: server-4459b3a1-3046-e4a6-ea59-ba718b18f246: 2021-08-25T09:28:15.801Z [INFO] server-4459b3a1-3046-e4a6-ea59-ba718b18f246: entering leader state: leader="Node at 4459b3a1-3046-e4a6-ea59-ba718b18f246 [Leader]" +testLoggerAdapter verbose: server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: 2021-08-25T09:31:11.759Z [INFO] server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2 Address:3d7137dd-6195-1bb9-9bca-8359c07eb2a2}]" + testing.go:149: server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: 2021-08-25T09:31:11.759Z [INFO] server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2 Address:3d7137dd-6195-1bb9-9bca-8359c07eb2a2}]" +testLoggerAdapter verbose: server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: 2021-08-25T09:31:11.759Z [INFO] server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: entering leader state: leader="Node at 3d7137dd-6195-1bb9-9bca-8359c07eb2a2 [Leader]" + testing.go:149: server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: 2021-08-25T09:31:11.759Z [INFO] server-3d7137dd-6195-1bb9-9bca-8359c07eb2a2: entering leader state: leader="Node at 3d7137dd-6195-1bb9-9bca-8359c07eb2a2 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh -testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf Address:f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf}]" - testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf Address:f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf}]" -testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering follower state: follower="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Follower]" leader= - testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.803Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering follower state: follower="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Follower]" leader= -testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [WARN] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [WARN] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering candidate state: node="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Candidate]" term=2 - testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering candidate state: node="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Candidate]" term=2 -testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: election won: tally=1 - testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: election won: tally=1 -testLoggerAdapter verbose: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering leader state: leader="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Leader]" - testing.go:149: server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: 2021-08-25T09:28:15.869Z [INFO] server-f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf: entering leader state: leader="Node at f1ea1e46-dc73-7b81-bf6c-b5c02e8e53bf [Leader]" ---- PASS: TestRaft_NotifyCh (0.07s) +testLoggerAdapter verbose: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.761Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7744c0db-b751-2cf5-cf8e-674ca958b595 Address:7744c0db-b751-2cf5-cf8e-674ca958b595}]" + testing.go:149: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.761Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7744c0db-b751-2cf5-cf8e-674ca958b595 Address:7744c0db-b751-2cf5-cf8e-674ca958b595}]" +testLoggerAdapter verbose: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.761Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: entering follower state: follower="Node at 7744c0db-b751-2cf5-cf8e-674ca958b595 [Follower]" leader= + testing.go:149: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.761Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: entering follower state: follower="Node at 7744c0db-b751-2cf5-cf8e-674ca958b595 [Follower]" leader= +testLoggerAdapter verbose: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [WARN] server-7744c0db-b751-2cf5-cf8e-674ca958b595: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [WARN] server-7744c0db-b751-2cf5-cf8e-674ca958b595: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: entering candidate state: node="Node at 7744c0db-b751-2cf5-cf8e-674ca958b595 [Candidate]" term=2 + testing.go:149: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: entering candidate state: node="Node at 7744c0db-b751-2cf5-cf8e-674ca958b595 [Candidate]" term=2 +testLoggerAdapter verbose: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: election won: tally=1 + testing.go:149: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: election won: tally=1 +testLoggerAdapter verbose: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: entering leader state: leader="Node at 7744c0db-b751-2cf5-cf8e-674ca958b595 [Leader]" + testing.go:149: server-7744c0db-b751-2cf5-cf8e-674ca958b595: 2021-08-25T09:31:11.844Z [INFO] server-7744c0db-b751-2cf5-cf8e-674ca958b595: entering leader state: leader="Node at 7744c0db-b751-2cf5-cf8e-674ca958b595 [Leader]" +--- PASS: TestRaft_NotifyCh (0.08s) === RUN TestRaft_Voting -testLoggerAdapter verbose: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.873Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" - testing.go:149: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.873Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" -testLoggerAdapter verbose: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.874Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" - testing.go:149: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.874Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.874Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.874Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd15992-4d28-455a-9c76-9735991c0cb2 Address:2bd15992-4d28-455a-9c76-9735991c0cb2} {Suffrage:Voter ID:server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 Address:7000105f-a0d1-5762-c2b6-75a830a9c2e9} {Suffrage:Voter ID:server-3bdcbafe-2ba8-d631-15c0-c8c80189a015 Address:3bdcbafe-2ba8-d631-15c0-c8c80189a015}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:15.875Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:15.875Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.875Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: entering follower state: follower="Node at 7000105f-a0d1-5762-c2b6-75a830a9c2e9 [Follower]" leader= - testing.go:149: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:15.875Z [INFO] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: entering follower state: follower="Node at 7000105f-a0d1-5762-c2b6-75a830a9c2e9 [Follower]" leader= -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.875Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering follower state: follower="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Follower]" leader= - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.875Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering follower state: follower="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Follower]" leader= -testLoggerAdapter verbose: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.875Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: entering follower state: follower="Node at 2bd15992-4d28-455a-9c76-9735991c0cb2 [Follower]" leader= - testing.go:149: server-2bd15992-4d28-455a-9c76-9735991c0cb2: 2021-08-25T09:28:15.875Z [INFO] server-2bd15992-4d28-455a-9c76-9735991c0cb2: entering follower state: follower="Node at 2bd15992-4d28-455a-9c76-9735991c0cb2 [Follower]" leader= -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [WARN] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [WARN] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering candidate state: node="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Candidate]" term=2 - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering candidate state: node="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Candidate]" term=2 -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: election won: tally=2 - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.966Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: election won: tally=2 -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering leader state: leader="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Leader]" - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: entering leader state: leader="Node at 3bdcbafe-2ba8-d631-15c0-c8c80189a015 [Leader]" -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-2bd15992-4d28-455a-9c76-9735991c0cb2 - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-2bd15992-4d28-455a-9c76-9735991c0cb2 -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: added peer, starting replication: peer=server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:15.967Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: pipelining replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.072Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.967036825 +0000 UTC m=+15.731252714 (2 nodes), 91.963053ms from start of poll, 93.557905ms from cluster start. Timeout at 2021-08-25 09:28:16.072180793 +0000 UTC m=+15.836396702, 105.143988ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.072Z [INFO] stable state for Follower reached at 2021-08-25 09:28:15.967036825 +0000 UTC m=+15.731252714 (2 nodes), 91.963053ms from start of poll, 93.557905ms from cluster start. Timeout at 2021-08-25 09:28:16.072180793 +0000 UTC m=+15.836396702, 105.143988ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.072Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:16.072Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.177Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.072475565 +0000 UTC m=+15.836691454 (1 nodes), 2.799µs from start of poll, 198.996645ms from cluster start. Timeout at 2021-08-25 09:28:16.177642351 +0000 UTC m=+15.941858300, 105.166846ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.177Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.072475565 +0000 UTC m=+15.836691454 (1 nodes), 2.799µs from start of poll, 198.996645ms from cluster start. Timeout at 2021-08-25 09:28:16.177642351 +0000 UTC m=+15.941858300, 105.166846ms after stability -testLoggerAdapter verbose: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:16.177Z [WARN] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: rejecting vote request since we have a leader: from=2bd15992-4d28-455a-9c76-9735991c0cb2 leader=3bdcbafe-2ba8-d631-15c0-c8c80189a015 - testing.go:149: server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: 2021-08-25T09:28:16.177Z [WARN] server-7000105f-a0d1-5762-c2b6-75a830a9c2e9: rejecting vote request since we have a leader: from=2bd15992-4d28-455a-9c76-9735991c0cb2 leader=3bdcbafe-2ba8-d631-15c0-c8c80189a015 -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-7000105f-a0d1-5762-c2b6-75a830a9c2e9 7000105f-a0d1-5762-c2b6-75a830a9c2e9}" -testLoggerAdapter verbose: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" - testing.go:149: server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: 2021-08-25T09:28:16.178Z [INFO] server-3bdcbafe-2ba8-d631-15c0-c8c80189a015: aborting pipeline replication: peer="{Voter server-2bd15992-4d28-455a-9c76-9735991c0cb2 2bd15992-4d28-455a-9c76-9735991c0cb2}" ---- PASS: TestRaft_Voting (0.31s) +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.847Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 Address:ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12} {Suffrage:Voter ID:server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 Address:cfd7fc9e-5cf7-1217-f77d-dcfcad192f50} {Suffrage:Voter ID:server-f9eeae9a-4039-312d-7455-5b19e6edd20c Address:f9eeae9a-4039-312d-7455-5b19e6edd20c}]" + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.847Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 Address:ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12} {Suffrage:Voter ID:server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 Address:cfd7fc9e-5cf7-1217-f77d-dcfcad192f50} {Suffrage:Voter ID:server-f9eeae9a-4039-312d-7455-5b19e6edd20c Address:f9eeae9a-4039-312d-7455-5b19e6edd20c}]" +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.847Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: entering follower state: follower="Node at ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 [Follower]" leader= + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.847Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: entering follower state: follower="Node at ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 [Follower]" leader= +testLoggerAdapter verbose: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.847Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 Address:ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12} {Suffrage:Voter ID:server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 Address:cfd7fc9e-5cf7-1217-f77d-dcfcad192f50} {Suffrage:Voter ID:server-f9eeae9a-4039-312d-7455-5b19e6edd20c Address:f9eeae9a-4039-312d-7455-5b19e6edd20c}]" + testing.go:149: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.847Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 Address:ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12} {Suffrage:Voter ID:server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 Address:cfd7fc9e-5cf7-1217-f77d-dcfcad192f50} {Suffrage:Voter ID:server-f9eeae9a-4039-312d-7455-5b19e6edd20c Address:f9eeae9a-4039-312d-7455-5b19e6edd20c}]" +testLoggerAdapter verbose: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.847Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: entering follower state: follower="Node at cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 [Follower]" leader= + testing.go:149: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.847Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: entering follower state: follower="Node at cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 [Follower]" leader= +testLoggerAdapter verbose: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:11.848Z [INFO] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 Address:ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12} {Suffrage:Voter ID:server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 Address:cfd7fc9e-5cf7-1217-f77d-dcfcad192f50} {Suffrage:Voter ID:server-f9eeae9a-4039-312d-7455-5b19e6edd20c Address:f9eeae9a-4039-312d-7455-5b19e6edd20c}]" + testing.go:149: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:11.848Z [INFO] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 Address:ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12} {Suffrage:Voter ID:server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 Address:cfd7fc9e-5cf7-1217-f77d-dcfcad192f50} {Suffrage:Voter ID:server-f9eeae9a-4039-312d-7455-5b19e6edd20c Address:f9eeae9a-4039-312d-7455-5b19e6edd20c}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:11.848Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:11.848Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:11.848Z [INFO] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: entering follower state: follower="Node at f9eeae9a-4039-312d-7455-5b19e6edd20c [Follower]" leader= + testing.go:149: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:11.848Z [INFO] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: entering follower state: follower="Node at f9eeae9a-4039-312d-7455-5b19e6edd20c [Follower]" leader= +testLoggerAdapter verbose: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.903Z [WARN] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.903Z [WARN] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.903Z [WARN] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.903Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: entering candidate state: node="Node at cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 [Candidate]" term=2 + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.903Z [WARN] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.903Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: entering candidate state: node="Node at ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 [Candidate]" term=2 + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.903Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: entering candidate state: node="Node at ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 [Candidate]" term=2 + testing.go:149: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.903Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: entering candidate state: node="Node at cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 [Candidate]" term=2 +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.903Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: election won: tally=2 +testLoggerAdapter verbose: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.903Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:11.903Z [INFO] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: duplicate requestVote for same term: term=2 + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.903Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: election won: tally=2 +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: entering leader state: leader="Node at ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 [Leader]" + testing.go:149: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.903Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: duplicate requestVote for same term: term=2 + testing.go:149: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:11.903Z [INFO] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: duplicate requestVote for same term: term=2 + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: entering leader state: leader="Node at ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 [Leader]" +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: added peer, starting replication: peer=server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: added peer, starting replication: peer=server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: added peer, starting replication: peer=server-f9eeae9a-4039-312d-7455-5b19e6edd20c + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: added peer, starting replication: peer=server-f9eeae9a-4039-312d-7455-5b19e6edd20c +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [WARN] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: rejecting vote request since we have a leader: from=cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 leader=ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [WARN] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: rejecting vote request since we have a leader: from=cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 leader=ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 +testLoggerAdapter verbose: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.904Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: entering follower state: follower="Node at cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 [Follower]" leader=ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: pipelining replication: peer="{Voter server-f9eeae9a-4039-312d-7455-5b19e6edd20c f9eeae9a-4039-312d-7455-5b19e6edd20c}" + testing.go:149: server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: 2021-08-25T09:31:11.904Z [INFO] server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50: entering follower state: follower="Node at cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 [Follower]" leader=ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: pipelining replication: peer="{Voter server-f9eeae9a-4039-312d-7455-5b19e6edd20c f9eeae9a-4039-312d-7455-5b19e6edd20c}" +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: pipelining replication: peer="{Voter server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 cfd7fc9e-5cf7-1217-f77d-dcfcad192f50}" + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:11.904Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: pipelining replication: peer="{Voter server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 cfd7fc9e-5cf7-1217-f77d-dcfcad192f50}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.009Z [INFO] stable state for Follower reached at 2021-08-25 09:31:11.904619316 +0000 UTC m=+16.170835740 (2 nodes), 56.347337ms from start of poll, 57.424855ms from cluster start. Timeout at 2021-08-25 09:31:12.009738003 +0000 UTC m=+16.275954487, 105.118747ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.009Z [INFO] stable state for Follower reached at 2021-08-25 09:31:11.904619316 +0000 UTC m=+16.170835740 (2 nodes), 56.347337ms from start of poll, 57.424855ms from cluster start. Timeout at 2021-08-25 09:31:12.009738003 +0000 UTC m=+16.275954487, 105.118747ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.009Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:12.009Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.115Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.010038834 +0000 UTC m=+16.276255218 (1 nodes), 2.779µs from start of poll, 162.844333ms from cluster start. Timeout at 2021-08-25 09:31:12.115205356 +0000 UTC m=+16.381421820, 105.166602ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.115Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.010038834 +0000 UTC m=+16.276255218 (1 nodes), 2.779µs from start of poll, 162.844333ms from cluster start. Timeout at 2021-08-25 09:31:12.115205356 +0000 UTC m=+16.381421820, 105.166602ms after stability +testLoggerAdapter verbose: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:12.115Z [WARN] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: rejecting vote request since we have a leader: from=cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 leader=ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 + testing.go:149: server-f9eeae9a-4039-312d-7455-5b19e6edd20c: 2021-08-25T09:31:12.115Z [WARN] server-f9eeae9a-4039-312d-7455-5b19e6edd20c: rejecting vote request since we have a leader: from=cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 leader=ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12 +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:12.115Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: aborting pipeline replication: peer="{Voter server-f9eeae9a-4039-312d-7455-5b19e6edd20c f9eeae9a-4039-312d-7455-5b19e6edd20c}" + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:12.115Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: aborting pipeline replication: peer="{Voter server-f9eeae9a-4039-312d-7455-5b19e6edd20c f9eeae9a-4039-312d-7455-5b19e6edd20c}" +testLoggerAdapter verbose: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:12.115Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: aborting pipeline replication: peer="{Voter server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 cfd7fc9e-5cf7-1217-f77d-dcfcad192f50}" + testing.go:149: server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: 2021-08-25T09:31:12.115Z [INFO] server-ca1b40fe-fd32-2b2e-1f3e-0a8506fd8f12: aborting pipeline replication: peer="{Voter server-cfd7fc9e-5cf7-1217-f77d-dcfcad192f50 cfd7fc9e-5cf7-1217-f77d-dcfcad192f50}" +--- PASS: TestRaft_Voting (0.27s) === RUN TestRaft_ProtocolVersion_RejectRPC -testLoggerAdapter verbose: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" - testing.go:149: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" -testLoggerAdapter verbose: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: entering follower state: follower="Node at af220f1d-c3df-5c25-537f-4d761cc41a81 [Follower]" leader= - testing.go:149: server-af220f1d-c3df-5c25-537f-4d761cc41a81: 2021-08-25T09:28:16.182Z [INFO] server-af220f1d-c3df-5c25-537f-4d761cc41a81: entering follower state: follower="Node at af220f1d-c3df-5c25-537f-4d761cc41a81 [Follower]" leader= -testLoggerAdapter verbose: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" - testing.go:149: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" -testLoggerAdapter verbose: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: entering follower state: follower="Node at 04fc31fd-c621-9a31-78d8-e89c80f91b5d [Follower]" leader= - testing.go:149: server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: 2021-08-25T09:28:16.182Z [INFO] server-04fc31fd-c621-9a31-78d8-e89c80f91b5d: entering follower state: follower="Node at 04fc31fd-c621-9a31-78d8-e89c80f91b5d [Follower]" leader= -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af220f1d-c3df-5c25-537f-4d761cc41a81 Address:af220f1d-c3df-5c25-537f-4d761cc41a81} {Suffrage:Voter ID:server-04fc31fd-c621-9a31-78d8-e89c80f91b5d Address:04fc31fd-c621-9a31-78d8-e89c80f91b5d} {Suffrage:Voter ID:server-5f25179c-a632-ca43-c5df-f03743459518 Address:5f25179c-a632-ca43-c5df-f03743459518}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.183Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:16.183Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering follower state: follower="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Follower]" leader= - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.183Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering follower state: follower="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Follower]" leader= -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.258Z [WARN] server-5f25179c-a632-ca43-c5df-f03743459518: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.258Z [WARN] server-5f25179c-a632-ca43-c5df-f03743459518: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering candidate state: node="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Candidate]" term=2 - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering candidate state: node="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Candidate]" term=2 -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: election won: tally=2 - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: election won: tally=2 -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering leader state: leader="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Leader]" - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: entering leader state: leader="Node at 5f25179c-a632-ca43-c5df-f03743459518 [Leader]" -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-af220f1d-c3df-5c25-537f-4d761cc41a81 - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-af220f1d-c3df-5c25-537f-4d761cc41a81 -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-04fc31fd-c621-9a31-78d8-e89c80f91b5d - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: added peer, starting replication: peer=server-04fc31fd-c621-9a31-78d8-e89c80f91b5d -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.259Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: pipelining replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.364Z [INFO] stable state for Follower reached at 2021-08-25 09:28:16.259358737 +0000 UTC m=+16.023574646 (2 nodes), 75.96338ms from start of poll, 77.325174ms from cluster start. Timeout at 2021-08-25 09:28:16.364494885 +0000 UTC m=+16.128710854, 105.136208ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.364Z [INFO] stable state for Follower reached at 2021-08-25 09:28:16.259358737 +0000 UTC m=+16.023574646 (2 nodes), 75.96338ms from start of poll, 77.325174ms from cluster start. Timeout at 2021-08-25 09:28:16.364494885 +0000 UTC m=+16.128710854, 105.136208ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.364Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:16.364Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.469Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.364816216 +0000 UTC m=+16.129032125 (1 nodes), 2.42µs from start of poll, 182.782653ms from cluster start. Timeout at 2021-08-25 09:28:16.46988655 +0000 UTC m=+16.234102499, 105.070374ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.469Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.364816216 +0000 UTC m=+16.129032125 (1 nodes), 2.42µs from start of poll, 182.782653ms from cluster start. Timeout at 2021-08-25 09:28:16.46988655 +0000 UTC m=+16.234102499, 105.070374ms after stability -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-04fc31fd-c621-9a31-78d8-e89c80f91b5d 04fc31fd-c621-9a31-78d8-e89c80f91b5d}" -testLoggerAdapter verbose: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" - testing.go:149: server-5f25179c-a632-ca43-c5df-f03743459518: 2021-08-25T09:28:16.470Z [INFO] server-5f25179c-a632-ca43-c5df-f03743459518: aborting pipeline replication: peer="{Voter server-af220f1d-c3df-5c25-537f-4d761cc41a81 af220f1d-c3df-5c25-537f-4d761cc41a81}" ---- PASS: TestRaft_ProtocolVersion_RejectRPC (0.29s) +testLoggerAdapter verbose: server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: 2021-08-25T09:31:12.119Z [INFO] server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 Address:32bff70f-0d8e-2a90-18bd-68e65c07a9a7} {Suffrage:Voter ID:server-8eb64893-06c0-8572-5ae3-e1cb27037597 Address:8eb64893-06c0-8572-5ae3-e1cb27037597} {Suffrage:Voter ID:server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 Address:2bbabbba-66b2-4e00-b66a-0d3ca2412fe9}]" + testing.go:149: server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: 2021-08-25T09:31:12.119Z [INFO] server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 Address:32bff70f-0d8e-2a90-18bd-68e65c07a9a7} {Suffrage:Voter ID:server-8eb64893-06c0-8572-5ae3-e1cb27037597 Address:8eb64893-06c0-8572-5ae3-e1cb27037597} {Suffrage:Voter ID:server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 Address:2bbabbba-66b2-4e00-b66a-0d3ca2412fe9}]" +testLoggerAdapter verbose: server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: 2021-08-25T09:31:12.120Z [INFO] server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: entering follower state: follower="Node at 32bff70f-0d8e-2a90-18bd-68e65c07a9a7 [Follower]" leader= + testing.go:149: server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: 2021-08-25T09:31:12.120Z [INFO] server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7: entering follower state: follower="Node at 32bff70f-0d8e-2a90-18bd-68e65c07a9a7 [Follower]" leader= +testLoggerAdapter verbose: server-8eb64893-06c0-8572-5ae3-e1cb27037597: 2021-08-25T09:31:12.120Z [INFO] server-8eb64893-06c0-8572-5ae3-e1cb27037597: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 Address:32bff70f-0d8e-2a90-18bd-68e65c07a9a7} {Suffrage:Voter ID:server-8eb64893-06c0-8572-5ae3-e1cb27037597 Address:8eb64893-06c0-8572-5ae3-e1cb27037597} {Suffrage:Voter ID:server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 Address:2bbabbba-66b2-4e00-b66a-0d3ca2412fe9}]" + testing.go:149: server-8eb64893-06c0-8572-5ae3-e1cb27037597: 2021-08-25T09:31:12.120Z [INFO] server-8eb64893-06c0-8572-5ae3-e1cb27037597: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 Address:32bff70f-0d8e-2a90-18bd-68e65c07a9a7} {Suffrage:Voter ID:server-8eb64893-06c0-8572-5ae3-e1cb27037597 Address:8eb64893-06c0-8572-5ae3-e1cb27037597} {Suffrage:Voter ID:server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 Address:2bbabbba-66b2-4e00-b66a-0d3ca2412fe9}]" +testLoggerAdapter verbose: server-8eb64893-06c0-8572-5ae3-e1cb27037597: 2021-08-25T09:31:12.120Z [INFO] server-8eb64893-06c0-8572-5ae3-e1cb27037597: entering follower state: follower="Node at 8eb64893-06c0-8572-5ae3-e1cb27037597 [Follower]" leader= + testing.go:149: server-8eb64893-06c0-8572-5ae3-e1cb27037597: 2021-08-25T09:31:12.120Z [INFO] server-8eb64893-06c0-8572-5ae3-e1cb27037597: entering follower state: follower="Node at 8eb64893-06c0-8572-5ae3-e1cb27037597 [Follower]" leader= +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.120Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 Address:32bff70f-0d8e-2a90-18bd-68e65c07a9a7} {Suffrage:Voter ID:server-8eb64893-06c0-8572-5ae3-e1cb27037597 Address:8eb64893-06c0-8572-5ae3-e1cb27037597} {Suffrage:Voter ID:server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 Address:2bbabbba-66b2-4e00-b66a-0d3ca2412fe9}]" + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.120Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 Address:32bff70f-0d8e-2a90-18bd-68e65c07a9a7} {Suffrage:Voter ID:server-8eb64893-06c0-8572-5ae3-e1cb27037597 Address:8eb64893-06c0-8572-5ae3-e1cb27037597} {Suffrage:Voter ID:server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 Address:2bbabbba-66b2-4e00-b66a-0d3ca2412fe9}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.120Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:12.120Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.120Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: entering follower state: follower="Node at 2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 [Follower]" leader= + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.120Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: entering follower state: follower="Node at 2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 [Follower]" leader= +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.182Z [WARN] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.182Z [WARN] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.182Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: entering candidate state: node="Node at 2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 [Candidate]" term=2 + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.182Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: entering candidate state: node="Node at 2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 [Candidate]" term=2 +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: election won: tally=2 + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: election won: tally=2 +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: entering leader state: leader="Node at 2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 [Leader]" + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: entering leader state: leader="Node at 2bbabbba-66b2-4e00-b66a-0d3ca2412fe9 [Leader]" +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: added peer, starting replication: peer=server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: added peer, starting replication: peer=server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: added peer, starting replication: peer=server-8eb64893-06c0-8572-5ae3-e1cb27037597 + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: added peer, starting replication: peer=server-8eb64893-06c0-8572-5ae3-e1cb27037597 +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: pipelining replication: peer="{Voter server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 32bff70f-0d8e-2a90-18bd-68e65c07a9a7}" + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: pipelining replication: peer="{Voter server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 32bff70f-0d8e-2a90-18bd-68e65c07a9a7}" +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: pipelining replication: peer="{Voter server-8eb64893-06c0-8572-5ae3-e1cb27037597 8eb64893-06c0-8572-5ae3-e1cb27037597}" + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.183Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: pipelining replication: peer="{Voter server-8eb64893-06c0-8572-5ae3-e1cb27037597 8eb64893-06c0-8572-5ae3-e1cb27037597}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.288Z [INFO] stable state for Follower reached at 2021-08-25 09:31:12.183364902 +0000 UTC m=+16.449581345 (2 nodes), 62.641264ms from start of poll, 63.753099ms from cluster start. Timeout at 2021-08-25 09:31:12.288516505 +0000 UTC m=+16.554732909, 105.151564ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.288Z [INFO] stable state for Follower reached at 2021-08-25 09:31:12.183364902 +0000 UTC m=+16.449581345 (2 nodes), 62.641264ms from start of poll, 63.753099ms from cluster start. Timeout at 2021-08-25 09:31:12.288516505 +0000 UTC m=+16.554732909, 105.151564ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.288Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:12.288Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.394Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.288834695 +0000 UTC m=+16.555051099 (1 nodes), 2.28µs from start of poll, 169.222853ms from cluster start. Timeout at 2021-08-25 09:31:12.393888288 +0000 UTC m=+16.660104772, 105.053673ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.394Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.288834695 +0000 UTC m=+16.555051099 (1 nodes), 2.28µs from start of poll, 169.222853ms from cluster start. Timeout at 2021-08-25 09:31:12.393888288 +0000 UTC m=+16.660104772, 105.053673ms after stability +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.394Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: aborting pipeline replication: peer="{Voter server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 32bff70f-0d8e-2a90-18bd-68e65c07a9a7}" + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.394Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: aborting pipeline replication: peer="{Voter server-32bff70f-0d8e-2a90-18bd-68e65c07a9a7 32bff70f-0d8e-2a90-18bd-68e65c07a9a7}" +testLoggerAdapter verbose: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.394Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: aborting pipeline replication: peer="{Voter server-8eb64893-06c0-8572-5ae3-e1cb27037597 8eb64893-06c0-8572-5ae3-e1cb27037597}" + testing.go:149: server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: 2021-08-25T09:31:12.394Z [INFO] server-2bbabbba-66b2-4e00-b66a-0d3ca2412fe9: aborting pipeline replication: peer="{Voter server-8eb64893-06c0-8572-5ae3-e1cb27037597 8eb64893-06c0-8572-5ae3-e1cb27037597}" +--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering follower state: follower="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Follower]" leader= - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.474Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering follower state: follower="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Follower]" leader= -testLoggerAdapter verbose: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" - testing.go:149: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" -testLoggerAdapter verbose: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: entering follower state: follower="Node at d773f2e2-2931-7d53-9203-437d0936b98f [Follower]" leader= - testing.go:149: d773f2e2-2931-7d53-9203-437d0936b98f: 2021-08-25T09:28:16.474Z [INFO] d773f2e2-2931-7d53-9203-437d0936b98f: entering follower state: follower="Node at d773f2e2-2931-7d53-9203-437d0936b98f [Follower]" leader= -testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: initial configuration: index=0 servers=[] - testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.475Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:16.475Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: entering follower state: follower="Node at e526e47f-5207-5caa-3eae-ebfd83ed2534 [Follower]" leader= - testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.475Z [INFO] e526e47f-5207-5caa-3eae-ebfd83ed2534: entering follower state: follower="Node at e526e47f-5207-5caa-3eae-ebfd83ed2534 [Follower]" leader= -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering candidate state: node="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Candidate]" term=2 - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.526Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering candidate state: node="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Candidate]" term=2 -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: election won: tally=2 - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: election won: tally=2 -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering leader state: leader="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Leader]" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: entering leader state: leader="Node at 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 [Leader]" -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=d773f2e2-2931-7d53-9203-437d0936b98f - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=d773f2e2-2931-7d53-9203-437d0936b98f -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.527Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" -testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.545Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: no known peers, aborting election - testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.545Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.632Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.527397585 +0000 UTC m=+16.291613494 (1 nodes), 51.930953ms from start of poll, 53.57578ms from cluster start. Timeout at 2021-08-25 09:28:16.632482499 +0000 UTC m=+16.396698468, 105.084974ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.632Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.527397585 +0000 UTC m=+16.291613494 (1 nodes), 51.930953ms from start of poll, 53.57578ms from cluster start. Timeout at 2021-08-25 09:28:16.632482499 +0000 UTC m=+16.396698468, 105.084974ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.632Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:16.632Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.737Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.632755234 +0000 UTC m=+16.396971123 (1 nodes), 2.7µs from start of poll, 158.933409ms from cluster start. Timeout at 2021-08-25 09:28:16.737886703 +0000 UTC m=+16.502102692, 105.131569ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.737Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.632755234 +0000 UTC m=+16.396971123 (1 nodes), 2.7µs from start of poll, 158.933409ms from cluster start. Timeout at 2021-08-25 09:28:16.737886703 +0000 UTC m=+16.502102692, 105.131569ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.738Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:16.738Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.843Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.738193854 +0000 UTC m=+16.502409763 (1 nodes), 2.18µs from start of poll, 264.372049ms from cluster start. Timeout at 2021-08-25 09:28:16.843338782 +0000 UTC m=+16.607554771, 105.145008ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.843Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.738193854 +0000 UTC m=+16.502409763 (1 nodes), 2.18µs from start of poll, 264.372049ms from cluster start. Timeout at 2021-08-25 09:28:16.843338782 +0000 UTC m=+16.607554771, 105.145008ms after stability -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=AddStaging server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr=e526e47f-5207-5caa-3eae-ebfd83ed2534 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f} {Suffrage:Voter ID:e526e47f-5207-5caa-3eae-ebfd83ed2534 Address:e526e47f-5207-5caa-3eae-ebfd83ed2534}]" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=AddStaging server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr=e526e47f-5207-5caa-3eae-ebfd83ed2534 servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f} {Suffrage:Voter ID:e526e47f-5207-5caa-3eae-ebfd83ed2534 Address:e526e47f-5207-5caa-3eae-ebfd83ed2534}]" -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.843Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: added peer, starting replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 -testLoggerAdapter verbose: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.844Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: e526e47f-5207-5caa-3eae-ebfd83ed2534: 2021-08-25T09:28:16.844Z [WARN] e526e47f-5207-5caa-3eae-ebfd83ed2534: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: appendEntries rejected, sending older logs: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" next=1 - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [WARN] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: appendEntries rejected, sending older logs: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" next=1 -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:16.844Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: pipelining replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.849Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:16.849Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.954Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.849483091 +0000 UTC m=+16.613698980 (1 nodes), 2.08µs from start of poll, 375.661266ms from cluster start. Timeout at 2021-08-25 09:28:16.95462184 +0000 UTC m=+16.718837809, 105.138829ms after stability - testing.go:149: cluster: 2021-08-25T09:28:16.954Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.849483091 +0000 UTC m=+16.613698980 (1 nodes), 2.08µs from start of poll, 375.661266ms from cluster start. Timeout at 2021-08-25 09:28:16.95462184 +0000 UTC m=+16.718837809, 105.138829ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:16.954Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:16.954Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.060Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.954916492 +0000 UTC m=+16.719132381 (1 nodes), 2.24µs from start of poll, 481.094667ms from cluster start. Timeout at 2021-08-25 09:28:17.059997086 +0000 UTC m=+16.824213035, 105.080654ms after stability - testing.go:149: cluster: 2021-08-25T09:28:17.060Z [INFO] stable state for Leader reached at 2021-08-25 09:28:16.954916492 +0000 UTC m=+16.719132381 (1 nodes), 2.24µs from start of poll, 481.094667ms from cluster start. Timeout at 2021-08-25 09:28:17.059997086 +0000 UTC m=+16.824213035, 105.080654ms after stability -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=RemoveServer server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr= servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: updating configuration: command=RemoveServer server-id=e526e47f-5207-5caa-3eae-ebfd83ed2534 server-addr= servers="[{Suffrage:Voter ID:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8 Address:6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8} {Suffrage:Voter ID:d773f2e2-2931-7d53-9203-437d0936b98f Address:d773f2e2-2931-7d53-9203-437d0936b98f}]" -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: removed peer, stopping replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 last-index=4 - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: removed peer, stopping replication: peer=e526e47f-5207-5caa-3eae-ebfd83ed2534 last-index=4 -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter e526e47f-5207-5caa-3eae-ebfd83ed2534 e526e47f-5207-5caa-3eae-ebfd83ed2534}" -testLoggerAdapter verbose: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" - testing.go:149: 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: 2021-08-25T09:28:17.060Z [INFO] 6c16d7b8-5bdc-dc4c-b4fa-ca81967baee8: aborting pipeline replication: peer="{Voter d773f2e2-2931-7d53-9203-437d0936b98f d773f2e2-2931-7d53-9203-437d0936b98f}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.59s) +testLoggerAdapter verbose: 9bae2e09-24b9-2550-7023-7e8fcc774eff: 2021-08-25T09:31:12.397Z [INFO] 9bae2e09-24b9-2550-7023-7e8fcc774eff: initial configuration: index=1 servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5}]" + testing.go:149: 9bae2e09-24b9-2550-7023-7e8fcc774eff: 2021-08-25T09:31:12.397Z [INFO] 9bae2e09-24b9-2550-7023-7e8fcc774eff: initial configuration: index=1 servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5}]" +testLoggerAdapter verbose: 9bae2e09-24b9-2550-7023-7e8fcc774eff: 2021-08-25T09:31:12.397Z [INFO] 9bae2e09-24b9-2550-7023-7e8fcc774eff: entering follower state: follower="Node at 9bae2e09-24b9-2550-7023-7e8fcc774eff [Follower]" leader= +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.397Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5}]" + testing.go:149: 9bae2e09-24b9-2550-7023-7e8fcc774eff: 2021-08-25T09:31:12.397Z [INFO] 9bae2e09-24b9-2550-7023-7e8fcc774eff: entering follower state: follower="Node at 9bae2e09-24b9-2550-7023-7e8fcc774eff [Follower]" leader= + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.397Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5}]" +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.398Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: entering follower state: follower="Node at ffd23400-f971-4802-10ed-1e16f93069d5 [Follower]" leader= + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.398Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: entering follower state: follower="Node at ffd23400-f971-4802-10ed-1e16f93069d5 [Follower]" leader= +testLoggerAdapter verbose: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.398Z [INFO] 2263bd07-accf-402d-e2a8-6ed62da1cf25: initial configuration: index=0 servers=[] + testing.go:149: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.398Z [INFO] 2263bd07-accf-402d-e2a8-6ed62da1cf25: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.398Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:12.398Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.398Z [INFO] 2263bd07-accf-402d-e2a8-6ed62da1cf25: entering follower state: follower="Node at 2263bd07-accf-402d-e2a8-6ed62da1cf25 [Follower]" leader= + testing.go:149: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.398Z [INFO] 2263bd07-accf-402d-e2a8-6ed62da1cf25: entering follower state: follower="Node at 2263bd07-accf-402d-e2a8-6ed62da1cf25 [Follower]" leader= +testLoggerAdapter verbose: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.459Z [WARN] 2263bd07-accf-402d-e2a8-6ed62da1cf25: no known peers, aborting election + testing.go:149: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.459Z [WARN] 2263bd07-accf-402d-e2a8-6ed62da1cf25: no known peers, aborting election +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.470Z [WARN] ffd23400-f971-4802-10ed-1e16f93069d5: heartbeat timeout reached, starting election: last-leader= + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.470Z [WARN] ffd23400-f971-4802-10ed-1e16f93069d5: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.470Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: entering candidate state: node="Node at ffd23400-f971-4802-10ed-1e16f93069d5 [Candidate]" term=2 + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.470Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: entering candidate state: node="Node at ffd23400-f971-4802-10ed-1e16f93069d5 [Candidate]" term=2 +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.470Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: election won: tally=2 + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.470Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: election won: tally=2 +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.471Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: entering leader state: leader="Node at ffd23400-f971-4802-10ed-1e16f93069d5 [Leader]" + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.471Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: entering leader state: leader="Node at ffd23400-f971-4802-10ed-1e16f93069d5 [Leader]" +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.471Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: added peer, starting replication: peer=9bae2e09-24b9-2550-7023-7e8fcc774eff + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.471Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: added peer, starting replication: peer=9bae2e09-24b9-2550-7023-7e8fcc774eff +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.471Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: pipelining replication: peer="{Voter 9bae2e09-24b9-2550-7023-7e8fcc774eff 9bae2e09-24b9-2550-7023-7e8fcc774eff}" + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.471Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: pipelining replication: peer="{Voter 9bae2e09-24b9-2550-7023-7e8fcc774eff 9bae2e09-24b9-2550-7023-7e8fcc774eff}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.576Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.471196172 +0000 UTC m=+16.737412616 (1 nodes), 72.260859ms from start of poll, 73.818193ms from cluster start. Timeout at 2021-08-25 09:31:12.576347236 +0000 UTC m=+16.842563740, 105.151124ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.576Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.471196172 +0000 UTC m=+16.737412616 (1 nodes), 72.260859ms from start of poll, 73.818193ms from cluster start. Timeout at 2021-08-25 09:31:12.576347236 +0000 UTC m=+16.842563740, 105.151124ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.576Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:12.576Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.681Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.576652127 +0000 UTC m=+16.842868531 (1 nodes), 2.66µs from start of poll, 179.274108ms from cluster start. Timeout at 2021-08-25 09:31:12.681728798 +0000 UTC m=+16.947945362, 105.076831ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.681Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.576652127 +0000 UTC m=+16.842868531 (1 nodes), 2.66µs from start of poll, 179.274108ms from cluster start. Timeout at 2021-08-25 09:31:12.681728798 +0000 UTC m=+16.947945362, 105.076831ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.681Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:12.681Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.787Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.68201779 +0000 UTC m=+16.948234234 (1 nodes), 2.739µs from start of poll, 284.639811ms from cluster start. Timeout at 2021-08-25 09:31:12.787190412 +0000 UTC m=+17.053406916, 105.172682ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.787Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.68201779 +0000 UTC m=+16.948234234 (1 nodes), 2.739µs from start of poll, 284.639811ms from cluster start. Timeout at 2021-08-25 09:31:12.787190412 +0000 UTC m=+17.053406916, 105.172682ms after stability +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.787Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: updating configuration: command=AddStaging server-id=2263bd07-accf-402d-e2a8-6ed62da1cf25 server-addr=2263bd07-accf-402d-e2a8-6ed62da1cf25 servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5} {Suffrage:Voter ID:2263bd07-accf-402d-e2a8-6ed62da1cf25 Address:2263bd07-accf-402d-e2a8-6ed62da1cf25}]" + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.787Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: updating configuration: command=AddStaging server-id=2263bd07-accf-402d-e2a8-6ed62da1cf25 server-addr=2263bd07-accf-402d-e2a8-6ed62da1cf25 servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5} {Suffrage:Voter ID:2263bd07-accf-402d-e2a8-6ed62da1cf25 Address:2263bd07-accf-402d-e2a8-6ed62da1cf25}]" +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.787Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: added peer, starting replication: peer=2263bd07-accf-402d-e2a8-6ed62da1cf25 + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.787Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: added peer, starting replication: peer=2263bd07-accf-402d-e2a8-6ed62da1cf25 +testLoggerAdapter verbose: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.787Z [WARN] 2263bd07-accf-402d-e2a8-6ed62da1cf25: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: 2263bd07-accf-402d-e2a8-6ed62da1cf25: 2021-08-25T09:31:12.787Z [WARN] 2263bd07-accf-402d-e2a8-6ed62da1cf25: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.788Z [WARN] ffd23400-f971-4802-10ed-1e16f93069d5: appendEntries rejected, sending older logs: peer="{Voter 2263bd07-accf-402d-e2a8-6ed62da1cf25 2263bd07-accf-402d-e2a8-6ed62da1cf25}" next=1 + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.788Z [WARN] ffd23400-f971-4802-10ed-1e16f93069d5: appendEntries rejected, sending older logs: peer="{Voter 2263bd07-accf-402d-e2a8-6ed62da1cf25 2263bd07-accf-402d-e2a8-6ed62da1cf25}" next=1 +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.788Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: pipelining replication: peer="{Voter 2263bd07-accf-402d-e2a8-6ed62da1cf25 2263bd07-accf-402d-e2a8-6ed62da1cf25}" + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:12.788Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: pipelining replication: peer="{Voter 2263bd07-accf-402d-e2a8-6ed62da1cf25 2263bd07-accf-402d-e2a8-6ed62da1cf25}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.793Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:12.793Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.898Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.793260501 +0000 UTC m=+17.059476905 (1 nodes), 2.3µs from start of poll, 395.882482ms from cluster start. Timeout at 2021-08-25 09:31:12.898397866 +0000 UTC m=+17.164614330, 105.137425ms after stability + testing.go:149: cluster: 2021-08-25T09:31:12.898Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.793260501 +0000 UTC m=+17.059476905 (1 nodes), 2.3µs from start of poll, 395.882482ms from cluster start. Timeout at 2021-08-25 09:31:12.898397866 +0000 UTC m=+17.164614330, 105.137425ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:12.898Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:12.898Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.003Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.898679179 +0000 UTC m=+17.164895583 (1 nodes), 2.12µs from start of poll, 501.30116ms from cluster start. Timeout at 2021-08-25 09:31:13.003697815 +0000 UTC m=+17.269914259, 105.018676ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.003Z [INFO] stable state for Leader reached at 2021-08-25 09:31:12.898679179 +0000 UTC m=+17.164895583 (1 nodes), 2.12µs from start of poll, 501.30116ms from cluster start. Timeout at 2021-08-25 09:31:13.003697815 +0000 UTC m=+17.269914259, 105.018676ms after stability +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: updating configuration: command=RemoveServer server-id=2263bd07-accf-402d-e2a8-6ed62da1cf25 server-addr= servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5}]" + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: updating configuration: command=RemoveServer server-id=2263bd07-accf-402d-e2a8-6ed62da1cf25 server-addr= servers="[{Suffrage:Voter ID:9bae2e09-24b9-2550-7023-7e8fcc774eff Address:9bae2e09-24b9-2550-7023-7e8fcc774eff} {Suffrage:Voter ID:ffd23400-f971-4802-10ed-1e16f93069d5 Address:ffd23400-f971-4802-10ed-1e16f93069d5}]" +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: removed peer, stopping replication: peer=2263bd07-accf-402d-e2a8-6ed62da1cf25 last-index=4 + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: removed peer, stopping replication: peer=2263bd07-accf-402d-e2a8-6ed62da1cf25 last-index=4 +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: aborting pipeline replication: peer="{Voter 2263bd07-accf-402d-e2a8-6ed62da1cf25 2263bd07-accf-402d-e2a8-6ed62da1cf25}" + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: aborting pipeline replication: peer="{Voter 2263bd07-accf-402d-e2a8-6ed62da1cf25 2263bd07-accf-402d-e2a8-6ed62da1cf25}" +testLoggerAdapter verbose: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: aborting pipeline replication: peer="{Voter 9bae2e09-24b9-2550-7023-7e8fcc774eff 9bae2e09-24b9-2550-7023-7e8fcc774eff}" + testing.go:149: ffd23400-f971-4802-10ed-1e16f93069d5: 2021-08-25T09:31:13.004Z [INFO] ffd23400-f971-4802-10ed-1e16f93069d5: aborting pipeline replication: peer="{Voter 9bae2e09-24b9-2550-7023-7e8fcc774eff 9bae2e09-24b9-2550-7023-7e8fcc774eff}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 -testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.063Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" - testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.063Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.064Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:17.064Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.064Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= - testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.064Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering follower state: follower="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Follower]" leader= - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.064Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering follower state: follower="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Follower]" leader= -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.144Z [WARN] 18323061-8241-df37-0c12-8c3d6b8b309c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=2 - testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.144Z [WARN] 18323061-8241-df37-0c12-8c3d6b8b309c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering candidate state: node="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Candidate]" term=2 - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.144Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=2 - testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering candidate state: node="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Candidate]" term=2 -testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: duplicate requestVote for same term: term=2 - testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.145Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.145Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: duplicate requestVote for same term: term=2 - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.145Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.209Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: Election timeout reached, restarting election - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.209Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: Election timeout reached, restarting election -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=3 - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering candidate state: node="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Candidate]" term=3 -testLoggerAdapter verbose: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.210Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= - testing.go:149: 18323061-8241-df37-0c12-8c3d6b8b309c: 2021-08-25T09:28:17.210Z [INFO] 18323061-8241-df37-0c12-8c3d6b8b309c: entering follower state: follower="Node at 18323061-8241-df37-0c12-8c3d6b8b309c [Follower]" leader= -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: election won: tally=2 - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: election won: tally=2 -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering leader state: leader="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Leader]" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: entering leader state: leader="Node at 07270d5a-1384-f258-112b-3142e87f0e0b [Leader]" -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.210Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.316Z [INFO] stable state for Follower reached at 2021-08-25 09:28:17.210901721 +0000 UTC m=+16.975117610 (1 nodes), 146.534701ms from start of poll, 147.1927ms from cluster start. Timeout at 2021-08-25 09:28:17.316014672 +0000 UTC m=+17.080230721, 105.113111ms after stability - testing.go:149: cluster: 2021-08-25T09:28:17.316Z [INFO] stable state for Follower reached at 2021-08-25 09:28:17.210901721 +0000 UTC m=+16.975117610 (1 nodes), 146.534701ms from start of poll, 147.1927ms from cluster start. Timeout at 2021-08-25 09:28:17.316014672 +0000 UTC m=+17.080230721, 105.113111ms after stability -testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: initial configuration: index=0 servers=[] - testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.317Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:17.317Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: entering follower state: follower="Node at e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 [Follower]" leader= - testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.317Z [INFO] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: entering follower state: follower="Node at e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 [Follower]" leader= -testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.415Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: no known peers, aborting election - testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.415Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.422Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.317206621 +0000 UTC m=+17.081422510 (1 nodes), 1.959µs from start of poll, 253.4976ms from cluster start. Timeout at 2021-08-25 09:28:17.422336791 +0000 UTC m=+17.186552700, 105.13019ms after stability - testing.go:149: cluster: 2021-08-25T09:28:17.422Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.317206621 +0000 UTC m=+17.081422510 (1 nodes), 1.959µs from start of poll, 253.4976ms from cluster start. Timeout at 2021-08-25 09:28:17.422336791 +0000 UTC m=+17.186552700, 105.13019ms after stability -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=AddStaging server-id=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 server-addr=e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=AddStaging server-id=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 server-addr=e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 servers="[{Suffrage:Voter ID:18323061-8241-df37-0c12-8c3d6b8b309c Address:18323061-8241-df37-0c12-8c3d6b8b309c} {Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.422Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: added peer, starting replication: peer=server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 -testLoggerAdapter verbose: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.422Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: 2021-08-25T09:28:17.422Z [WARN] server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: appendEntries rejected, sending older logs: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" next=1 - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [WARN] 07270d5a-1384-f258-112b-3142e87f0e0b: appendEntries rejected, sending older logs: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" next=1 -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.423Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: pipelining replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.428Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:17.428Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.533Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.428436224 +0000 UTC m=+17.192652113 (1 nodes), 2.28µs from start of poll, 364.727203ms from cluster start. Timeout at 2021-08-25 09:28:17.533530896 +0000 UTC m=+17.297746805, 105.094692ms after stability - testing.go:149: cluster: 2021-08-25T09:28:17.533Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.428436224 +0000 UTC m=+17.192652113 (1 nodes), 2.28µs from start of poll, 364.727203ms from cluster start. Timeout at 2021-08-25 09:28:17.533530896 +0000 UTC m=+17.297746805, 105.094692ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.533Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:17.533Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.638Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.533790332 +0000 UTC m=+17.298006221 (1 nodes), 2.139µs from start of poll, 470.081311ms from cluster start. Timeout at 2021-08-25 09:28:17.638846029 +0000 UTC m=+17.403061898, 105.055677ms after stability - testing.go:149: cluster: 2021-08-25T09:28:17.638Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.533790332 +0000 UTC m=+17.298006221 (1 nodes), 2.139µs from start of poll, 470.081311ms from cluster start. Timeout at 2021-08-25 09:28:17.638846029 +0000 UTC m=+17.403061898, 105.055677ms after stability -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=RemoveServer server-id=18323061-8241-df37-0c12-8c3d6b8b309c server-addr= servers="[{Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: updating configuration: command=RemoveServer server-id=18323061-8241-df37-0c12-8c3d6b8b309c server-addr= servers="[{Suffrage:Voter ID:07270d5a-1384-f258-112b-3142e87f0e0b Address:07270d5a-1384-f258-112b-3142e87f0e0b} {Suffrage:Voter ID:server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 Address:e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}]" -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: removed peer, stopping replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c last-index=4 - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: removed peer, stopping replication: peer=18323061-8241-df37-0c12-8c3d6b8b309c last-index=4 -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter 18323061-8241-df37-0c12-8c3d6b8b309c 18323061-8241-df37-0c12-8c3d6b8b309c}" -testLoggerAdapter verbose: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" - testing.go:149: 07270d5a-1384-f258-112b-3142e87f0e0b: 2021-08-25T09:28:17.639Z [INFO] 07270d5a-1384-f258-112b-3142e87f0e0b: aborting pipeline replication: peer="{Voter server-e6e1fe8f-0782-82b5-a34b-6af5d0cfc333 e6e1fe8f-0782-82b5-a34b-6af5d0cfc333}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.58s) +testLoggerAdapter verbose: 4149dc47-5924-ffe4-78f2-b35df55172f5: 2021-08-25T09:31:13.008Z [INFO] 4149dc47-5924-ffe4-78f2-b35df55172f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:4149dc47-5924-ffe4-78f2-b35df55172f5 Address:4149dc47-5924-ffe4-78f2-b35df55172f5} {Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de}]" + testing.go:149: 4149dc47-5924-ffe4-78f2-b35df55172f5: 2021-08-25T09:31:13.008Z [INFO] 4149dc47-5924-ffe4-78f2-b35df55172f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:4149dc47-5924-ffe4-78f2-b35df55172f5 Address:4149dc47-5924-ffe4-78f2-b35df55172f5} {Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de}]" +testLoggerAdapter verbose: 4149dc47-5924-ffe4-78f2-b35df55172f5: 2021-08-25T09:31:13.008Z [INFO] 4149dc47-5924-ffe4-78f2-b35df55172f5: entering follower state: follower="Node at 4149dc47-5924-ffe4-78f2-b35df55172f5 [Follower]" leader= + testing.go:149: 4149dc47-5924-ffe4-78f2-b35df55172f5: 2021-08-25T09:31:13.008Z [INFO] 4149dc47-5924-ffe4-78f2-b35df55172f5: entering follower state: follower="Node at 4149dc47-5924-ffe4-78f2-b35df55172f5 [Follower]" leader= +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.008Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: initial configuration: index=1 servers="[{Suffrage:Voter ID:4149dc47-5924-ffe4-78f2-b35df55172f5 Address:4149dc47-5924-ffe4-78f2-b35df55172f5} {Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de}]" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.008Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: initial configuration: index=1 servers="[{Suffrage:Voter ID:4149dc47-5924-ffe4-78f2-b35df55172f5 Address:4149dc47-5924-ffe4-78f2-b35df55172f5} {Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.008Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:13.008Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.008Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: entering follower state: follower="Node at b7e00268-87d5-10b0-607a-c947482187de [Follower]" leader= + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.008Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: entering follower state: follower="Node at b7e00268-87d5-10b0-607a-c947482187de [Follower]" leader= +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [WARN] b7e00268-87d5-10b0-607a-c947482187de: heartbeat timeout reached, starting election: last-leader= + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [WARN] b7e00268-87d5-10b0-607a-c947482187de: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: entering candidate state: node="Node at b7e00268-87d5-10b0-607a-c947482187de [Candidate]" term=2 + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: entering candidate state: node="Node at b7e00268-87d5-10b0-607a-c947482187de [Candidate]" term=2 +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: election won: tally=2 + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: election won: tally=2 +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: entering leader state: leader="Node at b7e00268-87d5-10b0-607a-c947482187de [Leader]" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: entering leader state: leader="Node at b7e00268-87d5-10b0-607a-c947482187de [Leader]" +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: added peer, starting replication: peer=4149dc47-5924-ffe4-78f2-b35df55172f5 + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: added peer, starting replication: peer=4149dc47-5924-ffe4-78f2-b35df55172f5 +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: pipelining replication: peer="{Voter 4149dc47-5924-ffe4-78f2-b35df55172f5 4149dc47-5924-ffe4-78f2-b35df55172f5}" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.100Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: pipelining replication: peer="{Voter 4149dc47-5924-ffe4-78f2-b35df55172f5 4149dc47-5924-ffe4-78f2-b35df55172f5}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.205Z [INFO] stable state for Follower reached at 2021-08-25 09:31:13.100624013 +0000 UTC m=+17.366840417 (1 nodes), 92.11875ms from start of poll, 92.781348ms from cluster start. Timeout at 2021-08-25 09:31:13.205756038 +0000 UTC m=+17.471972502, 105.132085ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.205Z [INFO] stable state for Follower reached at 2021-08-25 09:31:13.100624013 +0000 UTC m=+17.366840417 (1 nodes), 92.11875ms from start of poll, 92.781348ms from cluster start. Timeout at 2021-08-25 09:31:13.205756038 +0000 UTC m=+17.471972502, 105.132085ms after stability +testLoggerAdapter verbose: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.206Z [INFO] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: initial configuration: index=0 servers=[] + testing.go:149: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.206Z [INFO] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.207Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:13.207Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.207Z [INFO] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: entering follower state: follower="Node at 92450e97-8bc9-f45e-0f54-459f2edf8aa1 [Follower]" leader= + testing.go:149: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.207Z [INFO] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: entering follower state: follower="Node at 92450e97-8bc9-f45e-0f54-459f2edf8aa1 [Follower]" leader= +testLoggerAdapter verbose: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.265Z [WARN] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: no known peers, aborting election + testing.go:149: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.265Z [WARN] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.312Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.207170205 +0000 UTC m=+17.473386709 (1 nodes), 33.637µs from start of poll, 199.32764ms from cluster start. Timeout at 2021-08-25 09:31:13.312302451 +0000 UTC m=+17.578518915, 105.132206ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.312Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.207170205 +0000 UTC m=+17.473386709 (1 nodes), 33.637µs from start of poll, 199.32764ms from cluster start. Timeout at 2021-08-25 09:31:13.312302451 +0000 UTC m=+17.578518915, 105.132206ms after stability +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.312Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: updating configuration: command=AddStaging server-id=server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 server-addr=92450e97-8bc9-f45e-0f54-459f2edf8aa1 servers="[{Suffrage:Voter ID:4149dc47-5924-ffe4-78f2-b35df55172f5 Address:4149dc47-5924-ffe4-78f2-b35df55172f5} {Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de} {Suffrage:Voter ID:server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 Address:92450e97-8bc9-f45e-0f54-459f2edf8aa1}]" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.312Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: updating configuration: command=AddStaging server-id=server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 server-addr=92450e97-8bc9-f45e-0f54-459f2edf8aa1 servers="[{Suffrage:Voter ID:4149dc47-5924-ffe4-78f2-b35df55172f5 Address:4149dc47-5924-ffe4-78f2-b35df55172f5} {Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de} {Suffrage:Voter ID:server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 Address:92450e97-8bc9-f45e-0f54-459f2edf8aa1}]" +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.312Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: added peer, starting replication: peer=server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.312Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: added peer, starting replication: peer=server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 +testLoggerAdapter verbose: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.312Z [WARN] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: 2021-08-25T09:31:13.312Z [WARN] server-92450e97-8bc9-f45e-0f54-459f2edf8aa1: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.313Z [WARN] b7e00268-87d5-10b0-607a-c947482187de: appendEntries rejected, sending older logs: peer="{Voter server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 92450e97-8bc9-f45e-0f54-459f2edf8aa1}" next=1 + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.313Z [WARN] b7e00268-87d5-10b0-607a-c947482187de: appendEntries rejected, sending older logs: peer="{Voter server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 92450e97-8bc9-f45e-0f54-459f2edf8aa1}" next=1 +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.313Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: pipelining replication: peer="{Voter server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 92450e97-8bc9-f45e-0f54-459f2edf8aa1}" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.313Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: pipelining replication: peer="{Voter server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 92450e97-8bc9-f45e-0f54-459f2edf8aa1}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.318Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:13.318Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.423Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.318405976 +0000 UTC m=+17.584622380 (1 nodes), 2.36µs from start of poll, 310.563311ms from cluster start. Timeout at 2021-08-25 09:31:13.42355154 +0000 UTC m=+17.689768044, 105.145664ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.423Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.318405976 +0000 UTC m=+17.584622380 (1 nodes), 2.36µs from start of poll, 310.563311ms from cluster start. Timeout at 2021-08-25 09:31:13.42355154 +0000 UTC m=+17.689768044, 105.145664ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.423Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:13.423Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.529Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.423892908 +0000 UTC m=+17.690109292 (1 nodes), 2.38µs from start of poll, 416.050223ms from cluster start. Timeout at 2021-08-25 09:31:13.529041372 +0000 UTC m=+17.795257836, 105.148544ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.529Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.423892908 +0000 UTC m=+17.690109292 (1 nodes), 2.38µs from start of poll, 416.050223ms from cluster start. Timeout at 2021-08-25 09:31:13.529041372 +0000 UTC m=+17.795257836, 105.148544ms after stability +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: updating configuration: command=RemoveServer server-id=4149dc47-5924-ffe4-78f2-b35df55172f5 server-addr= servers="[{Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de} {Suffrage:Voter ID:server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 Address:92450e97-8bc9-f45e-0f54-459f2edf8aa1}]" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: updating configuration: command=RemoveServer server-id=4149dc47-5924-ffe4-78f2-b35df55172f5 server-addr= servers="[{Suffrage:Voter ID:b7e00268-87d5-10b0-607a-c947482187de Address:b7e00268-87d5-10b0-607a-c947482187de} {Suffrage:Voter ID:server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 Address:92450e97-8bc9-f45e-0f54-459f2edf8aa1}]" +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: removed peer, stopping replication: peer=4149dc47-5924-ffe4-78f2-b35df55172f5 last-index=4 + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: removed peer, stopping replication: peer=4149dc47-5924-ffe4-78f2-b35df55172f5 last-index=4 +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: aborting pipeline replication: peer="{Voter 4149dc47-5924-ffe4-78f2-b35df55172f5 4149dc47-5924-ffe4-78f2-b35df55172f5}" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: aborting pipeline replication: peer="{Voter 4149dc47-5924-ffe4-78f2-b35df55172f5 4149dc47-5924-ffe4-78f2-b35df55172f5}" +testLoggerAdapter verbose: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: aborting pipeline replication: peer="{Voter server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 92450e97-8bc9-f45e-0f54-459f2edf8aa1}" + testing.go:149: b7e00268-87d5-10b0-607a-c947482187de: 2021-08-25T09:31:13.529Z [INFO] b7e00268-87d5-10b0-607a-c947482187de: aborting pipeline replication: peer="{Voter server-92450e97-8bc9-f45e-0f54-459f2edf8aa1 92450e97-8bc9-f45e-0f54-459f2edf8aa1}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.52s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.643Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering follower state: follower="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Follower]" leader= - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.643Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering follower state: follower="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Follower]" leader= -testLoggerAdapter verbose: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" - testing.go:149: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91e07238-9313-b639-00d7-4caf2cd4ff45 Address:91e07238-9313-b639-00d7-4caf2cd4ff45} {Suffrage:Voter ID:server-1d41a700-ab7c-4948-cf83-b63a833e14fc Address:1d41a700-ab7c-4948-cf83-b63a833e14fc} {Suffrage:Voter ID:server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 Address:7b9835ce-b8b3-5c56-a8f5-df64f9056b76}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.643Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:17.643Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: entering follower state: follower="Node at 7b9835ce-b8b3-5c56-a8f5-df64f9056b76 [Follower]" leader= - testing.go:149: server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: 2021-08-25T09:28:17.643Z [INFO] server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76: entering follower state: follower="Node at 7b9835ce-b8b3-5c56-a8f5-df64f9056b76 [Follower]" leader= -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [WARN] server-91e07238-9313-b639-00d7-4caf2cd4ff45: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [WARN] server-91e07238-9313-b639-00d7-4caf2cd4ff45: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering candidate state: node="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Candidate]" term=2 - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering candidate state: node="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Candidate]" term=2 -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: election won: tally=2 - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: election won: tally=2 -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering leader state: leader="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Leader]" - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.716Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering leader state: leader="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Leader]" -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-1d41a700-ab7c-4948-cf83-b63a833e14fc - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-1d41a700-ab7c-4948-cf83-b63a833e14fc -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.717Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: pipelining replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.822Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.717666283 +0000 UTC m=+17.481882172 (1 nodes), 73.747426ms from start of poll, 74.761892ms from cluster start. Timeout at 2021-08-25 09:28:17.822807931 +0000 UTC m=+17.587023880, 105.141708ms after stability - testing.go:149: cluster: 2021-08-25T09:28:17.822Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.717666283 +0000 UTC m=+17.481882172 (1 nodes), 73.747426ms from start of poll, 74.761892ms from cluster start. Timeout at 2021-08-25 09:28:17.822807931 +0000 UTC m=+17.587023880, 105.141708ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.823Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:17.823Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.928Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.823074347 +0000 UTC m=+17.587290236 (1 nodes), 2.18µs from start of poll, 180.169956ms from cluster start. Timeout at 2021-08-25 09:28:17.928154841 +0000 UTC m=+17.692370789, 105.080553ms after stability - testing.go:149: cluster: 2021-08-25T09:28:17.928Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.823074347 +0000 UTC m=+17.587290236 (1 nodes), 2.18µs from start of poll, 180.169956ms from cluster start. Timeout at 2021-08-25 09:28:17.928154841 +0000 UTC m=+17.692370789, 105.080553ms after stability -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering candidate state: node="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Candidate]" term=3 - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering candidate state: node="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Candidate]" term=3 -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: election won: tally=2 - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: election won: tally=2 -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering leader state: leader="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Leader]" -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: entering leader state: leader="Node at 1d41a700-ab7c-4948-cf83-b63a833e14fc [Leader]" - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: entering follower state: follower="Node at 91e07238-9313-b639-00d7-4caf2cd4ff45 [Follower]" leader= -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-91e07238-9313-b639-00d7-4caf2cd4ff45 -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-91e07238-9313-b639-00d7-4caf2cd4ff45 -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:17.928Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: added peer, starting replication: peer=server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-1d41a700-ab7c-4948-cf83-b63a833e14fc 1d41a700-ab7c-4948-cf83-b63a833e14fc}" -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" - testing.go:149: cluster: 2021-08-25T09:28:17.928Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.928Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" - testing.go:149: server-91e07238-9313-b639-00d7-4caf2cd4ff45: 2021-08-25T09:28:17.928Z [INFO] server-91e07238-9313-b639-00d7-4caf2cd4ff45: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.929Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:17.929Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: pipelining replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.034Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.929103372 +0000 UTC m=+17.693319301 (1 nodes), 41.996µs from start of poll, 286.199021ms from cluster start. Timeout at 2021-08-25 09:28:18.034235302 +0000 UTC m=+17.798451270, 105.131969ms after stability - testing.go:149: cluster: 2021-08-25T09:28:18.034Z [INFO] stable state for Leader reached at 2021-08-25 09:28:17.929103372 +0000 UTC m=+17.693319301 (1 nodes), 41.996µs from start of poll, 286.199021ms from cluster start. Timeout at 2021-08-25 09:28:18.034235302 +0000 UTC m=+17.798451270, 105.131969ms after stability -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-91e07238-9313-b639-00d7-4caf2cd4ff45 91e07238-9313-b639-00d7-4caf2cd4ff45}" -testLoggerAdapter verbose: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" - testing.go:149: server-1d41a700-ab7c-4948-cf83-b63a833e14fc: 2021-08-25T09:28:18.034Z [INFO] server-1d41a700-ab7c-4948-cf83-b63a833e14fc: aborting pipeline replication: peer="{Voter server-7b9835ce-b8b3-5c56-a8f5-df64f9056b76 7b9835ce-b8b3-5c56-a8f5-df64f9056b76}" ---- PASS: TestRaft_LeadershipTransfer (0.39s) +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.533Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 Address:737f9c17-f66c-2185-c80b-e9c02e1a04b5} {Suffrage:Voter ID:server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 Address:2661fbe6-8288-be2d-1199-d0e4b66b4a35} {Suffrage:Voter ID:server-c4a599c6-bd42-5ade-4559-1bdf940e5492 Address:c4a599c6-bd42-5ade-4559-1bdf940e5492}]" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.533Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 Address:737f9c17-f66c-2185-c80b-e9c02e1a04b5} {Suffrage:Voter ID:server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 Address:2661fbe6-8288-be2d-1199-d0e4b66b4a35} {Suffrage:Voter ID:server-c4a599c6-bd42-5ade-4559-1bdf940e5492 Address:c4a599c6-bd42-5ade-4559-1bdf940e5492}]" +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.533Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: entering follower state: follower="Node at 737f9c17-f66c-2185-c80b-e9c02e1a04b5 [Follower]" leader= + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.533Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: entering follower state: follower="Node at 737f9c17-f66c-2185-c80b-e9c02e1a04b5 [Follower]" leader= +testLoggerAdapter verbose: server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: 2021-08-25T09:31:13.534Z [INFO] server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 Address:737f9c17-f66c-2185-c80b-e9c02e1a04b5} {Suffrage:Voter ID:server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 Address:2661fbe6-8288-be2d-1199-d0e4b66b4a35} {Suffrage:Voter ID:server-c4a599c6-bd42-5ade-4559-1bdf940e5492 Address:c4a599c6-bd42-5ade-4559-1bdf940e5492}]" + testing.go:149: server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: 2021-08-25T09:31:13.534Z [INFO] server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 Address:737f9c17-f66c-2185-c80b-e9c02e1a04b5} {Suffrage:Voter ID:server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 Address:2661fbe6-8288-be2d-1199-d0e4b66b4a35} {Suffrage:Voter ID:server-c4a599c6-bd42-5ade-4559-1bdf940e5492 Address:c4a599c6-bd42-5ade-4559-1bdf940e5492}]" +testLoggerAdapter verbose: server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: 2021-08-25T09:31:13.534Z [INFO] server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: entering follower state: follower="Node at 2661fbe6-8288-be2d-1199-d0e4b66b4a35 [Follower]" leader= + testing.go:149: server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: 2021-08-25T09:31:13.534Z [INFO] server-2661fbe6-8288-be2d-1199-d0e4b66b4a35: entering follower state: follower="Node at 2661fbe6-8288-be2d-1199-d0e4b66b4a35 [Follower]" leader= +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.534Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 Address:737f9c17-f66c-2185-c80b-e9c02e1a04b5} {Suffrage:Voter ID:server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 Address:2661fbe6-8288-be2d-1199-d0e4b66b4a35} {Suffrage:Voter ID:server-c4a599c6-bd42-5ade-4559-1bdf940e5492 Address:c4a599c6-bd42-5ade-4559-1bdf940e5492}]" + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.534Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 Address:737f9c17-f66c-2185-c80b-e9c02e1a04b5} {Suffrage:Voter ID:server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 Address:2661fbe6-8288-be2d-1199-d0e4b66b4a35} {Suffrage:Voter ID:server-c4a599c6-bd42-5ade-4559-1bdf940e5492 Address:c4a599c6-bd42-5ade-4559-1bdf940e5492}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.534Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:13.534Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.534Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering follower state: follower="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Follower]" leader= + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.534Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering follower state: follower="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Follower]" leader= +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [WARN] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [WARN] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering candidate state: node="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Candidate]" term=2 + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering candidate state: node="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Candidate]" term=2 +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: election won: tally=2 + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: election won: tally=2 +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering leader state: leader="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Leader]" + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.587Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering leader state: leader="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Leader]" +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: added peer, starting replication: peer=server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: added peer, starting replication: peer=server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: added peer, starting replication: peer=server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: added peer, starting replication: peer=server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: pipelining replication: peer="{Voter server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 737f9c17-f66c-2185-c80b-e9c02e1a04b5}" + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: pipelining replication: peer="{Voter server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 737f9c17-f66c-2185-c80b-e9c02e1a04b5}" +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: pipelining replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.588Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: pipelining replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.693Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.588074716 +0000 UTC m=+17.854291180 (1 nodes), 53.377456ms from start of poll, 54.658176ms from cluster start. Timeout at 2021-08-25 09:31:13.693187743 +0000 UTC m=+17.959404147, 105.112967ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.693Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.588074716 +0000 UTC m=+17.854291180 (1 nodes), 53.377456ms from start of poll, 54.658176ms from cluster start. Timeout at 2021-08-25 09:31:13.693187743 +0000 UTC m=+17.959404147, 105.112967ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.693Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:13.693Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.798Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.69343666 +0000 UTC m=+17.959653064 (1 nodes), 2.56µs from start of poll, 160.02006ms from cluster start. Timeout at 2021-08-25 09:31:13.798559386 +0000 UTC m=+18.064775850, 105.122786ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.798Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.69343666 +0000 UTC m=+17.959653064 (1 nodes), 2.56µs from start of poll, 160.02006ms from cluster start. Timeout at 2021-08-25 09:31:13.798559386 +0000 UTC m=+18.064775850, 105.122786ms after stability +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.798Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: entering candidate state: node="Node at 737f9c17-f66c-2185-c80b-e9c02e1a04b5 [Candidate]" term=3 + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.798Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: entering candidate state: node="Node at 737f9c17-f66c-2185-c80b-e9c02e1a04b5 [Candidate]" term=3 +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.799Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering follower state: follower="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.799Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.799Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: entering follower state: follower="Node at c4a599c6-bd42-5ade-4559-1bdf940e5492 [Follower]" leader= +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: election won: tally=2 + testing.go:149: cluster: 2021-08-25T09:31:13.799Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.799Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: aborting pipeline replication: peer="{Voter server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 737f9c17-f66c-2185-c80b-e9c02e1a04b5}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: election won: tally=2 +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: entering leader state: leader="Node at 737f9c17-f66c-2185-c80b-e9c02e1a04b5 [Leader]" + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.799Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: aborting pipeline replication: peer="{Voter server-737f9c17-f66c-2185-c80b-e9c02e1a04b5 737f9c17-f66c-2185-c80b-e9c02e1a04b5}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: entering leader state: leader="Node at 737f9c17-f66c-2185-c80b-e9c02e1a04b5 [Leader]" +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: added peer, starting replication: peer=server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 +testLoggerAdapter verbose: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.799Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: aborting pipeline replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: added peer, starting replication: peer=server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: added peer, starting replication: peer=server-c4a599c6-bd42-5ade-4559-1bdf940e5492 + testing.go:149: server-c4a599c6-bd42-5ade-4559-1bdf940e5492: 2021-08-25T09:31:13.799Z [INFO] server-c4a599c6-bd42-5ade-4559-1bdf940e5492: aborting pipeline replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: added peer, starting replication: peer=server-c4a599c6-bd42-5ade-4559-1bdf940e5492 +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: pipelining replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: pipelining replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: pipelining replication: peer="{Voter server-c4a599c6-bd42-5ade-4559-1bdf940e5492 c4a599c6-bd42-5ade-4559-1bdf940e5492}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.799Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: pipelining replication: peer="{Voter server-c4a599c6-bd42-5ade-4559-1bdf940e5492 c4a599c6-bd42-5ade-4559-1bdf940e5492}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.904Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.799195906 +0000 UTC m=+18.065412290 (1 nodes), 79.492µs from start of poll, 265.779286ms from cluster start. Timeout at 2021-08-25 09:31:13.904354069 +0000 UTC m=+18.170570513, 105.158223ms after stability + testing.go:149: cluster: 2021-08-25T09:31:13.904Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.799195906 +0000 UTC m=+18.065412290 (1 nodes), 79.492µs from start of poll, 265.779286ms from cluster start. Timeout at 2021-08-25 09:31:13.904354069 +0000 UTC m=+18.170570513, 105.158223ms after stability +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.904Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: aborting pipeline replication: peer="{Voter server-c4a599c6-bd42-5ade-4559-1bdf940e5492 c4a599c6-bd42-5ade-4559-1bdf940e5492}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.904Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: aborting pipeline replication: peer="{Voter server-c4a599c6-bd42-5ade-4559-1bdf940e5492 c4a599c6-bd42-5ade-4559-1bdf940e5492}" +testLoggerAdapter verbose: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.904Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: aborting pipeline replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" + testing.go:149: server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: 2021-08-25T09:31:13.904Z [INFO] server-737f9c17-f66c-2185-c80b-e9c02e1a04b5: aborting pipeline replication: peer="{Voter server-2661fbe6-8288-be2d-1199-d0e4b66b4a35 2661fbe6-8288-be2d-1199-d0e4b66b4a35}" +--- PASS: TestRaft_LeadershipTransfer (0.37s) === RUN TestRaft_LeadershipTransferWithOneNode -testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b78342e0-7634-b610-2d55-bb4ad6b0538b Address:b78342e0-7634-b610-2d55-bb4ad6b0538b}]" - testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b78342e0-7634-b610-2d55-bb4ad6b0538b Address:b78342e0-7634-b610-2d55-bb4ad6b0538b}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.037Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering follower state: follower="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:18.037Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.037Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering follower state: follower="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Follower]" leader= -testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.125Z [WARN] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.125Z [WARN] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering candidate state: node="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Candidate]" term=2 - testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering candidate state: node="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Candidate]" term=2 -testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: election won: tally=1 - testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: election won: tally=1 -testLoggerAdapter verbose: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering leader state: leader="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Leader]" - testing.go:149: server-b78342e0-7634-b610-2d55-bb4ad6b0538b: 2021-08-25T09:28:18.126Z [INFO] server-b78342e0-7634-b610-2d55-bb4ad6b0538b: entering leader state: leader="Node at b78342e0-7634-b610-2d55-bb4ad6b0538b [Leader]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.231Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.126428453 +0000 UTC m=+17.890644342 (1 nodes), 88.966811ms from start of poll, 89.52434ms from cluster start. Timeout at 2021-08-25 09:28:18.231611018 +0000 UTC m=+17.995827007, 105.182665ms after stability - testing.go:149: cluster: 2021-08-25T09:28:18.231Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.126428453 +0000 UTC m=+17.890644342 (1 nodes), 88.966811ms from start of poll, 89.52434ms from cluster start. Timeout at 2021-08-25 09:28:18.231611018 +0000 UTC m=+17.995827007, 105.182665ms after stability ---- PASS: TestRaft_LeadershipTransferWithOneNode (0.20s) +testLoggerAdapter verbose: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.908Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5 Address:d1ec148f-980d-2f1d-5f32-05ba79c50fe5}]" + testing.go:149: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.908Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5 Address:d1ec148f-980d-2f1d-5f32-05ba79c50fe5}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:13.909Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:13.909Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.909Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: entering follower state: follower="Node at d1ec148f-980d-2f1d-5f32-05ba79c50fe5 [Follower]" leader= + testing.go:149: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.909Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: entering follower state: follower="Node at d1ec148f-980d-2f1d-5f32-05ba79c50fe5 [Follower]" leader= +testLoggerAdapter verbose: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [WARN] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [WARN] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: entering candidate state: node="Node at d1ec148f-980d-2f1d-5f32-05ba79c50fe5 [Candidate]" term=2 + testing.go:149: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: entering candidate state: node="Node at d1ec148f-980d-2f1d-5f32-05ba79c50fe5 [Candidate]" term=2 +testLoggerAdapter verbose: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: election won: tally=1 + testing.go:149: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: election won: tally=1 +testLoggerAdapter verbose: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: entering leader state: leader="Node at d1ec148f-980d-2f1d-5f32-05ba79c50fe5 [Leader]" + testing.go:149: server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: 2021-08-25T09:31:13.970Z [INFO] server-d1ec148f-980d-2f1d-5f32-05ba79c50fe5: entering leader state: leader="Node at d1ec148f-980d-2f1d-5f32-05ba79c50fe5 [Leader]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.076Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.970809515 +0000 UTC m=+18.237025919 (1 nodes), 61.708592ms from start of poll, 62.735436ms from cluster start. Timeout at 2021-08-25 09:31:14.075931321 +0000 UTC m=+18.342147765, 105.121846ms after stability + testing.go:149: cluster: 2021-08-25T09:31:14.076Z [INFO] stable state for Leader reached at 2021-08-25 09:31:13.970809515 +0000 UTC m=+18.237025919 (1 nodes), 61.708592ms from start of poll, 62.735436ms from cluster start. Timeout at 2021-08-25 09:31:14.075931321 +0000 UTC m=+18.342147765, 105.121846ms after stability +--- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s) === RUN TestRaft_LeadershipTransferWithSevenNodes -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.237Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.237Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" -testLoggerAdapter verbose: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.237Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" - testing.go:149: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.237Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" -testLoggerAdapter verbose: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.238Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" - testing.go:149: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.238Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" -testLoggerAdapter verbose: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.238Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" - testing.go:149: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.238Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" -testLoggerAdapter verbose: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.238Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" - testing.go:149: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.238Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.239Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.239Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" -testLoggerAdapter verbose: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.239Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" - testing.go:149: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.239Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8082c39-cd11-8752-6816-782956e2cdcf Address:a8082c39-cd11-8752-6816-782956e2cdcf} {Suffrage:Voter ID:server-c97ba618-5b89-1423-28c9-da77f6848816 Address:c97ba618-5b89-1423-28c9-da77f6848816} {Suffrage:Voter ID:server-72bcf709-298a-8071-db2b-49e8c68dc077 Address:72bcf709-298a-8071-db2b-49e8c68dc077} {Suffrage:Voter ID:server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb Address:29bdfba5-3282-fa54-dbf7-14ad41ce8ebb} {Suffrage:Voter ID:server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e Address:29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e} {Suffrage:Voter ID:server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 Address:7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6} {Suffrage:Voter ID:server-353d806d-e21e-17bb-365b-03cccd75b8ef Address:353d806d-e21e-17bb-365b-03cccd75b8ef}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.239Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:18.239Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.239Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering follower state: follower="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Follower]" leader= - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.239Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering follower state: follower="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Follower]" leader= -testLoggerAdapter verbose: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.240Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: entering follower state: follower="Node at c97ba618-5b89-1423-28c9-da77f6848816 [Follower]" leader= - testing.go:149: server-c97ba618-5b89-1423-28c9-da77f6848816: 2021-08-25T09:28:18.240Z [INFO] server-c97ba618-5b89-1423-28c9-da77f6848816: entering follower state: follower="Node at c97ba618-5b89-1423-28c9-da77f6848816 [Follower]" leader= -testLoggerAdapter verbose: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.240Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: entering follower state: follower="Node at 72bcf709-298a-8071-db2b-49e8c68dc077 [Follower]" leader= - testing.go:149: server-72bcf709-298a-8071-db2b-49e8c68dc077: 2021-08-25T09:28:18.240Z [INFO] server-72bcf709-298a-8071-db2b-49e8c68dc077: entering follower state: follower="Node at 72bcf709-298a-8071-db2b-49e8c68dc077 [Follower]" leader= -testLoggerAdapter verbose: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.240Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: entering follower state: follower="Node at 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb [Follower]" leader= - testing.go:149: server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: 2021-08-25T09:28:18.240Z [INFO] server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb: entering follower state: follower="Node at 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb [Follower]" leader= -testLoggerAdapter verbose: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.240Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: entering follower state: follower="Node at 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e [Follower]" leader= - testing.go:149: server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: 2021-08-25T09:28:18.240Z [INFO] server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e: entering follower state: follower="Node at 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e [Follower]" leader= -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.240Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.240Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= -testLoggerAdapter verbose: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.240Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: entering follower state: follower="Node at 353d806d-e21e-17bb-365b-03cccd75b8ef [Follower]" leader= - testing.go:149: server-353d806d-e21e-17bb-365b-03cccd75b8ef: 2021-08-25T09:28:18.240Z [INFO] server-353d806d-e21e-17bb-365b-03cccd75b8ef: entering follower state: follower="Node at 353d806d-e21e-17bb-365b-03cccd75b8ef [Follower]" leader= -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [WARN] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [WARN] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering candidate state: node="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Candidate]" term=2 - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.312Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering candidate state: node="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Candidate]" term=2 -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: election won: tally=4 - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: election won: tally=4 -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering leader state: leader="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Leader]" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering leader state: leader="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Leader]" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-a8082c39-cd11-8752-6816-782956e2cdcf - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-a8082c39-cd11-8752-6816-782956e2cdcf -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.314Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.313Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.418Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.313232632 +0000 UTC m=+18.077448521 (1 nodes), 73.361862ms from start of poll, 76.259373ms from cluster start. Timeout at 2021-08-25 09:28:18.418416056 +0000 UTC m=+18.182632025, 105.183504ms after stability - testing.go:149: cluster: 2021-08-25T09:28:18.418Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.313232632 +0000 UTC m=+18.077448521 (1 nodes), 73.361862ms from start of poll, 76.259373ms from cluster start. Timeout at 2021-08-25 09:28:18.418416056 +0000 UTC m=+18.182632025, 105.183504ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.418Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:18.418Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.523Z [INFO] stable state for Follower reached at 2021-08-25 09:28:18.418678932 +0000 UTC m=+18.182894821 (6 nodes), 3.9µs from start of poll, 181.705673ms from cluster start. Timeout at 2021-08-25 09:28:18.523750287 +0000 UTC m=+18.287966196, 105.071375ms after stability - testing.go:149: cluster: 2021-08-25T09:28:18.523Z [INFO] stable state for Follower reached at 2021-08-25 09:28:18.418678932 +0000 UTC m=+18.182894821 (6 nodes), 3.9µs from start of poll, 181.705673ms from cluster start. Timeout at 2021-08-25 09:28:18.523750287 +0000 UTC m=+18.287966196, 105.071375ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.523Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:18.523Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.629Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.52404034 +0000 UTC m=+18.288256229 (1 nodes), 3.48µs from start of poll, 287.067081ms from cluster start. Timeout at 2021-08-25 09:28:18.629165609 +0000 UTC m=+18.393381578, 105.125349ms after stability - testing.go:149: cluster: 2021-08-25T09:28:18.629Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.52404034 +0000 UTC m=+18.288256229 (1 nodes), 3.48µs from start of poll, 287.067081ms from cluster start. Timeout at 2021-08-25 09:28:18.629165609 +0000 UTC m=+18.393381578, 105.125349ms after stability -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering candidate state: node="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Candidate]" term=3 - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering candidate state: node="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.629Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:18.629Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: election won: tally=4 - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: entering follower state: follower="Node at 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 [Follower]" leader= - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: election won: tally=4 -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering leader state: leader="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Leader]" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.629Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.629Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: entering leader state: leader="Node at a8082c39-cd11-8752-6816-782956e2cdcf [Leader]" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-c97ba618-5b89-1423-28c9-da77f6848816 -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-72bcf709-298a-8071-db2b-49e8c68dc077 -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: added peer, starting replication: peer=server-353d806d-e21e-17bb-365b-03cccd75b8ef -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-a8082c39-cd11-8752-6816-782956e2cdcf a8082c39-cd11-8752-6816-782956e2cdcf}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" -testLoggerAdapter verbose: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" - testing.go:149: server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: 2021-08-25T09:28:18.630Z [INFO] server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.630Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: pipelining replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.735Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.630038248 +0000 UTC m=+18.394254137 (1 nodes), 159.905µs from start of poll, 393.064989ms from cluster start. Timeout at 2021-08-25 09:28:18.735174797 +0000 UTC m=+18.499390766, 105.136629ms after stability - testing.go:149: cluster: 2021-08-25T09:28:18.735Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.630038248 +0000 UTC m=+18.394254137 (1 nodes), 159.905µs from start of poll, 393.064989ms from cluster start. Timeout at 2021-08-25 09:28:18.735174797 +0000 UTC m=+18.499390766, 105.136629ms after stability -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-72bcf709-298a-8071-db2b-49e8c68dc077 72bcf709-298a-8071-db2b-49e8c68dc077}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6 7b13ab6a-8080-e2a5-ed67-f41b9da7cbf6}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29bdfba5-3282-fa54-dbf7-14ad41ce8ebb 29bdfba5-3282-fa54-dbf7-14ad41ce8ebb}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e 29aa5b39-e8ae-44bd-1e95-c3cbe89ac41e}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-353d806d-e21e-17bb-365b-03cccd75b8ef 353d806d-e21e-17bb-365b-03cccd75b8ef}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.735Z [INFO] server-a8082c39-cd11-8752-6816-782956e2cdcf: aborting pipeline replication: peer="{Voter server-c97ba618-5b89-1423-28c9-da77f6848816 c97ba618-5b89-1423-28c9-da77f6848816}" -testLoggerAdapter verbose: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.785Z [ERROR] server-a8082c39-cd11-8752-6816-782956e2cdcf: failed to heartbeat to: peer=29bdfba5-3282-fa54-dbf7-14ad41ce8ebb error="command timed out" - testing.go:149: server-a8082c39-cd11-8752-6816-782956e2cdcf: 2021-08-25T09:28:18.785Z [ERROR] server-a8082c39-cd11-8752-6816-782956e2cdcf: failed to heartbeat to: peer=29bdfba5-3282-fa54-dbf7-14ad41ce8ebb error="command timed out" ---- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.56s) +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.081Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.081Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.081Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: entering follower state: follower="Node at 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 [Follower]" leader= + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.081Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: entering follower state: follower="Node at 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 [Follower]" leader= +testLoggerAdapter verbose: server-41e0b663-49fe-ffe9-c040-225723f99c1a: 2021-08-25T09:31:14.082Z [INFO] server-41e0b663-49fe-ffe9-c040-225723f99c1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" + testing.go:149: server-41e0b663-49fe-ffe9-c040-225723f99c1a: 2021-08-25T09:31:14.082Z [INFO] server-41e0b663-49fe-ffe9-c040-225723f99c1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" +testLoggerAdapter verbose: server-41e0b663-49fe-ffe9-c040-225723f99c1a: 2021-08-25T09:31:14.082Z [INFO] server-41e0b663-49fe-ffe9-c040-225723f99c1a: entering follower state: follower="Node at 41e0b663-49fe-ffe9-c040-225723f99c1a [Follower]" leader= + testing.go:149: server-41e0b663-49fe-ffe9-c040-225723f99c1a: 2021-08-25T09:31:14.082Z [INFO] server-41e0b663-49fe-ffe9-c040-225723f99c1a: entering follower state: follower="Node at 41e0b663-49fe-ffe9-c040-225723f99c1a [Follower]" leader= +testLoggerAdapter verbose: server-588a5d96-ef8f-731a-9c27-b4567203686d: 2021-08-25T09:31:14.082Z [INFO] server-588a5d96-ef8f-731a-9c27-b4567203686d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" + testing.go:149: server-588a5d96-ef8f-731a-9c27-b4567203686d: 2021-08-25T09:31:14.082Z [INFO] server-588a5d96-ef8f-731a-9c27-b4567203686d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" +testLoggerAdapter verbose: server-588a5d96-ef8f-731a-9c27-b4567203686d: 2021-08-25T09:31:14.082Z [INFO] server-588a5d96-ef8f-731a-9c27-b4567203686d: entering follower state: follower="Node at 588a5d96-ef8f-731a-9c27-b4567203686d [Follower]" leader= + testing.go:149: server-588a5d96-ef8f-731a-9c27-b4567203686d: 2021-08-25T09:31:14.082Z [INFO] server-588a5d96-ef8f-731a-9c27-b4567203686d: entering follower state: follower="Node at 588a5d96-ef8f-731a-9c27-b4567203686d [Follower]" leader= +testLoggerAdapter verbose: server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: 2021-08-25T09:31:14.082Z [INFO] server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" + testing.go:149: server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: 2021-08-25T09:31:14.082Z [INFO] server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" +testLoggerAdapter verbose: server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: 2021-08-25T09:31:14.083Z [INFO] server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: entering follower state: follower="Node at 1fe4a98f-3e54-1fd5-cf32-62fafa19c811 [Follower]" leader= + testing.go:149: server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: 2021-08-25T09:31:14.083Z [INFO] server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811: entering follower state: follower="Node at 1fe4a98f-3e54-1fd5-cf32-62fafa19c811 [Follower]" leader= +testLoggerAdapter verbose: server-2003ea67-0ece-50a0-7a36-e2804e724e41: 2021-08-25T09:31:14.083Z [INFO] server-2003ea67-0ece-50a0-7a36-e2804e724e41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" + testing.go:149: server-2003ea67-0ece-50a0-7a36-e2804e724e41: 2021-08-25T09:31:14.083Z [INFO] server-2003ea67-0ece-50a0-7a36-e2804e724e41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" +testLoggerAdapter verbose: server-2003ea67-0ece-50a0-7a36-e2804e724e41: 2021-08-25T09:31:14.083Z [INFO] server-2003ea67-0ece-50a0-7a36-e2804e724e41: entering follower state: follower="Node at 2003ea67-0ece-50a0-7a36-e2804e724e41 [Follower]" leader= + testing.go:149: server-2003ea67-0ece-50a0-7a36-e2804e724e41: 2021-08-25T09:31:14.083Z [INFO] server-2003ea67-0ece-50a0-7a36-e2804e724e41: entering follower state: follower="Node at 2003ea67-0ece-50a0-7a36-e2804e724e41 [Follower]" leader= +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.083Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.083Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.083Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering follower state: follower="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Follower]" leader= + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.083Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering follower state: follower="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Follower]" leader= +testLoggerAdapter verbose: server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: 2021-08-25T09:31:14.084Z [INFO] server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" + testing.go:149: server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: 2021-08-25T09:31:14.084Z [INFO] server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 Address:42c1f716-2b47-4f4e-80a2-3ca8cc11dea6} {Suffrage:Voter ID:server-41e0b663-49fe-ffe9-c040-225723f99c1a Address:41e0b663-49fe-ffe9-c040-225723f99c1a} {Suffrage:Voter ID:server-588a5d96-ef8f-731a-9c27-b4567203686d Address:588a5d96-ef8f-731a-9c27-b4567203686d} {Suffrage:Voter ID:server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 Address:1fe4a98f-3e54-1fd5-cf32-62fafa19c811} {Suffrage:Voter ID:server-2003ea67-0ece-50a0-7a36-e2804e724e41 Address:2003ea67-0ece-50a0-7a36-e2804e724e41} {Suffrage:Voter ID:server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 Address:31286ec1-9bfe-8cf5-f555-4af451e5a6b2} {Suffrage:Voter ID:server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 Address:acdc9fbe-34fc-f5e5-5372-579b9c5ae972}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.084Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: 2021-08-25T09:31:14.084Z [INFO] server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: entering follower state: follower="Node at acdc9fbe-34fc-f5e5-5372-579b9c5ae972 [Follower]" leader= + testing.go:149: cluster: 2021-08-25T09:31:14.084Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: 2021-08-25T09:31:14.084Z [INFO] server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972: entering follower state: follower="Node at acdc9fbe-34fc-f5e5-5372-579b9c5ae972 [Follower]" leader= +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.135Z [WARN] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.135Z [WARN] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.135Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering candidate state: node="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Candidate]" term=2 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.135Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering candidate state: node="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Candidate]" term=2 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.135Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: election won: tally=4 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.135Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: election won: tally=4 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering leader state: leader="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Leader]" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering leader state: leader="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Leader]" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-41e0b663-49fe-ffe9-c040-225723f99c1a + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-41e0b663-49fe-ffe9-c040-225723f99c1a +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-588a5d96-ef8f-731a-9c27-b4567203686d + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-588a5d96-ef8f-731a-9c27-b4567203686d +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-2003ea67-0ece-50a0-7a36-e2804e724e41 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-2003ea67-0ece-50a0-7a36-e2804e724e41 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: added peer, starting replication: peer=server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.136Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: pipelining replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.241Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.136097439 +0000 UTC m=+18.402313883 (1 nodes), 51.8555ms from start of poll, 54.802682ms from cluster start. Timeout at 2021-08-25 09:31:14.241154991 +0000 UTC m=+18.507371395, 105.057512ms after stability + testing.go:149: cluster: 2021-08-25T09:31:14.241Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.136097439 +0000 UTC m=+18.402313883 (1 nodes), 51.8555ms from start of poll, 54.802682ms from cluster start. Timeout at 2021-08-25 09:31:14.241154991 +0000 UTC m=+18.507371395, 105.057512ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.241Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:14.241Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.346Z [INFO] stable state for Follower reached at 2021-08-25 09:31:14.241418166 +0000 UTC m=+18.507634570 (6 nodes), 4.959µs from start of poll, 160.123369ms from cluster start. Timeout at 2021-08-25 09:31:14.346476679 +0000 UTC m=+18.612693083, 105.058513ms after stability + testing.go:149: cluster: 2021-08-25T09:31:14.346Z [INFO] stable state for Follower reached at 2021-08-25 09:31:14.241418166 +0000 UTC m=+18.507634570 (6 nodes), 4.959µs from start of poll, 160.123369ms from cluster start. Timeout at 2021-08-25 09:31:14.346476679 +0000 UTC m=+18.612693083, 105.058513ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.346Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:14.346Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.452Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.346753113 +0000 UTC m=+18.612969557 (1 nodes), 3.44µs from start of poll, 265.458356ms from cluster start. Timeout at 2021-08-25 09:31:14.451965991 +0000 UTC m=+18.718182475, 105.212918ms after stability + testing.go:149: cluster: 2021-08-25T09:31:14.452Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.346753113 +0000 UTC m=+18.612969557 (1 nodes), 3.44µs from start of poll, 265.458356ms from cluster start. Timeout at 2021-08-25 09:31:14.451965991 +0000 UTC m=+18.718182475, 105.212918ms after stability +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: entering candidate state: node="Node at 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.452Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: entering candidate state: node="Node at 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 [Candidate]" term=3 + testing.go:149: cluster: 2021-08-25T09:31:14.452Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: election won: tally=4 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering follower state: follower="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Follower]" leader= + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: election won: tally=4 +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: entering leader state: leader="Node at 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 [Leader]" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: entering leader state: leader="Node at 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 [Leader]" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-41e0b663-49fe-ffe9-c040-225723f99c1a + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-41e0b663-49fe-ffe9-c040-225723f99c1a +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-588a5d96-ef8f-731a-9c27-b4567203686d + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: entering follower state: follower="Node at 31286ec1-9bfe-8cf5-f555-4af451e5a6b2 [Follower]" leader= + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-588a5d96-ef8f-731a-9c27-b4567203686d +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-2003ea67-0ece-50a0-7a36-e2804e724e41 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-2003ea67-0ece-50a0-7a36-e2804e724e41 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.452Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6 42c1f716-2b47-4f4e-80a2-3ca8cc11dea6}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 +testLoggerAdapter verbose: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 + testing.go:149: server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: 2021-08-25T09:31:14.452Z [INFO] server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2: aborting pipeline replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: added peer, starting replication: peer=server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 31286ec1-9bfe-8cf5-f555-4af451e5a6b2}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 31286ec1-9bfe-8cf5-f555-4af451e5a6b2}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.453Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: pipelining replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.557Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.452702441 +0000 UTC m=+18.718918845 (1 nodes), 95.331µs from start of poll, 371.407644ms from cluster start. Timeout at 2021-08-25 09:31:14.557740016 +0000 UTC m=+18.823956500, 105.037655ms after stability + testing.go:149: cluster: 2021-08-25T09:31:14.557Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.452702441 +0000 UTC m=+18.718918845 (1 nodes), 95.331µs from start of poll, 371.407644ms from cluster start. Timeout at 2021-08-25 09:31:14.557740016 +0000 UTC m=+18.823956500, 105.037655ms after stability +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-acdc9fbe-34fc-f5e5-5372-579b9c5ae972 acdc9fbe-34fc-f5e5-5372-579b9c5ae972}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-41e0b663-49fe-ffe9-c040-225723f99c1a 41e0b663-49fe-ffe9-c040-225723f99c1a}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-1fe4a98f-3e54-1fd5-cf32-62fafa19c811 1fe4a98f-3e54-1fd5-cf32-62fafa19c811}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-588a5d96-ef8f-731a-9c27-b4567203686d 588a5d96-ef8f-731a-9c27-b4567203686d}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-2003ea67-0ece-50a0-7a36-e2804e724e41 2003ea67-0ece-50a0-7a36-e2804e724e41}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 31286ec1-9bfe-8cf5-f555-4af451e5a6b2}" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.558Z [INFO] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: aborting pipeline replication: peer="{Voter server-31286ec1-9bfe-8cf5-f555-4af451e5a6b2 31286ec1-9bfe-8cf5-f555-4af451e5a6b2}" +testLoggerAdapter verbose: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.608Z [ERROR] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: failed to heartbeat to: peer=1fe4a98f-3e54-1fd5-cf32-62fafa19c811 error="command timed out" + testing.go:149: server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: 2021-08-25T09:31:14.608Z [ERROR] server-42c1f716-2b47-4f4e-80a2-3ca8cc11dea6: failed to heartbeat to: peer=1fe4a98f-3e54-1fd5-cf32-62fafa19c811 error="command timed out" +--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.54s) === RUN TestRaft_LeadershipTransferToInvalidID -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering follower state: follower="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Follower]" leader= - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.795Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering follower state: follower="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Follower]" leader= -testLoggerAdapter verbose: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.795Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" - testing.go:149: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.795Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" -testLoggerAdapter verbose: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" - testing.go:149: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c Address:34f0726e-82c8-3ea9-b7cb-ac91ba463d0c} {Suffrage:Voter ID:server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb Address:76da4f5b-7ebc-82a9-60e7-20ec4abda3bb} {Suffrage:Voter ID:server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d Address:78dfc85e-3399-7889-886a-3c2cbd2b8f9d}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.796Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:18.796Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.796Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: entering follower state: follower="Node at 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb [Follower]" leader= - testing.go:149: server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: 2021-08-25T09:28:18.796Z [INFO] server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb: entering follower state: follower="Node at 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb [Follower]" leader= -testLoggerAdapter verbose: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: entering follower state: follower="Node at 78dfc85e-3399-7889-886a-3c2cbd2b8f9d [Follower]" leader= - testing.go:149: server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: 2021-08-25T09:28:18.796Z [INFO] server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d: entering follower state: follower="Node at 78dfc85e-3399-7889-886a-3c2cbd2b8f9d [Follower]" leader= -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.860Z [WARN] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.860Z [WARN] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering candidate state: node="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Candidate]" term=2 - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering candidate state: node="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Candidate]" term=2 -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: election won: tally=2 - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: election won: tally=2 -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering leader state: leader="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Leader]" - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: entering leader state: leader="Node at 34f0726e-82c8-3ea9-b7cb-ac91ba463d0c [Leader]" -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: added peer, starting replication: peer=server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.861Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.862Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.862Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: pipelining replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.967Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.86200483 +0000 UTC m=+18.626220739 (1 nodes), 65.523611ms from start of poll, 66.895943ms from cluster start. Timeout at 2021-08-25 09:28:18.967196413 +0000 UTC m=+18.731412302, 105.191563ms after stability - testing.go:149: cluster: 2021-08-25T09:28:18.967Z [INFO] stable state for Leader reached at 2021-08-25 09:28:18.86200483 +0000 UTC m=+18.626220739 (1 nodes), 65.523611ms from start of poll, 66.895943ms from cluster start. Timeout at 2021-08-25 09:28:18.967196413 +0000 UTC m=+18.731412302, 105.191563ms after stability -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-78dfc85e-3399-7889-886a-3c2cbd2b8f9d 78dfc85e-3399-7889-886a-3c2cbd2b8f9d}" -testLoggerAdapter verbose: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" - testing.go:149: server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: 2021-08-25T09:28:18.967Z [INFO] server-34f0726e-82c8-3ea9-b7cb-ac91ba463d0c: aborting pipeline replication: peer="{Voter server-76da4f5b-7ebc-82a9-60e7-20ec4abda3bb 76da4f5b-7ebc-82a9-60e7-20ec4abda3bb}" +testLoggerAdapter verbose: server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: 2021-08-25T09:31:14.615Z [INFO] server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf Address:0b0e4fc4-3705-56ec-c1ec-997db03f9ebf} {Suffrage:Voter ID:server-25774ff1-1b28-6a84-d8b1-feda5ab95927 Address:25774ff1-1b28-6a84-d8b1-feda5ab95927} {Suffrage:Voter ID:server-30796896-b483-4c89-9cab-b3ed5be09305 Address:30796896-b483-4c89-9cab-b3ed5be09305}]" + testing.go:149: server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: 2021-08-25T09:31:14.615Z [INFO] server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf Address:0b0e4fc4-3705-56ec-c1ec-997db03f9ebf} {Suffrage:Voter ID:server-25774ff1-1b28-6a84-d8b1-feda5ab95927 Address:25774ff1-1b28-6a84-d8b1-feda5ab95927} {Suffrage:Voter ID:server-30796896-b483-4c89-9cab-b3ed5be09305 Address:30796896-b483-4c89-9cab-b3ed5be09305}]" +testLoggerAdapter verbose: server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: 2021-08-25T09:31:14.616Z [INFO] server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: entering follower state: follower="Node at 0b0e4fc4-3705-56ec-c1ec-997db03f9ebf [Follower]" leader= + testing.go:149: server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: 2021-08-25T09:31:14.616Z [INFO] server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf: entering follower state: follower="Node at 0b0e4fc4-3705-56ec-c1ec-997db03f9ebf [Follower]" leader= +testLoggerAdapter verbose: server-25774ff1-1b28-6a84-d8b1-feda5ab95927: 2021-08-25T09:31:14.616Z [INFO] server-25774ff1-1b28-6a84-d8b1-feda5ab95927: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf Address:0b0e4fc4-3705-56ec-c1ec-997db03f9ebf} {Suffrage:Voter ID:server-25774ff1-1b28-6a84-d8b1-feda5ab95927 Address:25774ff1-1b28-6a84-d8b1-feda5ab95927} {Suffrage:Voter ID:server-30796896-b483-4c89-9cab-b3ed5be09305 Address:30796896-b483-4c89-9cab-b3ed5be09305}]" + testing.go:149: server-25774ff1-1b28-6a84-d8b1-feda5ab95927: 2021-08-25T09:31:14.616Z [INFO] server-25774ff1-1b28-6a84-d8b1-feda5ab95927: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf Address:0b0e4fc4-3705-56ec-c1ec-997db03f9ebf} {Suffrage:Voter ID:server-25774ff1-1b28-6a84-d8b1-feda5ab95927 Address:25774ff1-1b28-6a84-d8b1-feda5ab95927} {Suffrage:Voter ID:server-30796896-b483-4c89-9cab-b3ed5be09305 Address:30796896-b483-4c89-9cab-b3ed5be09305}]" +testLoggerAdapter verbose: server-25774ff1-1b28-6a84-d8b1-feda5ab95927: 2021-08-25T09:31:14.616Z [INFO] server-25774ff1-1b28-6a84-d8b1-feda5ab95927: entering follower state: follower="Node at 25774ff1-1b28-6a84-d8b1-feda5ab95927 [Follower]" leader= + testing.go:149: server-25774ff1-1b28-6a84-d8b1-feda5ab95927: 2021-08-25T09:31:14.616Z [INFO] server-25774ff1-1b28-6a84-d8b1-feda5ab95927: entering follower state: follower="Node at 25774ff1-1b28-6a84-d8b1-feda5ab95927 [Follower]" leader= +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.616Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf Address:0b0e4fc4-3705-56ec-c1ec-997db03f9ebf} {Suffrage:Voter ID:server-25774ff1-1b28-6a84-d8b1-feda5ab95927 Address:25774ff1-1b28-6a84-d8b1-feda5ab95927} {Suffrage:Voter ID:server-30796896-b483-4c89-9cab-b3ed5be09305 Address:30796896-b483-4c89-9cab-b3ed5be09305}]" + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.616Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf Address:0b0e4fc4-3705-56ec-c1ec-997db03f9ebf} {Suffrage:Voter ID:server-25774ff1-1b28-6a84-d8b1-feda5ab95927 Address:25774ff1-1b28-6a84-d8b1-feda5ab95927} {Suffrage:Voter ID:server-30796896-b483-4c89-9cab-b3ed5be09305 Address:30796896-b483-4c89-9cab-b3ed5be09305}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.616Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:14.616Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.616Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: entering follower state: follower="Node at 30796896-b483-4c89-9cab-b3ed5be09305 [Follower]" leader= + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.616Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: entering follower state: follower="Node at 30796896-b483-4c89-9cab-b3ed5be09305 [Follower]" leader= +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [WARN] server-30796896-b483-4c89-9cab-b3ed5be09305: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [WARN] server-30796896-b483-4c89-9cab-b3ed5be09305: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: entering candidate state: node="Node at 30796896-b483-4c89-9cab-b3ed5be09305 [Candidate]" term=2 + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: entering candidate state: node="Node at 30796896-b483-4c89-9cab-b3ed5be09305 [Candidate]" term=2 +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: election won: tally=2 + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: election won: tally=2 +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: entering leader state: leader="Node at 30796896-b483-4c89-9cab-b3ed5be09305 [Leader]" + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: entering leader state: leader="Node at 30796896-b483-4c89-9cab-b3ed5be09305 [Leader]" +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: added peer, starting replication: peer=server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: added peer, starting replication: peer=server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: added peer, starting replication: peer=server-25774ff1-1b28-6a84-d8b1-feda5ab95927 + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: added peer, starting replication: peer=server-25774ff1-1b28-6a84-d8b1-feda5ab95927 +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: pipelining replication: peer="{Voter server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf 0b0e4fc4-3705-56ec-c1ec-997db03f9ebf}" + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: pipelining replication: peer="{Voter server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf 0b0e4fc4-3705-56ec-c1ec-997db03f9ebf}" +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: pipelining replication: peer="{Voter server-25774ff1-1b28-6a84-d8b1-feda5ab95927 25774ff1-1b28-6a84-d8b1-feda5ab95927}" + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.681Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: pipelining replication: peer="{Voter server-25774ff1-1b28-6a84-d8b1-feda5ab95927 25774ff1-1b28-6a84-d8b1-feda5ab95927}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.786Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.681787741 +0000 UTC m=+18.948004145 (1 nodes), 65.126491ms from start of poll, 66.19117ms from cluster start. Timeout at 2021-08-25 09:31:14.786818456 +0000 UTC m=+19.053034920, 105.030775ms after stability + testing.go:149: cluster: 2021-08-25T09:31:14.786Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.681787741 +0000 UTC m=+18.948004145 (1 nodes), 65.126491ms from start of poll, 66.19117ms from cluster start. Timeout at 2021-08-25 09:31:14.786818456 +0000 UTC m=+19.053034920, 105.030775ms after stability +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.787Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: aborting pipeline replication: peer="{Voter server-25774ff1-1b28-6a84-d8b1-feda5ab95927 25774ff1-1b28-6a84-d8b1-feda5ab95927}" + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.787Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: aborting pipeline replication: peer="{Voter server-25774ff1-1b28-6a84-d8b1-feda5ab95927 25774ff1-1b28-6a84-d8b1-feda5ab95927}" +testLoggerAdapter verbose: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.787Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: aborting pipeline replication: peer="{Voter server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf 0b0e4fc4-3705-56ec-c1ec-997db03f9ebf}" + testing.go:149: server-30796896-b483-4c89-9cab-b3ed5be09305: 2021-08-25T09:31:14.787Z [INFO] server-30796896-b483-4c89-9cab-b3ed5be09305: aborting pipeline replication: peer="{Voter server-0b0e4fc4-3705-56ec-c1ec-997db03f9ebf 0b0e4fc4-3705-56ec-c1ec-997db03f9ebf}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s) === RUN TestRaft_LeadershipTransferToInvalidAddress -testLoggerAdapter verbose: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" - testing.go:149: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" -testLoggerAdapter verbose: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: entering follower state: follower="Node at f964fe5d-6ec0-adef-29a9-bb2bf093482b [Follower]" leader= - testing.go:149: server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: 2021-08-25T09:28:18.971Z [INFO] server-f964fe5d-6ec0-adef-29a9-bb2bf093482b: entering follower state: follower="Node at f964fe5d-6ec0-adef-29a9-bb2bf093482b [Follower]" leader= -testLoggerAdapter verbose: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" - testing.go:149: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" -testLoggerAdapter verbose: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: entering follower state: follower="Node at 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb [Follower]" leader= - testing.go:149: server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: 2021-08-25T09:28:18.971Z [INFO] server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb: entering follower state: follower="Node at 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb [Follower]" leader= -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f964fe5d-6ec0-adef-29a9-bb2bf093482b Address:f964fe5d-6ec0-adef-29a9-bb2bf093482b} {Suffrage:Voter ID:server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb Address:9bb68fd8-217c-79c1-a5ba-b80a222ef6eb} {Suffrage:Voter ID:server-95183a1a-a854-3cf0-1f2a-ec0e624f7351 Address:95183a1a-a854-3cf0-1f2a-ec0e624f7351}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:18.971Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:18.971Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering follower state: follower="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Follower]" leader= - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:18.971Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering follower state: follower="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Follower]" leader= -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [WARN] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [WARN] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering candidate state: node="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Candidate]" term=2 - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.047Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering candidate state: node="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Candidate]" term=2 -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: election won: tally=2 - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: election won: tally=2 -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering leader state: leader="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Leader]" - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: entering leader state: leader="Node at 95183a1a-a854-3cf0-1f2a-ec0e624f7351 [Leader]" -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-f964fe5d-6ec0-adef-29a9-bb2bf093482b - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-f964fe5d-6ec0-adef-29a9-bb2bf093482b -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: added peer, starting replication: peer=server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.048Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: pipelining replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.153Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.048277438 +0000 UTC m=+18.812493347 (2 nodes), 76.254833ms from start of poll, 77.37053ms from cluster start. Timeout at 2021-08-25 09:28:19.153407987 +0000 UTC m=+18.917623956, 105.130609ms after stability - testing.go:149: cluster: 2021-08-25T09:28:19.153Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.048277438 +0000 UTC m=+18.812493347 (2 nodes), 76.254833ms from start of poll, 77.37053ms from cluster start. Timeout at 2021-08-25 09:28:19.153407987 +0000 UTC m=+18.917623956, 105.130609ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.153Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:19.153Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.258Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.153733857 +0000 UTC m=+18.917949746 (1 nodes), 2.44µs from start of poll, 182.826929ms from cluster start. Timeout at 2021-08-25 09:28:19.258833329 +0000 UTC m=+19.023049278, 105.099532ms after stability - testing.go:149: cluster: 2021-08-25T09:28:19.258Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.153733857 +0000 UTC m=+18.917949746 (1 nodes), 2.44µs from start of poll, 182.826929ms from cluster start. Timeout at 2021-08-25 09:28:19.258833329 +0000 UTC m=+19.023049278, 105.099532ms after stability -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-9bb68fd8-217c-79c1-a5ba-b80a222ef6eb 9bb68fd8-217c-79c1-a5ba-b80a222ef6eb}" -testLoggerAdapter verbose: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" - testing.go:149: server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: 2021-08-25T09:28:19.259Z [INFO] server-95183a1a-a854-3cf0-1f2a-ec0e624f7351: aborting pipeline replication: peer="{Voter server-f964fe5d-6ec0-adef-29a9-bb2bf093482b f964fe5d-6ec0-adef-29a9-bb2bf093482b}" ---- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) +testLoggerAdapter verbose: server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: 2021-08-25T09:31:14.791Z [INFO] server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 Address:b53d6d52-0ad5-72c0-ae98-e594ec3a5080} {Suffrage:Voter ID:server-89f5186a-fa60-689c-2325-d07daafc52b2 Address:89f5186a-fa60-689c-2325-d07daafc52b2} {Suffrage:Voter ID:server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 Address:0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}]" + testing.go:149: server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: 2021-08-25T09:31:14.791Z [INFO] server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 Address:b53d6d52-0ad5-72c0-ae98-e594ec3a5080} {Suffrage:Voter ID:server-89f5186a-fa60-689c-2325-d07daafc52b2 Address:89f5186a-fa60-689c-2325-d07daafc52b2} {Suffrage:Voter ID:server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 Address:0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}]" +testLoggerAdapter verbose: server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: 2021-08-25T09:31:14.791Z [INFO] server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: entering follower state: follower="Node at b53d6d52-0ad5-72c0-ae98-e594ec3a5080 [Follower]" leader= + testing.go:149: server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: 2021-08-25T09:31:14.791Z [INFO] server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080: entering follower state: follower="Node at b53d6d52-0ad5-72c0-ae98-e594ec3a5080 [Follower]" leader= +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.791Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 Address:b53d6d52-0ad5-72c0-ae98-e594ec3a5080} {Suffrage:Voter ID:server-89f5186a-fa60-689c-2325-d07daafc52b2 Address:89f5186a-fa60-689c-2325-d07daafc52b2} {Suffrage:Voter ID:server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 Address:0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}]" + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.791Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 Address:b53d6d52-0ad5-72c0-ae98-e594ec3a5080} {Suffrage:Voter ID:server-89f5186a-fa60-689c-2325-d07daafc52b2 Address:89f5186a-fa60-689c-2325-d07daafc52b2} {Suffrage:Voter ID:server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 Address:0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}]" +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.791Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: entering follower state: follower="Node at 89f5186a-fa60-689c-2325-d07daafc52b2 [Follower]" leader= + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.791Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: entering follower state: follower="Node at 89f5186a-fa60-689c-2325-d07daafc52b2 [Follower]" leader= +testLoggerAdapter verbose: server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: 2021-08-25T09:31:14.791Z [INFO] server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 Address:b53d6d52-0ad5-72c0-ae98-e594ec3a5080} {Suffrage:Voter ID:server-89f5186a-fa60-689c-2325-d07daafc52b2 Address:89f5186a-fa60-689c-2325-d07daafc52b2} {Suffrage:Voter ID:server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 Address:0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}]" + testing.go:149: server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: 2021-08-25T09:31:14.791Z [INFO] server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 Address:b53d6d52-0ad5-72c0-ae98-e594ec3a5080} {Suffrage:Voter ID:server-89f5186a-fa60-689c-2325-d07daafc52b2 Address:89f5186a-fa60-689c-2325-d07daafc52b2} {Suffrage:Voter ID:server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 Address:0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.792Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:14.792Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: 2021-08-25T09:31:14.792Z [INFO] server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: entering follower state: follower="Node at 0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 [Follower]" leader= + testing.go:149: server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: 2021-08-25T09:31:14.792Z [INFO] server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865: entering follower state: follower="Node at 0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 [Follower]" leader= +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [WARN] server-89f5186a-fa60-689c-2325-d07daafc52b2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [WARN] server-89f5186a-fa60-689c-2325-d07daafc52b2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: entering candidate state: node="Node at 89f5186a-fa60-689c-2325-d07daafc52b2 [Candidate]" term=2 + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: entering candidate state: node="Node at 89f5186a-fa60-689c-2325-d07daafc52b2 [Candidate]" term=2 +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: election won: tally=2 + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: election won: tally=2 +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: entering leader state: leader="Node at 89f5186a-fa60-689c-2325-d07daafc52b2 [Leader]" + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: entering leader state: leader="Node at 89f5186a-fa60-689c-2325-d07daafc52b2 [Leader]" +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: added peer, starting replication: peer=server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: added peer, starting replication: peer=server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: added peer, starting replication: peer=server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: added peer, starting replication: peer=server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: pipelining replication: peer="{Voter server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 b53d6d52-0ad5-72c0-ae98-e594ec3a5080}" + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: pipelining replication: peer="{Voter server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 b53d6d52-0ad5-72c0-ae98-e594ec3a5080}" +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: pipelining replication: peer="{Voter server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}" + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:14.847Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: pipelining replication: peer="{Voter server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.952Z [INFO] stable state for Follower reached at 2021-08-25 09:31:14.847494885 +0000 UTC m=+19.113711309 (2 nodes), 55.333492ms from start of poll, 56.560996ms from cluster start. Timeout at 2021-08-25 09:31:14.952609353 +0000 UTC m=+19.218825836, 105.114527ms after stability + testing.go:149: cluster: 2021-08-25T09:31:14.952Z [INFO] stable state for Follower reached at 2021-08-25 09:31:14.847494885 +0000 UTC m=+19.113711309 (2 nodes), 55.333492ms from start of poll, 56.560996ms from cluster start. Timeout at 2021-08-25 09:31:14.952609353 +0000 UTC m=+19.218825836, 105.114527ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:14.952Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:14.952Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.058Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.952929842 +0000 UTC m=+19.219146246 (1 nodes), 2.319µs from start of poll, 161.995933ms from cluster start. Timeout at 2021-08-25 09:31:15.058016992 +0000 UTC m=+19.324233496, 105.08725ms after stability + testing.go:149: cluster: 2021-08-25T09:31:15.058Z [INFO] stable state for Leader reached at 2021-08-25 09:31:14.952929842 +0000 UTC m=+19.219146246 (1 nodes), 2.319µs from start of poll, 161.995933ms from cluster start. Timeout at 2021-08-25 09:31:15.058016992 +0000 UTC m=+19.324233496, 105.08725ms after stability +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:15.058Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: aborting pipeline replication: peer="{Voter server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}" + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:15.058Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: aborting pipeline replication: peer="{Voter server-0931e6f0-ca2c-1493-3aa3-ed47b8ae2865 0931e6f0-ca2c-1493-3aa3-ed47b8ae2865}" +testLoggerAdapter verbose: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:15.058Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: aborting pipeline replication: peer="{Voter server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 b53d6d52-0ad5-72c0-ae98-e594ec3a5080}" + testing.go:149: server-89f5186a-fa60-689c-2325-d07daafc52b2: 2021-08-25T09:31:15.058Z [INFO] server-89f5186a-fa60-689c-2325-d07daafc52b2: aborting pipeline replication: peer="{Voter server-b53d6d52-0ad5-72c0-ae98-e594ec3a5080 b53d6d52-0ad5-72c0-ae98-e594ec3a5080}" +--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering follower state: follower="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Follower]" leader= - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.263Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering follower state: follower="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Follower]" leader= -testLoggerAdapter verbose: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" - testing.go:149: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" -testLoggerAdapter verbose: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: entering follower state: follower="Node at 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf [Follower]" leader= - testing.go:149: server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: 2021-08-25T09:28:19.263Z [INFO] server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf: entering follower state: follower="Node at 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf [Follower]" leader= -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cda7d2-fe36-9464-1536-6f6844ad3282 Address:96cda7d2-fe36-9464-1536-6f6844ad3282} {Suffrage:Voter ID:server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf Address:5e246de8-3a20-dc9b-40c0-f4f3fcc616bf} {Suffrage:Voter ID:server-86af87f3-da1d-8428-5c02-ceabc30af9c2 Address:86af87f3-da1d-8428-5c02-ceabc30af9c2}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.264Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:19.264Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.264Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [WARN] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [WARN] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering candidate state: node="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Candidate]" term=2 - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering candidate state: node="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Candidate]" term=2 -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: election won: tally=2 - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: election won: tally=2 -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering leader state: leader="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Leader]" - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.328Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering leader state: leader="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Leader]" -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-96cda7d2-fe36-9464-1536-6f6844ad3282 - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-96cda7d2-fe36-9464-1536-6f6844ad3282 -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.329Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.329094759 +0000 UTC m=+19.093310648 (1 nodes), 64.655971ms from start of poll, 65.970649ms from cluster start. Timeout at 2021-08-25 09:28:19.434246306 +0000 UTC m=+19.198462215, 105.151567ms after stability - testing.go:149: cluster: 2021-08-25T09:28:19.434Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.329094759 +0000 UTC m=+19.093310648 (1 nodes), 64.655971ms from start of poll, 65.970649ms from cluster start. Timeout at 2021-08-25 09:28:19.434246306 +0000 UTC m=+19.198462215, 105.151567ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.434Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:19.434Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.539Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.43452714 +0000 UTC m=+19.198743049 (2 nodes), 2.78µs from start of poll, 171.40305ms from cluster start. Timeout at 2021-08-25 09:28:19.539688307 +0000 UTC m=+19.303904276, 105.161227ms after stability - testing.go:149: cluster: 2021-08-25T09:28:19.539Z [INFO] stable state for Follower reached at 2021-08-25 09:28:19.43452714 +0000 UTC m=+19.198743049 (2 nodes), 2.78µs from start of poll, 171.40305ms from cluster start. Timeout at 2021-08-25 09:28:19.539688307 +0000 UTC m=+19.303904276, 105.161227ms after stability -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering candidate state: node="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Candidate]" term=3 - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering candidate state: node="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.578Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= - testing.go:149: cluster: 2021-08-25T09:28:19.578Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: election won: tally=2 - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: entering follower state: follower="Node at 86af87f3-da1d-8428-5c02-ceabc30af9c2 [Follower]" leader= -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: election won: tally=2 -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering leader state: leader="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Leader]" - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-96cda7d2-fe36-9464-1536-6f6844ad3282 96cda7d2-fe36-9464-1536-6f6844ad3282}" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: entering leader state: leader="Node at 96cda7d2-fe36-9464-1536-6f6844ad3282 [Leader]" -testLoggerAdapter verbose: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf - testing.go:149: server-86af87f3-da1d-8428-5c02-ceabc30af9c2: 2021-08-25T09:28:19.578Z [INFO] server-86af87f3-da1d-8428-5c02-ceabc30af9c2: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.578Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-86af87f3-da1d-8428-5c02-ceabc30af9c2 - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: added peer, starting replication: peer=server-86af87f3-da1d-8428-5c02-ceabc30af9c2 -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.579Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: pipelining replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.684Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.57881843 +0000 UTC m=+19.343034319 (1 nodes), 291.713µs from start of poll, 315.69432ms from cluster start. Timeout at 2021-08-25 09:28:19.684143501 +0000 UTC m=+19.448359450, 105.325131ms after stability - testing.go:149: cluster: 2021-08-25T09:28:19.684Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.57881843 +0000 UTC m=+19.343034319 (1 nodes), 291.713µs from start of poll, 315.69432ms from cluster start. Timeout at 2021-08-25 09:28:19.684143501 +0000 UTC m=+19.448359450, 105.325131ms after stability -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-86af87f3-da1d-8428-5c02-ceabc30af9c2 86af87f3-da1d-8428-5c02-ceabc30af9c2}" -testLoggerAdapter verbose: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" - testing.go:149: server-96cda7d2-fe36-9464-1536-6f6844ad3282: 2021-08-25T09:28:19.684Z [INFO] server-96cda7d2-fe36-9464-1536-6f6844ad3282: aborting pipeline replication: peer="{Voter server-5e246de8-3a20-dc9b-40c0-f4f3fcc616bf 5e246de8-3a20-dc9b-40c0-f4f3fcc616bf}" +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.062Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c188701-f54a-d72f-8c33-efb56875b3c5 Address:1c188701-f54a-d72f-8c33-efb56875b3c5} {Suffrage:Voter ID:server-72c2c775-6105-6649-87b2-62bdbb54ab42 Address:72c2c775-6105-6649-87b2-62bdbb54ab42} {Suffrage:Voter ID:server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 Address:5971c68c-2edf-c1f2-b958-95bf0e2008a0}]" + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.062Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c188701-f54a-d72f-8c33-efb56875b3c5 Address:1c188701-f54a-d72f-8c33-efb56875b3c5} {Suffrage:Voter ID:server-72c2c775-6105-6649-87b2-62bdbb54ab42 Address:72c2c775-6105-6649-87b2-62bdbb54ab42} {Suffrage:Voter ID:server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 Address:5971c68c-2edf-c1f2-b958-95bf0e2008a0}]" +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.063Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: entering follower state: follower="Node at 1c188701-f54a-d72f-8c33-efb56875b3c5 [Follower]" leader= + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.063Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: entering follower state: follower="Node at 1c188701-f54a-d72f-8c33-efb56875b3c5 [Follower]" leader= +testLoggerAdapter verbose: server-72c2c775-6105-6649-87b2-62bdbb54ab42: 2021-08-25T09:31:15.063Z [INFO] server-72c2c775-6105-6649-87b2-62bdbb54ab42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c188701-f54a-d72f-8c33-efb56875b3c5 Address:1c188701-f54a-d72f-8c33-efb56875b3c5} {Suffrage:Voter ID:server-72c2c775-6105-6649-87b2-62bdbb54ab42 Address:72c2c775-6105-6649-87b2-62bdbb54ab42} {Suffrage:Voter ID:server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 Address:5971c68c-2edf-c1f2-b958-95bf0e2008a0}]" + testing.go:149: server-72c2c775-6105-6649-87b2-62bdbb54ab42: 2021-08-25T09:31:15.063Z [INFO] server-72c2c775-6105-6649-87b2-62bdbb54ab42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c188701-f54a-d72f-8c33-efb56875b3c5 Address:1c188701-f54a-d72f-8c33-efb56875b3c5} {Suffrage:Voter ID:server-72c2c775-6105-6649-87b2-62bdbb54ab42 Address:72c2c775-6105-6649-87b2-62bdbb54ab42} {Suffrage:Voter ID:server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 Address:5971c68c-2edf-c1f2-b958-95bf0e2008a0}]" +testLoggerAdapter verbose: server-72c2c775-6105-6649-87b2-62bdbb54ab42: 2021-08-25T09:31:15.063Z [INFO] server-72c2c775-6105-6649-87b2-62bdbb54ab42: entering follower state: follower="Node at 72c2c775-6105-6649-87b2-62bdbb54ab42 [Follower]" leader= + testing.go:149: server-72c2c775-6105-6649-87b2-62bdbb54ab42: 2021-08-25T09:31:15.063Z [INFO] server-72c2c775-6105-6649-87b2-62bdbb54ab42: entering follower state: follower="Node at 72c2c775-6105-6649-87b2-62bdbb54ab42 [Follower]" leader= +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.063Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c188701-f54a-d72f-8c33-efb56875b3c5 Address:1c188701-f54a-d72f-8c33-efb56875b3c5} {Suffrage:Voter ID:server-72c2c775-6105-6649-87b2-62bdbb54ab42 Address:72c2c775-6105-6649-87b2-62bdbb54ab42} {Suffrage:Voter ID:server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 Address:5971c68c-2edf-c1f2-b958-95bf0e2008a0}]" + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.063Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c188701-f54a-d72f-8c33-efb56875b3c5 Address:1c188701-f54a-d72f-8c33-efb56875b3c5} {Suffrage:Voter ID:server-72c2c775-6105-6649-87b2-62bdbb54ab42 Address:72c2c775-6105-6649-87b2-62bdbb54ab42} {Suffrage:Voter ID:server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 Address:5971c68c-2edf-c1f2-b958-95bf0e2008a0}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.063Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:15.063Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.063Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering follower state: follower="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Follower]" leader= + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.063Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering follower state: follower="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Follower]" leader= +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [WARN] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [WARN] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering candidate state: node="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Candidate]" term=2 + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering candidate state: node="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Candidate]" term=2 +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: election won: tally=2 + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: election won: tally=2 +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering leader state: leader="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Leader]" + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering leader state: leader="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Leader]" +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: added peer, starting replication: peer=server-1c188701-f54a-d72f-8c33-efb56875b3c5 + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: added peer, starting replication: peer=server-1c188701-f54a-d72f-8c33-efb56875b3c5 +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: added peer, starting replication: peer=server-72c2c775-6105-6649-87b2-62bdbb54ab42 + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.135Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: added peer, starting replication: peer=server-72c2c775-6105-6649-87b2-62bdbb54ab42 +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.136Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: pipelining replication: peer="{Voter server-1c188701-f54a-d72f-8c33-efb56875b3c5 1c188701-f54a-d72f-8c33-efb56875b3c5}" + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.136Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: pipelining replication: peer="{Voter server-1c188701-f54a-d72f-8c33-efb56875b3c5 1c188701-f54a-d72f-8c33-efb56875b3c5}" +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.136Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: pipelining replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.136Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: pipelining replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.241Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.135962816 +0000 UTC m=+19.402179220 (1 nodes), 72.045799ms from start of poll, 73.276283ms from cluster start. Timeout at 2021-08-25 09:31:15.241088422 +0000 UTC m=+19.507304866, 105.125646ms after stability + testing.go:149: cluster: 2021-08-25T09:31:15.241Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.135962816 +0000 UTC m=+19.402179220 (1 nodes), 72.045799ms from start of poll, 73.276283ms from cluster start. Timeout at 2021-08-25 09:31:15.241088422 +0000 UTC m=+19.507304866, 105.125646ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.241Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:15.241Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.346Z [INFO] stable state for Follower reached at 2021-08-25 09:31:15.241389694 +0000 UTC m=+19.507606078 (2 nodes), 3.66µs from start of poll, 178.703141ms from cluster start. Timeout at 2021-08-25 09:31:15.346504461 +0000 UTC m=+19.612720865, 105.114787ms after stability + testing.go:149: cluster: 2021-08-25T09:31:15.346Z [INFO] stable state for Follower reached at 2021-08-25 09:31:15.241389694 +0000 UTC m=+19.507606078 (2 nodes), 3.66µs from start of poll, 178.703141ms from cluster start. Timeout at 2021-08-25 09:31:15.346504461 +0000 UTC m=+19.612720865, 105.114787ms after stability +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: entering candidate state: node="Node at 1c188701-f54a-d72f-8c33-efb56875b3c5 [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.380Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: entering candidate state: node="Node at 1c188701-f54a-d72f-8c33-efb56875b3c5 [Candidate]" term=3 + testing.go:149: cluster: 2021-08-25T09:31:15.380Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.380Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering follower state: follower="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Follower]" leader= +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: election won: tally=2 + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.380Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: entering follower state: follower="Node at 5971c68c-2edf-c1f2-b958-95bf0e2008a0 [Follower]" leader= + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: election won: tally=2 +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.380Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: aborting pipeline replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: entering leader state: leader="Node at 1c188701-f54a-d72f-8c33-efb56875b3c5 [Leader]" + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.380Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: aborting pipeline replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" +testLoggerAdapter verbose: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.380Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: aborting pipeline replication: peer="{Voter server-1c188701-f54a-d72f-8c33-efb56875b3c5 1c188701-f54a-d72f-8c33-efb56875b3c5}" + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: entering leader state: leader="Node at 1c188701-f54a-d72f-8c33-efb56875b3c5 [Leader]" +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: added peer, starting replication: peer=server-72c2c775-6105-6649-87b2-62bdbb54ab42 + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: added peer, starting replication: peer=server-72c2c775-6105-6649-87b2-62bdbb54ab42 +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: added peer, starting replication: peer=server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: added peer, starting replication: peer=server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: pipelining replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" + testing.go:149: server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: 2021-08-25T09:31:15.380Z [INFO] server-5971c68c-2edf-c1f2-b958-95bf0e2008a0: aborting pipeline replication: peer="{Voter server-1c188701-f54a-d72f-8c33-efb56875b3c5 1c188701-f54a-d72f-8c33-efb56875b3c5}" + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: pipelining replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: pipelining replication: peer="{Voter server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 5971c68c-2edf-c1f2-b958-95bf0e2008a0}" + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.380Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: pipelining replication: peer="{Voter server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 5971c68c-2edf-c1f2-b958-95bf0e2008a0}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.485Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.380616791 +0000 UTC m=+19.646833195 (1 nodes), 120.669µs from start of poll, 317.930258ms from cluster start. Timeout at 2021-08-25 09:31:15.485726138 +0000 UTC m=+19.751942542, 105.109347ms after stability + testing.go:149: cluster: 2021-08-25T09:31:15.485Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.380616791 +0000 UTC m=+19.646833195 (1 nodes), 120.669µs from start of poll, 317.930258ms from cluster start. Timeout at 2021-08-25 09:31:15.485726138 +0000 UTC m=+19.751942542, 105.109347ms after stability +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.486Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: aborting pipeline replication: peer="{Voter server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 5971c68c-2edf-c1f2-b958-95bf0e2008a0}" + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.486Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: aborting pipeline replication: peer="{Voter server-5971c68c-2edf-c1f2-b958-95bf0e2008a0 5971c68c-2edf-c1f2-b958-95bf0e2008a0}" +testLoggerAdapter verbose: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.486Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: aborting pipeline replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" + testing.go:149: server-1c188701-f54a-d72f-8c33-efb56875b3c5: 2021-08-25T09:31:15.486Z [INFO] server-1c188701-f54a-d72f-8c33-efb56875b3c5: aborting pipeline replication: peer="{Voter server-72c2c775-6105-6649-87b2-62bdbb54ab42 72c2c775-6105-6649-87b2-62bdbb54ab42}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.43s) === RUN TestRaft_LeadershipTransferToItself -testLoggerAdapter verbose: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.688Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" - testing.go:149: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.688Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" -testLoggerAdapter verbose: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.689Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: entering follower state: follower="Node at 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab [Follower]" leader= - testing.go:149: server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: 2021-08-25T09:28:19.689Z [INFO] server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab: entering follower state: follower="Node at 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab [Follower]" leader= -testLoggerAdapter verbose: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" - testing.go:149: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" -testLoggerAdapter verbose: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: entering follower state: follower="Node at 532e826a-77e4-388a-9282-45151c6bf272 [Follower]" leader= - testing.go:149: server-532e826a-77e4-388a-9282-45151c6bf272: 2021-08-25T09:28:19.689Z [INFO] server-532e826a-77e4-388a-9282-45151c6bf272: entering follower state: follower="Node at 532e826a-77e4-388a-9282-45151c6bf272 [Follower]" leader= -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab Address:972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab} {Suffrage:Voter ID:server-532e826a-77e4-388a-9282-45151c6bf272 Address:532e826a-77e4-388a-9282-45151c6bf272} {Suffrage:Voter ID:server-922da962-4151-f9f1-4842-3cc8cada4fa5 Address:922da962-4151-f9f1-4842-3cc8cada4fa5}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.689Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:19.689Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering follower state: follower="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Follower]" leader= - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.689Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering follower state: follower="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Follower]" leader= -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [WARN] server-922da962-4151-f9f1-4842-3cc8cada4fa5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [WARN] server-922da962-4151-f9f1-4842-3cc8cada4fa5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering candidate state: node="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Candidate]" term=2 - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering candidate state: node="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Candidate]" term=2 -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: election won: tally=2 - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: election won: tally=2 -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering leader state: leader="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Leader]" - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: entering leader state: leader="Node at 922da962-4151-f9f1-4842-3cc8cada4fa5 [Leader]" -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-532e826a-77e4-388a-9282-45151c6bf272 - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: added peer, starting replication: peer=server-532e826a-77e4-388a-9282-45151c6bf272 -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.740Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: pipelining replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.846Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.740711584 +0000 UTC m=+19.504927473 (1 nodes), 50.714907ms from start of poll, 52.172471ms from cluster start. Timeout at 2021-08-25 09:28:19.84587445 +0000 UTC m=+19.610090419, 105.162946ms after stability - testing.go:149: cluster: 2021-08-25T09:28:19.846Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.740711584 +0000 UTC m=+19.504927473 (1 nodes), 50.714907ms from start of poll, 52.172471ms from cluster start. Timeout at 2021-08-25 09:28:19.84587445 +0000 UTC m=+19.610090419, 105.162946ms after stability -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: cannot transfer leadership to itself - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: cannot transfer leadership to itself -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-532e826a-77e4-388a-9282-45151c6bf272 532e826a-77e4-388a-9282-45151c6bf272}" -testLoggerAdapter verbose: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" - testing.go:149: server-922da962-4151-f9f1-4842-3cc8cada4fa5: 2021-08-25T09:28:19.846Z [INFO] server-922da962-4151-f9f1-4842-3cc8cada4fa5: aborting pipeline replication: peer="{Voter server-972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab 972c3ad3-5f5d-eb42-c4a1-3bc5e05d0eab}" +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.490Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af8e90da-a68e-7707-3f85-a7573c1bc576 Address:af8e90da-a68e-7707-3f85-a7573c1bc576} {Suffrage:Voter ID:server-556cd876-19b4-7074-ebf5-ea51065c65b3 Address:556cd876-19b4-7074-ebf5-ea51065c65b3} {Suffrage:Voter ID:server-f28d238d-b992-d629-c208-c9c9b4cc3218 Address:f28d238d-b992-d629-c208-c9c9b4cc3218}]" + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.490Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af8e90da-a68e-7707-3f85-a7573c1bc576 Address:af8e90da-a68e-7707-3f85-a7573c1bc576} {Suffrage:Voter ID:server-556cd876-19b4-7074-ebf5-ea51065c65b3 Address:556cd876-19b4-7074-ebf5-ea51065c65b3} {Suffrage:Voter ID:server-f28d238d-b992-d629-c208-c9c9b4cc3218 Address:f28d238d-b992-d629-c208-c9c9b4cc3218}]" +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.490Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: entering follower state: follower="Node at af8e90da-a68e-7707-3f85-a7573c1bc576 [Follower]" leader= + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.490Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: entering follower state: follower="Node at af8e90da-a68e-7707-3f85-a7573c1bc576 [Follower]" leader= +testLoggerAdapter verbose: server-556cd876-19b4-7074-ebf5-ea51065c65b3: 2021-08-25T09:31:15.490Z [INFO] server-556cd876-19b4-7074-ebf5-ea51065c65b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af8e90da-a68e-7707-3f85-a7573c1bc576 Address:af8e90da-a68e-7707-3f85-a7573c1bc576} {Suffrage:Voter ID:server-556cd876-19b4-7074-ebf5-ea51065c65b3 Address:556cd876-19b4-7074-ebf5-ea51065c65b3} {Suffrage:Voter ID:server-f28d238d-b992-d629-c208-c9c9b4cc3218 Address:f28d238d-b992-d629-c208-c9c9b4cc3218}]" + testing.go:149: server-556cd876-19b4-7074-ebf5-ea51065c65b3: 2021-08-25T09:31:15.490Z [INFO] server-556cd876-19b4-7074-ebf5-ea51065c65b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af8e90da-a68e-7707-3f85-a7573c1bc576 Address:af8e90da-a68e-7707-3f85-a7573c1bc576} {Suffrage:Voter ID:server-556cd876-19b4-7074-ebf5-ea51065c65b3 Address:556cd876-19b4-7074-ebf5-ea51065c65b3} {Suffrage:Voter ID:server-f28d238d-b992-d629-c208-c9c9b4cc3218 Address:f28d238d-b992-d629-c208-c9c9b4cc3218}]" +testLoggerAdapter verbose: server-556cd876-19b4-7074-ebf5-ea51065c65b3: 2021-08-25T09:31:15.490Z [INFO] server-556cd876-19b4-7074-ebf5-ea51065c65b3: entering follower state: follower="Node at 556cd876-19b4-7074-ebf5-ea51065c65b3 [Follower]" leader= + testing.go:149: server-556cd876-19b4-7074-ebf5-ea51065c65b3: 2021-08-25T09:31:15.490Z [INFO] server-556cd876-19b4-7074-ebf5-ea51065c65b3: entering follower state: follower="Node at 556cd876-19b4-7074-ebf5-ea51065c65b3 [Follower]" leader= +testLoggerAdapter verbose: server-f28d238d-b992-d629-c208-c9c9b4cc3218: 2021-08-25T09:31:15.491Z [INFO] server-f28d238d-b992-d629-c208-c9c9b4cc3218: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af8e90da-a68e-7707-3f85-a7573c1bc576 Address:af8e90da-a68e-7707-3f85-a7573c1bc576} {Suffrage:Voter ID:server-556cd876-19b4-7074-ebf5-ea51065c65b3 Address:556cd876-19b4-7074-ebf5-ea51065c65b3} {Suffrage:Voter ID:server-f28d238d-b992-d629-c208-c9c9b4cc3218 Address:f28d238d-b992-d629-c208-c9c9b4cc3218}]" + testing.go:149: server-f28d238d-b992-d629-c208-c9c9b4cc3218: 2021-08-25T09:31:15.491Z [INFO] server-f28d238d-b992-d629-c208-c9c9b4cc3218: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af8e90da-a68e-7707-3f85-a7573c1bc576 Address:af8e90da-a68e-7707-3f85-a7573c1bc576} {Suffrage:Voter ID:server-556cd876-19b4-7074-ebf5-ea51065c65b3 Address:556cd876-19b4-7074-ebf5-ea51065c65b3} {Suffrage:Voter ID:server-f28d238d-b992-d629-c208-c9c9b4cc3218 Address:f28d238d-b992-d629-c208-c9c9b4cc3218}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.491Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:15.491Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f28d238d-b992-d629-c208-c9c9b4cc3218: 2021-08-25T09:31:15.491Z [INFO] server-f28d238d-b992-d629-c208-c9c9b4cc3218: entering follower state: follower="Node at f28d238d-b992-d629-c208-c9c9b4cc3218 [Follower]" leader= + testing.go:149: server-f28d238d-b992-d629-c208-c9c9b4cc3218: 2021-08-25T09:31:15.491Z [INFO] server-f28d238d-b992-d629-c208-c9c9b4cc3218: entering follower state: follower="Node at f28d238d-b992-d629-c208-c9c9b4cc3218 [Follower]" leader= +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.544Z [WARN] server-af8e90da-a68e-7707-3f85-a7573c1bc576: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.544Z [WARN] server-af8e90da-a68e-7707-3f85-a7573c1bc576: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.544Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: entering candidate state: node="Node at af8e90da-a68e-7707-3f85-a7573c1bc576 [Candidate]" term=2 + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.544Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: entering candidate state: node="Node at af8e90da-a68e-7707-3f85-a7573c1bc576 [Candidate]" term=2 +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.544Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: election won: tally=2 + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.544Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: election won: tally=2 +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: entering leader state: leader="Node at af8e90da-a68e-7707-3f85-a7573c1bc576 [Leader]" + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: entering leader state: leader="Node at af8e90da-a68e-7707-3f85-a7573c1bc576 [Leader]" +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: added peer, starting replication: peer=server-556cd876-19b4-7074-ebf5-ea51065c65b3 + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: added peer, starting replication: peer=server-556cd876-19b4-7074-ebf5-ea51065c65b3 +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: added peer, starting replication: peer=server-f28d238d-b992-d629-c208-c9c9b4cc3218 + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: added peer, starting replication: peer=server-f28d238d-b992-d629-c208-c9c9b4cc3218 +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: pipelining replication: peer="{Voter server-556cd876-19b4-7074-ebf5-ea51065c65b3 556cd876-19b4-7074-ebf5-ea51065c65b3}" + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: pipelining replication: peer="{Voter server-556cd876-19b4-7074-ebf5-ea51065c65b3 556cd876-19b4-7074-ebf5-ea51065c65b3}" +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: pipelining replication: peer="{Voter server-f28d238d-b992-d629-c208-c9c9b4cc3218 f28d238d-b992-d629-c208-c9c9b4cc3218}" + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.545Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: pipelining replication: peer="{Voter server-f28d238d-b992-d629-c208-c9c9b4cc3218 f28d238d-b992-d629-c208-c9c9b4cc3218}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.650Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.545090651 +0000 UTC m=+19.811307055 (1 nodes), 53.886468ms from start of poll, 55.14121ms from cluster start. Timeout at 2021-08-25 09:31:15.650212398 +0000 UTC m=+19.916428862, 105.121807ms after stability + testing.go:149: cluster: 2021-08-25T09:31:15.650Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.545090651 +0000 UTC m=+19.811307055 (1 nodes), 53.886468ms from start of poll, 55.14121ms from cluster start. Timeout at 2021-08-25 09:31:15.650212398 +0000 UTC m=+19.916428862, 105.121807ms after stability +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.650Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: cannot transfer leadership to itself + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.650Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: cannot transfer leadership to itself +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.650Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: aborting pipeline replication: peer="{Voter server-556cd876-19b4-7074-ebf5-ea51065c65b3 556cd876-19b4-7074-ebf5-ea51065c65b3}" + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.650Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: aborting pipeline replication: peer="{Voter server-556cd876-19b4-7074-ebf5-ea51065c65b3 556cd876-19b4-7074-ebf5-ea51065c65b3}" +testLoggerAdapter verbose: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.650Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: aborting pipeline replication: peer="{Voter server-f28d238d-b992-d629-c208-c9c9b4cc3218 f28d238d-b992-d629-c208-c9c9b4cc3218}" + testing.go:149: server-af8e90da-a68e-7707-3f85-a7573c1bc576: 2021-08-25T09:31:15.650Z [INFO] server-af8e90da-a68e-7707-3f85-a7573c1bc576: aborting pipeline replication: peer="{Voter server-f28d238d-b992-d629-c208-c9c9b4cc3218 f28d238d-b992-d629-c208-c9c9b4cc3218}" --- PASS: TestRaft_LeadershipTransferToItself (0.16s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout -testLoggerAdapter verbose: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" - testing.go:149: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" -testLoggerAdapter verbose: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: entering follower state: follower="Node at c2171324-e927-bb2b-3893-ea136cd96306 [Follower]" leader= - testing.go:149: server-c2171324-e927-bb2b-3893-ea136cd96306: 2021-08-25T09:28:19.851Z [INFO] server-c2171324-e927-bb2b-3893-ea136cd96306: entering follower state: follower="Node at c2171324-e927-bb2b-3893-ea136cd96306 [Follower]" leader= -testLoggerAdapter verbose: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.851Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" - testing.go:149: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.851Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.851Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.851Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2171324-e927-bb2b-3893-ea136cd96306 Address:c2171324-e927-bb2b-3893-ea136cd96306} {Suffrage:Voter ID:server-17720e6c-540b-3eb4-44e0-d74473b8ec95 Address:17720e6c-540b-3eb4-44e0-d74473b8ec95} {Suffrage:Voter ID:server-0b037b32-f225-08b7-e7e3-10aa57b38c0d Address:0b037b32-f225-08b7-e7e3-10aa57b38c0d}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:19.851Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:19.851Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.852Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: entering follower state: follower="Node at 17720e6c-540b-3eb4-44e0-d74473b8ec95 [Follower]" leader= - testing.go:149: server-17720e6c-540b-3eb4-44e0-d74473b8ec95: 2021-08-25T09:28:19.852Z [INFO] server-17720e6c-540b-3eb4-44e0-d74473b8ec95: entering follower state: follower="Node at 17720e6c-540b-3eb4-44e0-d74473b8ec95 [Follower]" leader= -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.852Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering follower state: follower="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Follower]" leader= - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.852Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering follower state: follower="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Follower]" leader= -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [WARN] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [WARN] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering candidate state: node="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Candidate]" term=2 - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering candidate state: node="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Candidate]" term=2 -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: election won: tally=2 - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: election won: tally=2 -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering leader state: leader="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Leader]" - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.921Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: entering leader state: leader="Node at 0b037b32-f225-08b7-e7e3-10aa57b38c0d [Leader]" -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-c2171324-e927-bb2b-3893-ea136cd96306 - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-c2171324-e927-bb2b-3893-ea136cd96306 -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-17720e6c-540b-3eb4-44e0-d74473b8ec95 - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: added peer, starting replication: peer=server-17720e6c-540b-3eb4-44e0-d74473b8ec95 -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:19.922Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: pipelining replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.027Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.922468052 +0000 UTC m=+19.686683941 (1 nodes), 70.37324ms from start of poll, 71.736453ms from cluster start. Timeout at 2021-08-25 09:28:20.027613719 +0000 UTC m=+19.791829688, 105.145747ms after stability - testing.go:149: cluster: 2021-08-25T09:28:20.027Z [INFO] stable state for Leader reached at 2021-08-25 09:28:19.922468052 +0000 UTC m=+19.686683941 (1 nodes), 70.37324ms from start of poll, 71.736453ms from cluster start. Timeout at 2021-08-25 09:28:20.027613719 +0000 UTC m=+19.791829688, 105.145747ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.027Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:20.027Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.133Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.027880535 +0000 UTC m=+19.792096424 (2 nodes), 2.54µs from start of poll, 177.148936ms from cluster start. Timeout at 2021-08-25 09:28:20.132998845 +0000 UTC m=+19.897214834, 105.11841ms after stability - testing.go:149: cluster: 2021-08-25T09:28:20.133Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.027880535 +0000 UTC m=+19.792096424 (2 nodes), 2.54µs from start of poll, 177.148936ms from cluster start. Timeout at 2021-08-25 09:28:20.132998845 +0000 UTC m=+19.897214834, 105.11841ms after stability -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-c2171324-e927-bb2b-3893-ea136cd96306 c2171324-e927-bb2b-3893-ea136cd96306}" -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.566Z [INFO] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: aborting pipeline replication: peer="{Voter server-17720e6c-540b-3eb4-44e0-d74473b8ec95 17720e6c-540b-3eb4-44e0-d74473b8ec95}" -testLoggerAdapter verbose: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.616Z [ERROR] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: failed to heartbeat to: peer=17720e6c-540b-3eb4-44e0-d74473b8ec95 error="command timed out" - testing.go:149: server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: 2021-08-25T09:28:20.616Z [ERROR] server-0b037b32-f225-08b7-e7e3-10aa57b38c0d: failed to heartbeat to: peer=17720e6c-540b-3eb4-44e0-d74473b8ec95 error="command timed out" ---- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.77s) +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.655Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d8f6168-91a1-620d-c754-079ce18bbab7 Address:1d8f6168-91a1-620d-c754-079ce18bbab7} {Suffrage:Voter ID:server-56be31ef-e709-41a1-e1de-04689d3bf5f7 Address:56be31ef-e709-41a1-e1de-04689d3bf5f7} {Suffrage:Voter ID:server-0af2c5a3-3705-949c-6491-81f422b3e4fa Address:0af2c5a3-3705-949c-6491-81f422b3e4fa}]" + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.655Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d8f6168-91a1-620d-c754-079ce18bbab7 Address:1d8f6168-91a1-620d-c754-079ce18bbab7} {Suffrage:Voter ID:server-56be31ef-e709-41a1-e1de-04689d3bf5f7 Address:56be31ef-e709-41a1-e1de-04689d3bf5f7} {Suffrage:Voter ID:server-0af2c5a3-3705-949c-6491-81f422b3e4fa Address:0af2c5a3-3705-949c-6491-81f422b3e4fa}]" +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.655Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: entering follower state: follower="Node at 1d8f6168-91a1-620d-c754-079ce18bbab7 [Follower]" leader= + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.655Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: entering follower state: follower="Node at 1d8f6168-91a1-620d-c754-079ce18bbab7 [Follower]" leader= +testLoggerAdapter verbose: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:15.656Z [INFO] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d8f6168-91a1-620d-c754-079ce18bbab7 Address:1d8f6168-91a1-620d-c754-079ce18bbab7} {Suffrage:Voter ID:server-56be31ef-e709-41a1-e1de-04689d3bf5f7 Address:56be31ef-e709-41a1-e1de-04689d3bf5f7} {Suffrage:Voter ID:server-0af2c5a3-3705-949c-6491-81f422b3e4fa Address:0af2c5a3-3705-949c-6491-81f422b3e4fa}]" + testing.go:149: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:15.656Z [INFO] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d8f6168-91a1-620d-c754-079ce18bbab7 Address:1d8f6168-91a1-620d-c754-079ce18bbab7} {Suffrage:Voter ID:server-56be31ef-e709-41a1-e1de-04689d3bf5f7 Address:56be31ef-e709-41a1-e1de-04689d3bf5f7} {Suffrage:Voter ID:server-0af2c5a3-3705-949c-6491-81f422b3e4fa Address:0af2c5a3-3705-949c-6491-81f422b3e4fa}]" +testLoggerAdapter verbose: server-0af2c5a3-3705-949c-6491-81f422b3e4fa: 2021-08-25T09:31:15.657Z [INFO] server-0af2c5a3-3705-949c-6491-81f422b3e4fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d8f6168-91a1-620d-c754-079ce18bbab7 Address:1d8f6168-91a1-620d-c754-079ce18bbab7} {Suffrage:Voter ID:server-56be31ef-e709-41a1-e1de-04689d3bf5f7 Address:56be31ef-e709-41a1-e1de-04689d3bf5f7} {Suffrage:Voter ID:server-0af2c5a3-3705-949c-6491-81f422b3e4fa Address:0af2c5a3-3705-949c-6491-81f422b3e4fa}]" + testing.go:149: server-0af2c5a3-3705-949c-6491-81f422b3e4fa: 2021-08-25T09:31:15.657Z [INFO] server-0af2c5a3-3705-949c-6491-81f422b3e4fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d8f6168-91a1-620d-c754-079ce18bbab7 Address:1d8f6168-91a1-620d-c754-079ce18bbab7} {Suffrage:Voter ID:server-56be31ef-e709-41a1-e1de-04689d3bf5f7 Address:56be31ef-e709-41a1-e1de-04689d3bf5f7} {Suffrage:Voter ID:server-0af2c5a3-3705-949c-6491-81f422b3e4fa Address:0af2c5a3-3705-949c-6491-81f422b3e4fa}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.657Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:15.657Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:15.657Z [INFO] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: entering follower state: follower="Node at 56be31ef-e709-41a1-e1de-04689d3bf5f7 [Follower]" leader= + testing.go:149: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:15.657Z [INFO] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: entering follower state: follower="Node at 56be31ef-e709-41a1-e1de-04689d3bf5f7 [Follower]" leader= +testLoggerAdapter verbose: server-0af2c5a3-3705-949c-6491-81f422b3e4fa: 2021-08-25T09:31:15.657Z [INFO] server-0af2c5a3-3705-949c-6491-81f422b3e4fa: entering follower state: follower="Node at 0af2c5a3-3705-949c-6491-81f422b3e4fa [Follower]" leader= + testing.go:149: server-0af2c5a3-3705-949c-6491-81f422b3e4fa: 2021-08-25T09:31:15.657Z [INFO] server-0af2c5a3-3705-949c-6491-81f422b3e4fa: entering follower state: follower="Node at 0af2c5a3-3705-949c-6491-81f422b3e4fa [Follower]" leader= +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.725Z [WARN] server-1d8f6168-91a1-620d-c754-079ce18bbab7: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.725Z [WARN] server-1d8f6168-91a1-620d-c754-079ce18bbab7: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: entering candidate state: node="Node at 1d8f6168-91a1-620d-c754-079ce18bbab7 [Candidate]" term=2 + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: entering candidate state: node="Node at 1d8f6168-91a1-620d-c754-079ce18bbab7 [Candidate]" term=2 +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: election won: tally=2 + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: election won: tally=2 +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: entering leader state: leader="Node at 1d8f6168-91a1-620d-c754-079ce18bbab7 [Leader]" + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: entering leader state: leader="Node at 1d8f6168-91a1-620d-c754-079ce18bbab7 [Leader]" +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: added peer, starting replication: peer=server-56be31ef-e709-41a1-e1de-04689d3bf5f7 + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: added peer, starting replication: peer=server-56be31ef-e709-41a1-e1de-04689d3bf5f7 +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: added peer, starting replication: peer=server-0af2c5a3-3705-949c-6491-81f422b3e4fa + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: added peer, starting replication: peer=server-0af2c5a3-3705-949c-6491-81f422b3e4fa +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: pipelining replication: peer="{Voter server-56be31ef-e709-41a1-e1de-04689d3bf5f7 56be31ef-e709-41a1-e1de-04689d3bf5f7}" + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: pipelining replication: peer="{Voter server-56be31ef-e709-41a1-e1de-04689d3bf5f7 56be31ef-e709-41a1-e1de-04689d3bf5f7}" +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: pipelining replication: peer="{Voter server-0af2c5a3-3705-949c-6491-81f422b3e4fa 0af2c5a3-3705-949c-6491-81f422b3e4fa}" + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:15.726Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: pipelining replication: peer="{Voter server-0af2c5a3-3705-949c-6491-81f422b3e4fa 0af2c5a3-3705-949c-6491-81f422b3e4fa}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.831Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.726345972 +0000 UTC m=+19.992562416 (1 nodes), 69.018724ms from start of poll, 71.782884ms from cluster start. Timeout at 2021-08-25 09:31:15.831477678 +0000 UTC m=+20.097694122, 105.131706ms after stability + testing.go:149: cluster: 2021-08-25T09:31:15.831Z [INFO] stable state for Leader reached at 2021-08-25 09:31:15.726345972 +0000 UTC m=+19.992562416 (1 nodes), 69.018724ms from start of poll, 71.782884ms from cluster start. Timeout at 2021-08-25 09:31:15.831477678 +0000 UTC m=+20.097694122, 105.131706ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.831Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:15.831Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:15.937Z [INFO] stable state for Follower reached at 2021-08-25 09:31:15.831805167 +0000 UTC m=+20.098021571 (2 nodes), 2.7µs from start of poll, 177.242039ms from cluster start. Timeout at 2021-08-25 09:31:15.936943212 +0000 UTC m=+20.203160256, 105.138685ms after stability + testing.go:149: cluster: 2021-08-25T09:31:15.937Z [INFO] stable state for Follower reached at 2021-08-25 09:31:15.831805167 +0000 UTC m=+20.098021571 (2 nodes), 2.7µs from start of poll, 177.242039ms from cluster start. Timeout at 2021-08-25 09:31:15.936943212 +0000 UTC m=+20.203160256, 105.138685ms after stability +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:16.322Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: aborting pipeline replication: peer="{Voter server-0af2c5a3-3705-949c-6491-81f422b3e4fa 0af2c5a3-3705-949c-6491-81f422b3e4fa}" + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:16.322Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: aborting pipeline replication: peer="{Voter server-0af2c5a3-3705-949c-6491-81f422b3e4fa 0af2c5a3-3705-949c-6491-81f422b3e4fa}" +testLoggerAdapter verbose: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:16.322Z [INFO] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: entering candidate state: node="Node at 56be31ef-e709-41a1-e1de-04689d3bf5f7 [Shutdown]" term=3 + testing.go:149: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:16.322Z [INFO] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: entering candidate state: node="Node at 56be31ef-e709-41a1-e1de-04689d3bf5f7 [Shutdown]" term=3 +testLoggerAdapter verbose: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:16.322Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: aborting pipeline replication: peer="{Voter server-56be31ef-e709-41a1-e1de-04689d3bf5f7 56be31ef-e709-41a1-e1de-04689d3bf5f7}" + testing.go:149: server-1d8f6168-91a1-620d-c754-079ce18bbab7: 2021-08-25T09:31:16.322Z [INFO] server-1d8f6168-91a1-620d-c754-079ce18bbab7: aborting pipeline replication: peer="{Voter server-56be31ef-e709-41a1-e1de-04689d3bf5f7 56be31ef-e709-41a1-e1de-04689d3bf5f7}" +testLoggerAdapter verbose: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:16.372Z [ERROR] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: failed to make requestVote RPC: target="{Voter server-1d8f6168-91a1-620d-c754-079ce18bbab7 1d8f6168-91a1-620d-c754-079ce18bbab7}" error="command timed out" + testing.go:149: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:16.372Z [ERROR] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: failed to make requestVote RPC: target="{Voter server-1d8f6168-91a1-620d-c754-079ce18bbab7 1d8f6168-91a1-620d-c754-079ce18bbab7}" error="command timed out" +testLoggerAdapter verbose: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:16.372Z [ERROR] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: failed to make requestVote RPC: target="{Voter server-0af2c5a3-3705-949c-6491-81f422b3e4fa 0af2c5a3-3705-949c-6491-81f422b3e4fa}" error="command timed out" + testing.go:149: server-56be31ef-e709-41a1-e1de-04689d3bf5f7: 2021-08-25T09:31:16.372Z [ERROR] server-56be31ef-e709-41a1-e1de-04689d3bf5f7: failed to make requestVote RPC: target="{Voter server-0af2c5a3-3705-949c-6491-81f422b3e4fa 0af2c5a3-3705-949c-6491-81f422b3e4fa}" error="command timed out" +--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.72s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.620Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.620Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.621Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering follower state: follower="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Follower]" leader= - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.621Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering follower state: follower="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Follower]" leader= -testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" - testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.621Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-08-25T09:28:20.621Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= - testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.621Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= -testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [WARN] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [WARN] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering candidate state: node="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Candidate]" term=2 - testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.698Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering candidate state: node="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Candidate]" term=2 -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.698Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.698Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=2 - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=2 -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: duplicate requestVote for same term: term=2 - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.699Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.699Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: duplicate requestVote for same term: term=2 - testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.699Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: Election timeout reached, restarting election - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [WARN] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: Election timeout reached, restarting election -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=3 - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering candidate state: node="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Candidate]" term=3 -testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.755Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= - testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.755Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: entering follower state: follower="Node at 27b8f23c-1f52-9be4-0647-d3a0e8dec358 [Follower]" leader= -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: election won: tally=2 - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: election won: tally=2 -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering leader state: leader="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Leader]" - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: entering leader state: leader="Node at 7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 [Leader]" -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: added peer, starting replication: peer=server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.755Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: added peer, starting replication: peer=server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.756Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: pipelining replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.756Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: pipelining replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.861Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.755932431 +0000 UTC m=+20.520148320 (1 nodes), 134.026444ms from start of poll, 135.295946ms from cluster start. Timeout at 2021-08-25 09:28:20.861053341 +0000 UTC m=+20.625269310, 105.12099ms after stability - testing.go:149: cluster: 2021-08-25T09:28:20.861Z [INFO] stable state for Follower reached at 2021-08-25 09:28:20.755932431 +0000 UTC m=+20.520148320 (1 nodes), 134.026444ms from start of poll, 135.295946ms from cluster start. Timeout at 2021-08-25 09:28:20.861053341 +0000 UTC m=+20.625269310, 105.12099ms after stability -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.861Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-08-25T09:28:20.861Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-08-25T09:28:20.966Z [INFO] stable state for Leader reached at 2021-08-25 09:28:20.86148872 +0000 UTC m=+20.625704609 (1 nodes), 2.239µs from start of poll, 240.852235ms from cluster start. Timeout at 2021-08-25 09:28:20.966610131 +0000 UTC m=+20.730826060, 105.121451ms after stability - testing.go:149: cluster: 2021-08-25T09:28:20.966Z [INFO] stable state for Leader reached at 2021-08-25 09:28:20.86148872 +0000 UTC m=+20.625704609 (1 nodes), 2.239µs from start of poll, 240.852235ms from cluster start. Timeout at 2021-08-25 09:28:20.966610131 +0000 UTC m=+20.730826060, 105.121451ms after stability -testLoggerAdapter verbose: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.977Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" - testing.go:149: server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: 2021-08-25T09:28:20.977Z [INFO] server-27b8f23c-1f52-9be4-0647-d3a0e8dec358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2 Address:7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2} {Suffrage:Voter ID:server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 Address:27b8f23c-1f52-9be4-0647-d3a0e8dec358}]" -testLoggerAdapter verbose: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.977Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: aborting pipeline replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" - testing.go:149: server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: 2021-08-25T09:28:20.977Z [INFO] server-7e35b2cb-8f87-e8c8-22fd-762d3d8c43e2: aborting pipeline replication: peer="{Voter server-27b8f23c-1f52-9be4-0647-d3a0e8dec358 27b8f23c-1f52-9be4-0647-d3a0e8dec358}" ---- PASS: TestRaft_GetConfigurationNoBootstrap (0.36s) +testLoggerAdapter verbose: server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: 2021-08-25T09:31:16.376Z [INFO] server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a Address:00e96bd7-3d38-a6b8-e404-23bc45ebed3a} {Suffrage:Voter ID:server-c3d9dd09-9515-e49d-50fe-328d27ee0d20 Address:c3d9dd09-9515-e49d-50fe-328d27ee0d20}]" + testing.go:149: server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: 2021-08-25T09:31:16.376Z [INFO] server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a Address:00e96bd7-3d38-a6b8-e404-23bc45ebed3a} {Suffrage:Voter ID:server-c3d9dd09-9515-e49d-50fe-328d27ee0d20 Address:c3d9dd09-9515-e49d-50fe-328d27ee0d20}]" +testLoggerAdapter verbose: server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: 2021-08-25T09:31:16.376Z [INFO] server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: entering follower state: follower="Node at 00e96bd7-3d38-a6b8-e404-23bc45ebed3a [Follower]" leader= + testing.go:149: server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: 2021-08-25T09:31:16.376Z [INFO] server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a: entering follower state: follower="Node at 00e96bd7-3d38-a6b8-e404-23bc45ebed3a [Follower]" leader= +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.376Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a Address:00e96bd7-3d38-a6b8-e404-23bc45ebed3a} {Suffrage:Voter ID:server-c3d9dd09-9515-e49d-50fe-328d27ee0d20 Address:c3d9dd09-9515-e49d-50fe-328d27ee0d20}]" + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.376Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a Address:00e96bd7-3d38-a6b8-e404-23bc45ebed3a} {Suffrage:Voter ID:server-c3d9dd09-9515-e49d-50fe-328d27ee0d20 Address:c3d9dd09-9515-e49d-50fe-328d27ee0d20}]" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:16.376Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2021-08-25T09:31:16.376Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.377Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: entering follower state: follower="Node at c3d9dd09-9515-e49d-50fe-328d27ee0d20 [Follower]" leader= + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.377Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: entering follower state: follower="Node at c3d9dd09-9515-e49d-50fe-328d27ee0d20 [Follower]" leader= +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.444Z [WARN] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.444Z [WARN] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.444Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: entering candidate state: node="Node at c3d9dd09-9515-e49d-50fe-328d27ee0d20 [Candidate]" term=2 + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.444Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: entering candidate state: node="Node at c3d9dd09-9515-e49d-50fe-328d27ee0d20 [Candidate]" term=2 +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: election won: tally=2 + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: election won: tally=2 +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: entering leader state: leader="Node at c3d9dd09-9515-e49d-50fe-328d27ee0d20 [Leader]" + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: entering leader state: leader="Node at c3d9dd09-9515-e49d-50fe-328d27ee0d20 [Leader]" +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: added peer, starting replication: peer=server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: added peer, starting replication: peer=server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: pipelining replication: peer="{Voter server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a 00e96bd7-3d38-a6b8-e404-23bc45ebed3a}" + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.445Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: pipelining replication: peer="{Voter server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a 00e96bd7-3d38-a6b8-e404-23bc45ebed3a}" +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:16.550Z [INFO] stable state for Follower reached at 2021-08-25 09:31:16.445417097 +0000 UTC m=+20.711633541 (1 nodes), 68.388783ms from start of poll, 69.233864ms from cluster start. Timeout at 2021-08-25 09:31:16.550554742 +0000 UTC m=+20.816771266, 105.137725ms after stability + testing.go:149: cluster: 2021-08-25T09:31:16.550Z [INFO] stable state for Follower reached at 2021-08-25 09:31:16.445417097 +0000 UTC m=+20.711633541 (1 nodes), 68.388783ms from start of poll, 69.233864ms from cluster start. Timeout at 2021-08-25 09:31:16.550554742 +0000 UTC m=+20.816771266, 105.137725ms after stability +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:16.550Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2021-08-25T09:31:16.550Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2021-08-25T09:31:16.656Z [INFO] stable state for Leader reached at 2021-08-25 09:31:16.550839295 +0000 UTC m=+20.817055679 (1 nodes), 2.139µs from start of poll, 174.656002ms from cluster start. Timeout at 2021-08-25 09:31:16.655969561 +0000 UTC m=+20.922186025, 105.130346ms after stability + testing.go:149: cluster: 2021-08-25T09:31:16.656Z [INFO] stable state for Leader reached at 2021-08-25 09:31:16.550839295 +0000 UTC m=+20.817055679 (1 nodes), 2.139µs from start of poll, 174.656002ms from cluster start. Timeout at 2021-08-25 09:31:16.655969561 +0000 UTC m=+20.922186025, 105.130346ms after stability +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.666Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a Address:00e96bd7-3d38-a6b8-e404-23bc45ebed3a} {Suffrage:Voter ID:server-c3d9dd09-9515-e49d-50fe-328d27ee0d20 Address:c3d9dd09-9515-e49d-50fe-328d27ee0d20}]" + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.666Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a Address:00e96bd7-3d38-a6b8-e404-23bc45ebed3a} {Suffrage:Voter ID:server-c3d9dd09-9515-e49d-50fe-328d27ee0d20 Address:c3d9dd09-9515-e49d-50fe-328d27ee0d20}]" +testLoggerAdapter verbose: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.667Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: aborting pipeline replication: peer="{Voter server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a 00e96bd7-3d38-a6b8-e404-23bc45ebed3a}" + testing.go:149: server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: 2021-08-25T09:31:16.667Z [INFO] server-c3d9dd09-9515-e49d-50fe-328d27ee0d20: aborting pipeline replication: peer="{Voter server-00e96bd7-3d38-a6b8-e404-23bc45ebed3a 00e96bd7-3d38-a6b8-e404-23bc45ebed3a}" +--- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise @@ -3913,11 +4046,11 @@ === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID ---- PASS: TestGenerateUUID (0.00s) +--- PASS: TestGenerateUUID (0.01s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS -ok github.com/hashicorp/raft 20.764s +ok github.com/hashicorp/raft 20.962s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary @@ -3949,12 +4082,14 @@ dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration +I: user script /srv/workspace/pbuilder/7057/tmp/hooks/B01_cleanup starting +I: user script /srv/workspace/pbuilder/7057/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/23540 and its subdirectories -I: Current time: Tue Aug 24 21:28:32 -12 2021 -I: pbuilder-time-stamp: 1629883712 +I: removing directory /srv/workspace/pbuilder/7057 and its subdirectories +I: Current time: Wed Aug 25 23:31:30 +14 2021 +I: pbuilder-time-stamp: 1629883890