I: pbuilder: network access will be disabled during build I: Current time: Tue May 16 04:03:58 +14 2023 I: pbuilder-time-stamp: 1684159438 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bookworm-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: Signature made Sun Jan 19 21:24:28 2020 +14 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.1.2-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D01_modify_environment starting debug: Running on codethink10-arm64. I: Changing host+domainname to test build reproducibility I: Adding a custom variable just for the fun of it... I: Changing /bin/sh to bash '/bin/sh' -> '/bin/bash' lrwxrwxrwx 1 root root 9 May 16 04:04 /bin/sh -> /bin/bash I: Setting pbuilder2's login shell to /bin/bash I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D01_modify_environment finished I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D02_print_environment starting I: set BASH=/bin/sh BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_LINENO=([0]="12" [1]="0") BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:. BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") BASH_VERSINFO=([0]="5" [1]="2" [2]="15" [3]="1" [4]="release" [5]="aarch64-unknown-linux-gnu") BASH_VERSION='5.2.15(1)-release' BUILDDIR=/build BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' BUILDUSERNAME=pbuilder2 BUILD_ARCH=arm64 DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=8' DIRSTACK=() DISTRIBUTION=bookworm EUID=0 FUNCNAME=([0]="Echo" [1]="main") GROUPS=() HOME=/var/lib/jenkins HOSTNAME=i-capture-the-hostname HOSTTYPE=aarch64 HOST_ARCH=arm64 IFS=' ' LANG=C LANGUAGE=nl_BE:nl LC_ALL=C MACHTYPE=aarch64-unknown-linux-gnu MAIL=/var/mail/root OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path PBCURRENTCOMMANDLINEOPERATION=build PBUILDER_OPERATION=build PBUILDER_PKGDATADIR=/usr/share/pbuilder PBUILDER_PKGLIBDIR=/usr/lib/pbuilder PBUILDER_SYSCONFDIR=/etc PIPESTATUS=([0]="0") POSIXLY_CORRECT=y PPID=12871 PS4='+ ' PWD=/ SHELL=/bin/bash SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix SHLVL=3 SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/pbuilderrc_o6zw --distribution bookworm --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/b2 --logfile b2/build.log --extrapackages usrmerge golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID=117 SUDO_UID=110 SUDO_USER=jenkins TERM=unknown TZ=/usr/share/zoneinfo/Etc/GMT-14 UID=0 USER=root USERNAME=root _='I: set' http_proxy=http://192.168.101.16:3128 I: uname -a Linux i-capture-the-hostname 4.15.0-210-generic #221-Ubuntu SMP Tue Apr 18 08:32:48 UTC 2023 aarch64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 May 15 00:25 /bin -> usr/bin I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: arm64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19616 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.19-go{a} golang-1.19-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-3-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-diff-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 98 newly installed, 0 to remove and 0 not upgraded. Need to get 112 MB of archives. After unpacking 785 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bookworm/main arm64 sensible-utils all 0.0.17+nmu1 [19.0 kB] Get: 2 http://deb.debian.org/debian bookworm/main arm64 libmagic-mgc arm64 1:5.44-3 [305 kB] Get: 3 http://deb.debian.org/debian bookworm/main arm64 libmagic1 arm64 1:5.44-3 [98.5 kB] Get: 4 http://deb.debian.org/debian bookworm/main arm64 file arm64 1:5.44-3 [42.5 kB] Get: 5 http://deb.debian.org/debian bookworm/main arm64 gettext-base arm64 0.21-12 [159 kB] Get: 6 http://deb.debian.org/debian bookworm/main arm64 libuchardet0 arm64 0.0.7-1 [67.9 kB] Get: 7 http://deb.debian.org/debian bookworm/main arm64 groff-base arm64 1.22.4-10 [861 kB] Get: 8 http://deb.debian.org/debian bookworm/main arm64 bsdextrautils arm64 2.38.1-5+b1 [86.9 kB] Get: 9 http://deb.debian.org/debian bookworm/main arm64 libpipeline1 arm64 1.5.7-1 [36.4 kB] Get: 10 http://deb.debian.org/debian bookworm/main arm64 man-db arm64 2.11.2-2 [1369 kB] Get: 11 http://deb.debian.org/debian bookworm/main arm64 m4 arm64 1.4.19-3 [276 kB] Get: 12 http://deb.debian.org/debian bookworm/main arm64 autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian bookworm/main arm64 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian bookworm/main arm64 automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian bookworm/main arm64 autopoint all 0.21-12 [495 kB] Get: 16 http://deb.debian.org/debian bookworm/main arm64 libdebhelper-perl all 13.11.4 [81.2 kB] Get: 17 http://deb.debian.org/debian bookworm/main arm64 libtool all 2.4.7-5 [517 kB] Get: 18 http://deb.debian.org/debian bookworm/main arm64 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian bookworm/main arm64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian bookworm/main arm64 libsub-override-perl all 0.09-4 [9304 B] Get: 21 http://deb.debian.org/debian bookworm/main arm64 libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB] Get: 22 http://deb.debian.org/debian bookworm/main arm64 dh-strip-nondeterminism all 1.13.1-1 [8620 B] Get: 23 http://deb.debian.org/debian bookworm/main arm64 libelf1 arm64 0.188-2.1 [173 kB] Get: 24 http://deb.debian.org/debian bookworm/main arm64 dwz arm64 0.15-1 [101 kB] Get: 25 http://deb.debian.org/debian bookworm/main arm64 libicu72 arm64 72.1-3 [9204 kB] Get: 26 http://deb.debian.org/debian bookworm/main arm64 libxml2 arm64 2.9.14+dfsg-1.2 [620 kB] Get: 27 http://deb.debian.org/debian bookworm/main arm64 gettext arm64 0.21-12 [1248 kB] Get: 28 http://deb.debian.org/debian bookworm/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian bookworm/main arm64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian bookworm/main arm64 debhelper all 13.11.4 [942 kB] Get: 31 http://deb.debian.org/debian bookworm/main arm64 dh-golang all 1.59 [27.8 kB] Get: 32 http://deb.debian.org/debian bookworm/main arm64 golang-1.19-src all 1.19.8-2 [18.3 MB] Get: 33 http://deb.debian.org/debian bookworm/main arm64 golang-1.19-go arm64 1.19.8-2 [56.6 MB] Get: 34 http://deb.debian.org/debian bookworm/main arm64 golang-src all 2:1.19~1 [5756 B] Get: 35 http://deb.debian.org/debian bookworm/main arm64 golang-go arm64 2:1.19~1 [45.0 kB] Get: 36 http://deb.debian.org/debian bookworm/main arm64 golang-any arm64 2:1.19~1 [5876 B] Get: 37 http://deb.debian.org/debian bookworm/main arm64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 38 http://deb.debian.org/debian bookworm/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-sys-dev all 0.3.0-1 [393 kB] Get: 41 http://deb.debian.org/debian bookworm/main arm64 golang-github-mattn-go-isatty-dev all 0.0.17-1 [6016 B] Get: 42 http://deb.debian.org/debian bookworm/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian bookworm/main arm64 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 44 http://deb.debian.org/debian bookworm/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian bookworm/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian bookworm/main arm64 golang-github-stretchr-objx-dev all 0.5.0-1 [25.5 kB] Get: 47 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian bookworm/main arm64 golang-github-stretchr-testify-dev all 1.8.1-1 [67.9 kB] Get: 49 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian bookworm/main arm64 golang-github-pkg-errors-dev all 0.9.1-2 [13.2 kB] Get: 52 http://deb.debian.org/debian bookworm/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian bookworm/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian bookworm/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.2-2 [8480 B] Get: 56 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 57 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 58 http://deb.debian.org/debian bookworm/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 59 http://deb.debian.org/debian bookworm/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 60 http://deb.debian.org/debian bookworm/main arm64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 61 http://deb.debian.org/debian bookworm/main arm64 golang-github-golang-protobuf-1-3-dev all 1.3.5-4 [234 kB] Get: 62 http://deb.debian.org/debian bookworm/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 63 http://deb.debian.org/debian bookworm/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 64 http://deb.debian.org/debian bookworm/main arm64 golang-github-json-iterator-go-dev all 1.1.12-1 [62.2 kB] Get: 65 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-client-model-dev all 0.3.0-3 [10.5 kB] Get: 66 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 67 http://deb.debian.org/debian bookworm/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 68 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-text-dev all 0.7.0-1 [3863 kB] Get: 69 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-net-dev all 1:0.7.0+dfsg-1 [731 kB] Get: 70 http://deb.debian.org/debian bookworm/main arm64 golang-github-aws-aws-sdk-go-dev all 1.44.133-1 [10.5 MB] Get: 71 http://deb.debian.org/debian bookworm/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 72 http://deb.debian.org/debian bookworm/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 73 http://deb.debian.org/debian bookworm/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 74 http://deb.debian.org/debian bookworm/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 75 http://deb.debian.org/debian bookworm/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 76 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-oauth2-dev all 0.3.0-1 [35.0 kB] Get: 77 http://deb.debian.org/debian bookworm/main arm64 golang-google-protobuf-dev all 1.28.1-3 [542 kB] Get: 78 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 79 http://deb.debian.org/debian bookworm/main arm64 golang-protobuf-extensions-dev all 1.0.1-3 [29.5 kB] Get: 80 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-common-dev all 0.39.0-2 [110 kB] Get: 81 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-sync-dev all 0.1.0-1 [19.7 kB] Get: 82 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-procfs-dev all 0.8.0-3 [148 kB] Get: 83 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-client-golang-dev all 1.14.0-3 [174 kB] Get: 84 http://deb.debian.org/debian bookworm/main arm64 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 85 http://deb.debian.org/debian bookworm/main arm64 golang-github-creack-pty-dev all 1.1.18-1 [9796 B] Get: 86 http://deb.debian.org/debian bookworm/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 87 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 88 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 89 http://deb.debian.org/debian bookworm/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg-10 [20.8 kB] Get: 90 http://deb.debian.org/debian bookworm/main arm64 libsasl2-2 arm64 2.1.28+dfsg-10 [58.0 kB] Get: 91 http://deb.debian.org/debian bookworm/main arm64 libsasl2-dev arm64 2.1.28+dfsg-10 [230 kB] Get: 92 http://deb.debian.org/debian bookworm/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 93 http://deb.debian.org/debian bookworm/main arm64 golang-github-pkg-diff-dev all 0.0~git20210226.20ebb0f-1 [93.6 kB] Get: 94 http://deb.debian.org/debian bookworm/main arm64 golang-github-rogpeppe-go-internal-dev all 1.9.0-1 [117 kB] Get: 95 http://deb.debian.org/debian bookworm/main arm64 golang-github-kr-pretty-dev all 0.3.0-1 [14.9 kB] Get: 96 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 97 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 98 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 112 MB in 2s (49.8 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19616 files and directories currently installed.) Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ... Unpacking sensible-utils (0.0.17+nmu1) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../01-libmagic-mgc_1%3a5.44-3_arm64.deb ... Unpacking libmagic-mgc (1:5.44-3) ... Selecting previously unselected package libmagic1:arm64. Preparing to unpack .../02-libmagic1_1%3a5.44-3_arm64.deb ... Unpacking libmagic1:arm64 (1:5.44-3) ... Selecting previously unselected package file. Preparing to unpack .../03-file_1%3a5.44-3_arm64.deb ... Unpacking file (1:5.44-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../04-gettext-base_0.21-12_arm64.deb ... Unpacking gettext-base (0.21-12) ... Selecting previously unselected package libuchardet0:arm64. Preparing to unpack .../05-libuchardet0_0.0.7-1_arm64.deb ... Unpacking libuchardet0:arm64 (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../06-groff-base_1.22.4-10_arm64.deb ... Unpacking groff-base (1.22.4-10) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../07-bsdextrautils_2.38.1-5+b1_arm64.deb ... Unpacking bsdextrautils (2.38.1-5+b1) ... Selecting previously unselected package libpipeline1:arm64. Preparing to unpack .../08-libpipeline1_1.5.7-1_arm64.deb ... Unpacking libpipeline1:arm64 (1.5.7-1) ... Selecting previously unselected package man-db. Preparing to unpack .../09-man-db_2.11.2-2_arm64.deb ... Unpacking man-db (2.11.2-2) ... Selecting previously unselected package m4. Preparing to unpack .../10-m4_1.4.19-3_arm64.deb ... Unpacking m4 (1.4.19-3) ... Selecting previously unselected package autoconf. Preparing to unpack .../11-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../12-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../13-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../14-autopoint_0.21-12_all.deb ... Unpacking autopoint (0.21-12) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../15-libdebhelper-perl_13.11.4_all.deb ... Unpacking libdebhelper-perl (13.11.4) ... Selecting previously unselected package libtool. Preparing to unpack .../16-libtool_2.4.7-5_all.deb ... Unpacking libtool (2.4.7-5) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../17-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../18-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../19-libsub-override-perl_0.09-4_all.deb ... Unpacking libsub-override-perl (0.09-4) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../20-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../21-dh-strip-nondeterminism_1.13.1-1_all.deb ... Unpacking dh-strip-nondeterminism (1.13.1-1) ... Selecting previously unselected package libelf1:arm64. Preparing to unpack .../22-libelf1_0.188-2.1_arm64.deb ... Unpacking libelf1:arm64 (0.188-2.1) ... Selecting previously unselected package dwz. Preparing to unpack .../23-dwz_0.15-1_arm64.deb ... Unpacking dwz (0.15-1) ... Selecting previously unselected package libicu72:arm64. Preparing to unpack .../24-libicu72_72.1-3_arm64.deb ... Unpacking libicu72:arm64 (72.1-3) ... Selecting previously unselected package libxml2:arm64. Preparing to unpack .../25-libxml2_2.9.14+dfsg-1.2_arm64.deb ... Unpacking libxml2:arm64 (2.9.14+dfsg-1.2) ... Selecting previously unselected package gettext. Preparing to unpack .../26-gettext_0.21-12_arm64.deb ... Unpacking gettext (0.21-12) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../27-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../28-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../29-debhelper_13.11.4_all.deb ... Unpacking debhelper (13.11.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../30-dh-golang_1.59_all.deb ... Unpacking dh-golang (1.59) ... Selecting previously unselected package golang-1.19-src. Preparing to unpack .../31-golang-1.19-src_1.19.8-2_all.deb ... Unpacking golang-1.19-src (1.19.8-2) ... Selecting previously unselected package golang-1.19-go. Preparing to unpack .../32-golang-1.19-go_1.19.8-2_arm64.deb ... Unpacking golang-1.19-go (1.19.8-2) ... Selecting previously unselected package golang-src. Preparing to unpack .../33-golang-src_2%3a1.19~1_all.deb ... Unpacking golang-src (2:1.19~1) ... Selecting previously unselected package golang-go:arm64. Preparing to unpack .../34-golang-go_2%3a1.19~1_arm64.deb ... Unpacking golang-go:arm64 (2:1.19~1) ... Selecting previously unselected package golang-any:arm64. Preparing to unpack .../35-golang-any_2%3a1.19~1_arm64.deb ... Unpacking golang-any:arm64 (2:1.19~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../36-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../37-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../38-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../39-golang-golang-x-sys-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.3.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../40-golang-github-mattn-go-isatty-dev_0.0.17-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.17-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../41-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../42-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../43-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../44-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../45-golang-github-stretchr-objx-dev_0.5.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../46-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../47-golang-github-stretchr-testify-dev_1.8.1-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.8.1-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../48-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../49-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../50-golang-github-pkg-errors-dev_0.9.1-2_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-2) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../51-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../52-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../53-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../54-golang-github-hashicorp-go-uuid-dev_1.0.2-2_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../55-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../56-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../57-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../58-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../59-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-golang-protobuf-1-3-dev. Preparing to unpack .../60-golang-github-golang-protobuf-1-3-dev_1.3.5-4_all.deb ... Unpacking golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../61-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../62-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../63-golang-github-json-iterator-go-dev_1.1.12-1_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-1) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../64-golang-github-prometheus-client-model-dev_0.3.0-3_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.3.0-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../65-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../66-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../67-golang-golang-x-text-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.7.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../68-golang-golang-x-net-dev_1%3a0.7.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../69-golang-github-aws-aws-sdk-go-dev_1.44.133-1_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../70-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../71-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../72-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../73-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../74-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../75-golang-golang-x-oauth2-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.3.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../76-golang-google-protobuf-dev_1.28.1-3_all.deb ... Unpacking golang-google-protobuf-dev (1.28.1-3) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../77-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../78-golang-protobuf-extensions-dev_1.0.1-3_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-3) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../79-golang-github-prometheus-common-dev_0.39.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.39.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../80-golang-golang-x-sync-dev_0.1.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.1.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../81-golang-github-prometheus-procfs-dev_0.8.0-3_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.8.0-3) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../82-golang-github-prometheus-client-golang-dev_1.14.0-3_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.14.0-3) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../83-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../84-golang-github-creack-pty-dev_1.1.18-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.18-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../85-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../86-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../87-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:arm64. Preparing to unpack .../88-libsasl2-modules-db_2.1.28+dfsg-10_arm64.deb ... Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-2:arm64. Preparing to unpack .../89-libsasl2-2_2.1.28+dfsg-10_arm64.deb ... Unpacking libsasl2-2:arm64 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../90-libsasl2-dev_2.1.28+dfsg-10_arm64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg-10) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../91-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-github-pkg-diff-dev. Preparing to unpack .../92-golang-github-pkg-diff-dev_0.0~git20210226.20ebb0f-1_all.deb ... Unpacking golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../93-golang-github-rogpeppe-go-internal-dev_1.9.0-1_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../94-golang-github-kr-pretty-dev_0.3.0-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../95-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../96-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../97-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:arm64 (1.5.7-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Setting up libicu72:arm64 (72.1-3) ... Setting up bsdextrautils (2.38.1-5+b1) ... Setting up golang-github-creack-pty-dev (1.1.18-1) ... Setting up golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Setting up golang-github-stretchr-objx-dev (0.5.0-1) ... Setting up libmagic-mgc (1:5.44-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-2) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.11.4) ... Setting up golang-golang-x-sys-dev (0.3.0-1) ... Setting up libmagic1:arm64 (1:5.44-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-12) ... Setting up golang-1.19-src (1.19.8-2) ... Setting up m4 (1.4.19-3) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Setting up file (1:5.44-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg-10) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.28.1-3) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.8.1-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-golang-x-sync-dev (0.1.0-1) ... Setting up autopoint (0.21-12) ... Setting up libsasl2-2:arm64 (2.1.28+dfsg-10) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up sensible-utils (0.0.17+nmu1) ... Setting up golang-golang-x-text-dev (0.7.0-1) ... Setting up libuchardet0:arm64 (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.8.0-3) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up libsub-override-perl (0.09-4) ... Setting up golang-github-prometheus-client-model-dev (0.3.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Setting up golang-src (2:1.19~1) ... Setting up libelf1:arm64 (0.188-2.1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:arm64 (2.9.14+dfsg-1.2) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up golang-1.19-go (1.19.8-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-github-kr-pretty-dev (0.3.0-1) ... Setting up libfile-stripnondeterminism-perl (1.13.1-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.17-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up gettext (0.21-12) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-5) ... Setting up golang-go:arm64 (2:1.19~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-protobuf-extensions-dev (1.0.1-3) ... Setting up golang-github-json-iterator-go-dev (1.1.12-1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:arm64 (2:1.19~1) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg-10) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.13.1-1) ... Setting up dwz (0.15-1) ... Setting up groff-base (1.22.4-10) ... Setting up golang-golang-x-oauth2-dev (0.3.0-1) ... Setting up golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up man-db (2.11.2-2) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.11.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.59) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.39.0-2) ... Setting up golang-github-prometheus-client-golang-dev (1.14.0-3) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Processing triggers for libc-bin (2.36-9) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... usrmerge is already the newest version (35). fakeroot is already the newest version (1.31-1.2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: user script /srv/workspace/pbuilder/12871/tmp/hooks/A99_set_merged_usr starting Re-configuring usrmerge... I: user script /srv/workspace/pbuilder/12871/tmp/hooks/A99_set_merged_usr finished hostname: Temporary failure in name resolution I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture arm64 fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/goarch internal/race internal/goos internal/goexperiment internal/unsafeheader unicode internal/cpu sync/atomic runtime/internal/syscall runtime/internal/math internal/abi runtime/internal/sys unicode/utf8 container/list internal/itoa math/bits crypto/internal/subtle crypto/subtle internal/bytealg runtime/internal/atomic crypto/internal/boring/sig unicode/utf16 encoding internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle math runtime internal/reflectlite sync internal/singleflight internal/testlog math/rand internal/sysinfo runtime/cgo errors sort internal/oserror io internal/safefilepath path github.com/hashicorp/golang-lru/simplelru strconv syscall vendor/golang.org/x/net/dns/dnsmessage crypto/internal/nistec/fiat bytes strings hash crypto/internal/randutil hash/crc32 hash/crc64 vendor/golang.org/x/text/transform crypto reflect crypto/rc4 github.com/hashicorp/go-immutable-radix net/http/internal/ascii regexp/syntax bufio html time internal/syscall/unix internal/syscall/execenv regexp context io/fs internal/poll embed internal/fmtsort encoding/binary os encoding/base64 crypto/internal/edwards25519/field crypto/cipher crypto/md5 vendor/golang.org/x/crypto/curve25519/internal/field vendor/golang.org/x/crypto/internal/poly1305 crypto/internal/nistec encoding/pem golang.org/x/sys/unix crypto/des vendor/golang.org/x/crypto/chacha20 crypto/internal/boring crypto/internal/edwards25519 crypto/aes crypto/sha512 crypto/hmac fmt internal/godebug crypto/sha1 internal/intern crypto/sha256 net/netip path/filepath vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf os/signal io/ioutil encoding/json log net/url math/big encoding/hex compress/flate net vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/http2/hpack compress/gzip mime mime/quotedprintable net/http/internal vendor/golang.org/x/text/secure/bidirule encoding/gob github.com/mattn/go-isatty go/token github.com/mattn/go-colorable text/template/parse flag github.com/fatih/color runtime/debug vendor/golang.org/x/net/idna github.com/hashicorp/go-hclog runtime/trace crypto/rand crypto/elliptic crypto/internal/boring/bbig encoding/asn1 crypto/ed25519 crypto/rsa crypto/dsa text/template testing crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa html/template vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-aarch64-linux-gnu && go test -vet=off -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2023-05-16T04:06:03.312+1400 [INFO] creating new snapshot: path=/tmp/raft3407069546/raft3811143837/snapshots/3-10-1684159563312.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2023-05-16T04:06:03.313+1400 [INFO] creating new snapshot: path=/tmp/raft244181675/snapshots/3-10-1684159563313.tmp testing.go:153: 2023-05-16T04:06:03.314+1400 [WARN] found temporary snapshot: name=3-10-1684159563313.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2023-05-16T04:06:03.315+1400 [INFO] creating new snapshot: path=/tmp/raft1789053147/snapshots/3-10-1684159563315.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention testing.go:153: 2023-05-16T04:06:03.315+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-10-1684159563315.tmp testing.go:153: 2023-05-16T04:06:03.316+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-11-1684159563316.tmp testing.go:153: 2023-05-16T04:06:03.316+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-12-1684159563316.tmp testing.go:153: 2023-05-16T04:06:03.317+1400 [INFO] reaping snapshot: path=/tmp/raft3323901890/snapshots/3-10-1684159563315 testing.go:153: 2023-05-16T04:06:03.317+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-13-1684159563317.tmp testing.go:153: 2023-05-16T04:06:03.318+1400 [INFO] reaping snapshot: path=/tmp/raft3323901890/snapshots/3-11-1684159563316 testing.go:153: 2023-05-16T04:06:03.318+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-14-1684159563318.tmp testing.go:153: 2023-05-16T04:06:03.318+1400 [INFO] reaping snapshot: path=/tmp/raft3323901890/snapshots/3-12-1684159563316 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2023-05-16T04:06:03.320+1400 [INFO] creating new snapshot: path=/tmp/raft3270018814/snapshots/5-130350-1684159563320.tmp testing.go:153: 2023-05-16T04:06:03.320+1400 [INFO] creating new snapshot: path=/tmp/raft3270018814/snapshots/36-204917-1684159563320.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.08s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.01s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.444+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.450+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.464+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.488+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.532+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.616+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.777+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:04.098+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:04.739+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:05.740+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-05-16T04:06:06.744+1400 [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-84686a94-bc38-8170-28f0-f77a57ac3307: 2023-05-16T04:06:07.450+1400 [INFO] server-84686a94-bc38-8170-28f0-f77a57ac3307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84686a94-bc38-8170-28f0-f77a57ac3307 Address:84686a94-bc38-8170-28f0-f77a57ac3307}]" testing.go:149: server-84686a94-bc38-8170-28f0-f77a57ac3307: 2023-05-16T04:06:07.450+1400 [INFO] server-84686a94-bc38-8170-28f0-f77a57ac3307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84686a94-bc38-8170-28f0-f77a57ac3307 Address:84686a94-bc38-8170-28f0-f77a57ac3307}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: 2023-05-16T04:06:07.451+1400 [INFO] server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1 Address:f510d0eb-4ab4-b26f-78eb-06f157c1a8b1}]" testing.go:149: server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: 2023-05-16T04:06:07.451+1400 [INFO] server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1 Address:f510d0eb-4ab4-b26f-78eb-06f157c1a8b1}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: initial configuration: index=0 servers=[] testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: initial configuration: index=0 servers=[] testing.go:149: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering follower state: follower="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Follower]" leader= testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering follower state: follower="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Follower]" leader= testLoggerAdapter verbose: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: entering follower state: follower="Node at f9281b59-ecbf-f154-4840-0818345c6687 [Follower]" leader= testing.go:149: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: entering follower state: follower="Node at f9281b59-ecbf-f154-4840-0818345c6687 [Follower]" leader= testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: initial configuration: index=0 servers=[] testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.452+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:07.452+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: entering follower state: follower="Node at c1b136b0-48f0-5622-80f5-752005b26c00 [Follower]" leader= testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: entering follower state: follower="Node at c1b136b0-48f0-5622-80f5-752005b26c00 [Follower]" leader= testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.511+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: no known peers, aborting election testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.511+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: no known peers, aborting election testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering candidate state: node="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Candidate]" term=2 testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering candidate state: node="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Candidate]" term=2 testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: election won: tally=2 testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: election won: tally=2 testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering leader state: leader="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Leader]" testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering leader state: leader="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Leader]" testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-f9281b59-ecbf-f154-4840-0818345c6687 testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-f9281b59-ecbf-f154-4840-0818345c6687 testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-c1b136b0-48f0-5622-80f5-752005b26c00 testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-c1b136b0-48f0-5622-80f5-752005b26c00 testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.531+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.531+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" next=1 testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" next=1 testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" testLoggerAdapter verbose: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.532+1400 [WARN] server-f9281b59-ecbf-f154-4840-0818345c6687: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.532+1400 [WARN] server-f9281b59-ecbf-f154-4840-0818345c6687: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" next=1 testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" next=1 testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.637+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:07.532024362 +1400 +14 m=+4.334605997 (2 nodes), 79.147234ms from start of poll, 79.944063ms from cluster start. Timeout at 2023-05-16 04:06:07.637868271 +1400 +14 m=+4.440449906, 105.843909ms after stability testing.go:149: cluster: 2023-05-16T04:06:07.637+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:07.532024362 +1400 +14 m=+4.334605997 (2 nodes), 79.147234ms from start of poll, 79.944063ms from cluster start. Timeout at 2023-05-16 04:06:07.637868271 +1400 +14 m=+4.440449906, 105.843909ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.638+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:07.638+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.744+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.638175755 +1400 +14 m=+4.440757390 (1 nodes), 1.9µs from start of poll, 186.095456ms from cluster start. Timeout at 2023-05-16 04:06:07.744056605 +1400 +14 m=+4.546638240, 105.88085ms after stability testing.go:149: cluster: 2023-05-16T04:06:07.744+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.638175755 +1400 +14 m=+4.440757390 (1 nodes), 1.9µs from start of poll, 186.095456ms from cluster start. Timeout at 2023-05-16 04:06:07.744056605 +1400 +14 m=+4.546638240, 105.88085ms after stability testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" --- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-3717b61b-db44-903d-3823-d8fa15255f74: 2023-05-16T04:06:07.756+1400 [INFO] server-3717b61b-db44-903d-3823-d8fa15255f74: initial configuration: index=0 servers=[] testing.go:149: server-3717b61b-db44-903d-3823-d8fa15255f74: 2023-05-16T04:06:07.756+1400 [INFO] server-3717b61b-db44-903d-3823-d8fa15255f74: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" testing.go:149: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.758+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.758+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.758+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: initial configuration: index=0 servers=[] testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.758+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.758+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:07.758+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: entering follower state: follower="Node at 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 [Follower]" leader= testing.go:149: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: entering follower state: follower="Node at 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 [Follower]" leader= testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.759+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering follower state: follower="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Follower]" leader= testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.759+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering follower state: follower="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Follower]" leader= testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.759+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: entering follower state: follower="Node at 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 [Follower]" leader= testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.759+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: entering follower state: follower="Node at 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 [Follower]" leader= testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering candidate state: node="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Candidate]" term=2 testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering candidate state: node="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Candidate]" term=2 testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: election won: tally=2 testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: election won: tally=2 testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering leader state: leader="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Leader]" testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering leader state: leader="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Leader]" testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.824+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.824+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.827+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: no known peers, aborting election testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.827+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: no known peers, aborting election testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.929+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.823959428 +1400 +14 m=+4.626541063 (1 nodes), 65.054088ms from start of poll, 66.128741ms from cluster start. Timeout at 2023-05-16 04:06:07.929297351 +1400 +14 m=+4.731878986, 105.337923ms after stability testing.go:149: cluster: 2023-05-16T04:06:07.929+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.823959428 +1400 +14 m=+4.626541063 (1 nodes), 65.054088ms from start of poll, 66.128741ms from cluster start. Timeout at 2023-05-16 04:06:07.929297351 +1400 +14 m=+4.731878986, 105.337923ms after stability testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: updating configuration: command=AddStaging server-id=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 server-addr=5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd} {Suffrage:Voter ID:server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 Address:5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}]" testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: updating configuration: command=AddStaging server-id=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 server-addr=5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd} {Suffrage:Voter ID:server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 Address:5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}]" testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.930+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.930+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: appendEntries rejected, sending older logs: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" next=1 testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: appendEntries rejected, sending older logs: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" next=1 testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.930+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:07.930+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.036+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.930718028 +1400 +14 m=+4.733299663 (1 nodes), 1.4µs from start of poll, 172.887341ms from cluster start. Timeout at 2023-05-16 04:06:08.035940709 +1400 +14 m=+4.838522364, 105.222701ms after stability testing.go:149: cluster: 2023-05-16T04:06:08.036+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.930718028 +1400 +14 m=+4.733299663 (1 nodes), 1.4µs from start of poll, 172.887341ms from cluster start. Timeout at 2023-05-16 04:06:08.035940709 +1400 +14 m=+4.838522364, 105.222701ms after stability testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" --- PASS: TestRaft_HasExistingState (0.28s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.037+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35 Address:ca2d9894-d5b0-e52c-3865-c5408b8cdc35}]" testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.037+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35 Address:ca2d9894-d5b0-e52c-3865-c5408b8cdc35}]" testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.038+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering follower state: follower="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Follower]" leader= testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.038+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering follower state: follower="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Follower]" leader= testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [WARN] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [WARN] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering candidate state: node="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Candidate]" term=2 testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering candidate state: node="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Candidate]" term=2 testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: election won: tally=1 testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: election won: tally=1 testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.110+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering leader state: leader="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Leader]" testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.110+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering leader state: leader="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Leader]" --- PASS: TestRaft_SingleNode (0.07s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering follower state: follower="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Follower]" leader= testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering follower state: follower="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Follower]" leader= testLoggerAdapter verbose: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" testing.go:149: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" testLoggerAdapter verbose: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: entering follower state: follower="Node at ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 [Follower]" leader= testing.go:149: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: entering follower state: follower="Node at ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 [Follower]" leader= testLoggerAdapter verbose: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" testing.go:149: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.112+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:08.112+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: entering follower state: follower="Node at d512c739-8ebb-7e56-8d0d-5440bf285c29 [Follower]" leader= testing.go:149: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: entering follower state: follower="Node at d512c739-8ebb-7e56-8d0d-5440bf285c29 [Follower]" leader= testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [WARN] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [WARN] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering candidate state: node="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Candidate]" term=2 testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering candidate state: node="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Candidate]" term=2 testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: election won: tally=2 testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: election won: tally=2 testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering leader state: leader="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Leader]" testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering leader state: leader="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Leader]" testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-d512c739-8ebb-7e56-8d0d-5440bf285c29 testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-d512c739-8ebb-7e56-8d0d-5440bf285c29 testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.285+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.179678926 +1400 +14 m=+4.982260561 (2 nodes), 67.405116ms from start of poll, 68.440488ms from cluster start. Timeout at 2023-05-16 04:06:08.284991869 +1400 +14 m=+5.087573484, 105.312923ms after stability testing.go:149: cluster: 2023-05-16T04:06:08.285+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.179678926 +1400 +14 m=+4.982260561 (2 nodes), 67.405116ms from start of poll, 68.440488ms from cluster start. Timeout at 2023-05-16 04:06:08.284991869 +1400 +14 m=+5.087573484, 105.312923ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.285+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:08.285+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.390+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.285246352 +1400 +14 m=+5.087827987 (1 nodes), 2.24µs from start of poll, 174.007914ms from cluster start. Timeout at 2023-05-16 04:06:08.390862878 +1400 +14 m=+5.193444493, 105.616506ms after stability testing.go:149: cluster: 2023-05-16T04:06:08.390+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.285246352 +1400 +14 m=+5.087827987 (1 nodes), 2.24µs from start of poll, 174.007914ms from cluster start. Timeout at 2023-05-16 04:06:08.390862878 +1400 +14 m=+5.193444493, 105.616506ms after stability testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" --- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.404+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.404+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.405+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:08.405+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.405+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.405+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering follower state: follower="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Follower]" leader= testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering follower state: follower="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Follower]" leader= testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=2 testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=2 testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: election won: tally=2 testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: election won: tally=2 testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering leader state: leader="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Leader]" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering leader state: leader="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Leader]" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-b5d8336e-7956-5b47-63b1-d4661764bc2f testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-b5d8336e-7956-5b47-63b1-d4661764bc2f testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.568+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.462838907 +1400 +14 m=+5.265420523 (2 nodes), 57.280197ms from start of poll, 58.257288ms from cluster start. Timeout at 2023-05-16 04:06:08.568296772 +1400 +14 m=+5.370878427, 105.457904ms after stability testing.go:149: cluster: 2023-05-16T04:06:08.568+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.462838907 +1400 +14 m=+5.265420523 (2 nodes), 57.280197ms from start of poll, 58.257288ms from cluster start. Timeout at 2023-05-16 04:06:08.568296772 +1400 +14 m=+5.370878427, 105.457904ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.568+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:08.568+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.674+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.568603476 +1400 +14 m=+5.371185111 (1 nodes), 1.9µs from start of poll, 164.021876ms from cluster start. Timeout at 2023-05-16 04:06:08.674153461 +1400 +14 m=+5.476735116, 105.550005ms after stability testing.go:149: cluster: 2023-05-16T04:06:08.674+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.568603476 +1400 +14 m=+5.371185111 (1 nodes), 1.9µs from start of poll, 164.021876ms from cluster start. Timeout at 2023-05-16 04:06:08.674153461 +1400 +14 m=+5.476735116, 105.550005ms after stability raft_test.go:336: [INFO] Disconnecting Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Leader] testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="append pipeline closed" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="append pipeline closed" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.689+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.689+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.691+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:08.691+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="append pipeline closed" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="append pipeline closed" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.698+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.698+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.700+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.700+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.715+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.715+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.721+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.721+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.726+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.726+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.727+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.727+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.731+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=50.758979ms testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.731+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=50.758979ms testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.732+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.732+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.739+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.739+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-b5d8336e-7956-5b47-63b1-d4661764bc2f time=57.909424ms testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-b5d8336e-7956-5b47-63b1-d4661764bc2f time=57.909424ms testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=61.191202ms testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=61.191202ms testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact quorum of nodes, stepping down testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.754+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.754+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.758+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.758+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=3 testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=3 testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.781+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: rejecting vote request since we have a leader: from=b5d8336e-7956-5b47-63b1-d4661764bc2f leader=38d37955-a474-b68e-e73a-2da64fa8722b testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.781+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: rejecting vote request since we have a leader: from=b5d8336e-7956-5b47-63b1-d4661764bc2f leader=38d37955-a474-b68e-e73a-2da64fa8722b testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=3 testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=3 testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering candidate state: node="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Candidate]" term=3 testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering candidate state: node="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Candidate]" term=3 testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.806+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: duplicate requestVote for same term: term=3 testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.806+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [ERROR] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [ERROR] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: Election timeout reached, restarting election testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: Election timeout reached, restarting election testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=4 testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=4 testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.852+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.852+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: election won: tally=2 testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: election won: tally=2 testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering leader state: leader="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Leader]" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering leader state: leader="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Leader]" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-38d37955-a474-b68e-e73a-2da64fa8722b testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-38d37955-a474-b68e-e73a-2da64fa8722b testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.861+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.861+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.872+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.872+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.880+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.880+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.881+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: Election timeout reached, restarting election testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.881+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: Election timeout reached, restarting election testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=4 testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=4 testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.891+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.891+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.900+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.900+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.904+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=51.141963ms testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.904+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=51.141963ms testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.919+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.919+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.930+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.930+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.950+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=97.116286ms testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.950+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=97.116286ms testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.959+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.853334416 +1400 +14 m=+5.655916051 (1 nodes), 161.83455ms from start of poll, 448.752816ms from cluster start. Timeout at 2023-05-16 04:06:08.959014683 +1400 +14 m=+5.761596318, 105.680267ms after stability testing.go:149: cluster: 2023-05-16T04:06:08.959+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.853334416 +1400 +14 m=+5.655916051 (1 nodes), 161.83455ms from start of poll, 448.752816ms from cluster start. Timeout at 2023-05-16 04:06:08.959014683 +1400 +14 m=+5.761596318, 105.680267ms after stability raft_test.go:371: [INFO] Reconnecting Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate] testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.967+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader=b5d8336e-7956-5b47-63b1-d4661764bc2f testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.967+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader=b5d8336e-7956-5b47-63b1-d4661764bc2f testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.967+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.967+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" --- PASS: TestRaft_LeaderFail (0.56s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.969+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" testing.go:149: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.969+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" testLoggerAdapter verbose: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" testing.go:149: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.970+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:08.970+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.970+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: entering follower state: follower="Node at 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a [Follower]" leader= testing.go:149: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.970+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: entering follower state: follower="Node at 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a [Follower]" leader= testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering follower state: follower="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Follower]" leader= testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering follower state: follower="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Follower]" leader= testLoggerAdapter verbose: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: entering follower state: follower="Node at 7d849bf5-59ad-984d-0abd-fe66b861e8ef [Follower]" leader= testing.go:149: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: entering follower state: follower="Node at 7d849bf5-59ad-984d-0abd-fe66b861e8ef [Follower]" leader= testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [WARN] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [WARN] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering candidate state: node="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Candidate]" term=2 testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering candidate state: node="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Candidate]" term=2 testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: election won: tally=2 testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: election won: tally=2 testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering leader state: leader="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Leader]" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering leader state: leader="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Leader]" testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-7d849bf5-59ad-984d-0abd-fe66b861e8ef testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-7d849bf5-59ad-984d-0abd-fe66b861e8ef testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.128+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.022953398 +1400 +14 m=+5.825535033 (1 nodes), 52.437479ms from start of poll, 53.432211ms from cluster start. Timeout at 2023-05-16 04:06:09.128848627 +1400 +14 m=+5.931430282, 105.895249ms after stability testing.go:149: cluster: 2023-05-16T04:06:09.128+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.022953398 +1400 +14 m=+5.825535033 (1 nodes), 52.437479ms from start of poll, 53.432211ms from cluster start. Timeout at 2023-05-16 04:06:09.128848627 +1400 +14 m=+5.931430282, 105.895249ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.129+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:09.129+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.234+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.129128291 +1400 +14 m=+5.931709926 (2 nodes), 2.98µs from start of poll, 159.607104ms from cluster start. Timeout at 2023-05-16 04:06:09.234232611 +1400 +14 m=+6.036814226, 105.1043ms after stability testing.go:149: cluster: 2023-05-16T04:06:09.234+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.129128291 +1400 +14 m=+5.931709926 (2 nodes), 2.98µs from start of poll, 159.607104ms from cluster start. Timeout at 2023-05-16 04:06:09.234232611 +1400 +14 m=+6.036814226, 105.1043ms after stability testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to heartbeat to: peer=0b440392-2aab-a6c3-452a-6c9b5a4f2b6a error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to heartbeat to: peer=0b440392-2aab-a6c3-452a-6c9b5a4f2b6a error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to pipeline appendEntries: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="append pipeline closed" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to pipeline appendEntries: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="append pipeline closed" testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to appendEntries to: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to appendEntries to: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.246+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.246+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.256+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:09.256+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.361+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.256438813 +1400 +14 m=+6.059020428 (1 nodes), 2.38µs from start of poll, 286.917606ms from cluster start. Timeout at 2023-05-16 04:06:09.361539094 +1400 +14 m=+6.164120709, 105.100281ms after stability testing.go:149: cluster: 2023-05-16T04:06:09.361+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.256438813 +1400 +14 m=+6.059020428 (1 nodes), 2.38µs from start of poll, 286.917606ms from cluster start. Timeout at 2023-05-16 04:06:09.361539094 +1400 +14 m=+6.164120709, 105.100281ms after stability testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" --- PASS: TestRaft_BehindFollower (0.39s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.363+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.363+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" testLoggerAdapter verbose: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.363+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" testing.go:149: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.363+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" testLoggerAdapter verbose: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" testing.go:149: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.364+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:09.364+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.364+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering follower state: follower="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Follower]" leader= testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.364+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering follower state: follower="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Follower]" leader= testLoggerAdapter verbose: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.364+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: entering follower state: follower="Node at 7985cc23-7611-73cf-a113-89a9858f0e0c [Follower]" leader= testing.go:149: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.364+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: entering follower state: follower="Node at 7985cc23-7611-73cf-a113-89a9858f0e0c [Follower]" leader= testLoggerAdapter verbose: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: entering follower state: follower="Node at 9703effd-27b2-0749-7493-c09e6e6f7dcf [Follower]" leader= testing.go:149: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: entering follower state: follower="Node at 9703effd-27b2-0749-7493-c09e6e6f7dcf [Follower]" leader= testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.419+1400 [WARN] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.419+1400 [WARN] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering candidate state: node="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Candidate]" term=2 testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering candidate state: node="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Candidate]" term=2 testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: election won: tally=2 testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: election won: tally=2 testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering leader state: leader="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Leader]" testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering leader state: leader="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Leader]" testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-7985cc23-7611-73cf-a113-89a9858f0e0c testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-7985cc23-7611-73cf-a113-89a9858f0e0c testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-9703effd-27b2-0749-7493-c09e6e6f7dcf testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-9703effd-27b2-0749-7493-c09e6e6f7dcf testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.421+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.421+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.421234878 +1400 +14 m=+6.223816513 (1 nodes), 57.078394ms from start of poll, 57.975664ms from cluster start. Timeout at 2023-05-16 04:06:09.526416699 +1400 +14 m=+6.328998314, 105.181801ms after stability testing.go:149: cluster: 2023-05-16T04:06:09.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.421234878 +1400 +14 m=+6.223816513 (1 nodes), 57.078394ms from start of poll, 57.975664ms from cluster start. Timeout at 2023-05-16 04:06:09.526416699 +1400 +14 m=+6.328998314, 105.181801ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.526+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:09.526+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.632+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.526693663 +1400 +14 m=+6.329275278 (2 nodes), 2.64µs from start of poll, 163.434429ms from cluster start. Timeout at 2023-05-16 04:06:09.632340569 +1400 +14 m=+6.434922184, 105.646906ms after stability testing.go:149: cluster: 2023-05-16T04:06:09.632+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.526693663 +1400 +14 m=+6.329275278 (2 nodes), 2.64µs from start of poll, 163.434429ms from cluster start. Timeout at 2023-05-16 04:06:09.632340569 +1400 +14 m=+6.434922184, 105.646906ms after stability testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" --- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.634+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.634+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" testLoggerAdapter verbose: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.634+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" testing.go:149: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.634+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" testLoggerAdapter verbose: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.634+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" testing.go:149: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.634+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.635+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:09.635+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.635+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering follower state: follower="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Follower]" leader= testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.635+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering follower state: follower="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Follower]" leader= testLoggerAdapter verbose: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.635+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: entering follower state: follower="Node at c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 [Follower]" leader= testing.go:149: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.635+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: entering follower state: follower="Node at c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 [Follower]" leader= testLoggerAdapter verbose: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.635+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: entering follower state: follower="Node at 6c1d3058-ab57-c09f-f73b-038557ebf908 [Follower]" leader= testing.go:149: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.635+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: entering follower state: follower="Node at 6c1d3058-ab57-c09f-f73b-038557ebf908 [Follower]" leader= testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [WARN] server-54e4f296-2322-c79c-b366-f01788e2f8dd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [WARN] server-54e4f296-2322-c79c-b366-f01788e2f8dd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering candidate state: node="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Candidate]" term=2 testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering candidate state: node="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Candidate]" term=2 testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: election won: tally=2 testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: election won: tally=2 testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering leader state: leader="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Leader]" testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering leader state: leader="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Leader]" testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-6c1d3058-ab57-c09f-f73b-038557ebf908 testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-6c1d3058-ab57-c09f-f73b-038557ebf908 testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.006+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.801721708 +1400 +14 m=+6.604303323 (1 nodes), 166.589706ms from start of poll, 167.561057ms from cluster start. Timeout at 2023-05-16 04:06:10.006815189 +1400 +14 m=+6.809396804, 205.093481ms after stability testing.go:149: cluster: 2023-05-16T04:06:10.006+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.801721708 +1400 +14 m=+6.604303323 (1 nodes), 166.589706ms from start of poll, 167.561057ms from cluster start. Timeout at 2023-05-16 04:06:10.006815189 +1400 +14 m=+6.809396804, 205.093481ms after stability testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.023+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.023+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.024+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.024+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" --- PASS: TestRaft_ApplyConcurrent (0.39s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24cb2fea-5c16-88ba-4e61-5bf572919754 Address:24cb2fea-5c16-88ba-4e61-5bf572919754}]" testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24cb2fea-5c16-88ba-4e61-5bf572919754 Address:24cb2fea-5c16-88ba-4e61-5bf572919754}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.025+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:10.025+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering follower state: follower="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Follower]" leader= testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering follower state: follower="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Follower]" leader= testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [WARN] server-24cb2fea-5c16-88ba-4e61-5bf572919754: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [WARN] server-24cb2fea-5c16-88ba-4e61-5bf572919754: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering candidate state: node="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Candidate]" term=2 testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering candidate state: node="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Candidate]" term=2 testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: election won: tally=1 testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: election won: tally=1 testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering leader state: leader="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Leader]" testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering leader state: leader="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Leader]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.131085075 +1400 +14 m=+6.933666730 (1 nodes), 105.344463ms from start of poll, 105.781268ms from cluster start. Timeout at 2023-05-16 04:06:10.332868837 +1400 +14 m=+7.135450472, 201.783742ms after stability testing.go:149: cluster: 2023-05-16T04:06:10.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.131085075 +1400 +14 m=+6.933666730 (1 nodes), 105.344463ms from start of poll, 105.781268ms from cluster start. Timeout at 2023-05-16 04:06:10.332868837 +1400 +14 m=+7.135450472, 201.783742ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.31s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.340+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.340+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.341+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering follower state: follower="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Follower]" leader= testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.341+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering follower state: follower="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Follower]" leader= testLoggerAdapter verbose: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" testing.go:149: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" testLoggerAdapter verbose: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: entering follower state: follower="Node at 92c6272c-d873-ff1e-12a8-43fe516c3d6c [Follower]" leader= testing.go:149: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: entering follower state: follower="Node at 92c6272c-d873-ff1e-12a8-43fe516c3d6c [Follower]" leader= testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: initial configuration: index=0 servers=[] testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.341+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:10.341+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: entering follower state: follower="Node at c406396c-e0d4-7306-17ae-18698aa688af [Follower]" leader= testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: entering follower state: follower="Node at c406396c-e0d4-7306-17ae-18698aa688af [Follower]" leader= testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.410+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: no known peers, aborting election testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.410+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: no known peers, aborting election testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering candidate state: node="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Candidate]" term=2 testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering candidate state: node="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Candidate]" term=2 testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: election won: tally=2 testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: election won: tally=2 testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering leader state: leader="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Leader]" testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering leader state: leader="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Leader]" testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-92c6272c-d873-ff1e-12a8-43fe516c3d6c testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-92c6272c-d873-ff1e-12a8-43fe516c3d6c testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.518+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.412629998 +1400 +14 m=+7.215211633 (1 nodes), 70.959617ms from start of poll, 72.123231ms from cluster start. Timeout at 2023-05-16 04:06:10.518556428 +1400 +14 m=+7.321138063, 105.92643ms after stability testing.go:149: cluster: 2023-05-16T04:06:10.518+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.412629998 +1400 +14 m=+7.215211633 (1 nodes), 70.959617ms from start of poll, 72.123231ms from cluster start. Timeout at 2023-05-16 04:06:10.518556428 +1400 +14 m=+7.321138063, 105.92643ms after stability testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.518+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: updating configuration: command=AddStaging server-id=server-c406396c-e0d4-7306-17ae-18698aa688af server-addr=c406396c-e0d4-7306-17ae-18698aa688af servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c} {Suffrage:Voter ID:server-c406396c-e0d4-7306-17ae-18698aa688af Address:c406396c-e0d4-7306-17ae-18698aa688af}]" testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.518+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: updating configuration: command=AddStaging server-id=server-c406396c-e0d4-7306-17ae-18698aa688af server-addr=c406396c-e0d4-7306-17ae-18698aa688af servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c} {Suffrage:Voter ID:server-c406396c-e0d4-7306-17ae-18698aa688af Address:c406396c-e0d4-7306-17ae-18698aa688af}]" testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-c406396c-e0d4-7306-17ae-18698aa688af testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-c406396c-e0d4-7306-17ae-18698aa688af testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.519+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.519+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: appendEntries rejected, sending older logs: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" next=1 testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: appendEntries rejected, sending older logs: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" next=1 testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.519+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:10.519+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.625+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.519864664 +1400 +14 m=+7.322446299 (1 nodes), 1.72µs from start of poll, 179.357897ms from cluster start. Timeout at 2023-05-16 04:06:10.625726273 +1400 +14 m=+7.428307968, 105.861669ms after stability testing.go:149: cluster: 2023-05-16T04:06:10.625+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.519864664 +1400 +14 m=+7.322446299 (1 nodes), 1.72µs from start of poll, 179.357897ms from cluster start. Timeout at 2023-05-16 04:06:10.625726273 +1400 +14 m=+7.428307968, 105.861669ms after stability testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" --- PASS: TestRaft_JoinNode (0.29s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.627+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86}]" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.627+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86}]" testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.627+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: initial configuration: index=0 servers=[] testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.627+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.628+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: initial configuration: index=0 servers=[] testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.628+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.628+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:10.628+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.628+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering follower state: follower="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Follower]" leader= testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.628+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering follower state: follower="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Follower]" leader= testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.628+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: entering follower state: follower="Node at e6860ba8-8e12-83cd-f247-b2285f13e711 [Follower]" leader= testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.628+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: entering follower state: follower="Node at e6860ba8-8e12-83cd-f247-b2285f13e711 [Follower]" leader= testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.629+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: entering follower state: follower="Node at 57141e17-8ef4-1822-30de-a908d71f1065 [Follower]" leader= testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.629+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: entering follower state: follower="Node at 57141e17-8ef4-1822-30de-a908d71f1065 [Follower]" leader= testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.693+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: no known peers, aborting election testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.693+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: no known peers, aborting election testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.693+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: no known peers, aborting election testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.693+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: no known peers, aborting election testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering candidate state: node="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Candidate]" term=2 testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering candidate state: node="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Candidate]" term=2 testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: election won: tally=1 testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: election won: tally=1 testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering leader state: leader="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Leader]" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering leader state: leader="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Leader]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.813+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.708056825 +1400 +14 m=+7.510638460 (1 nodes), 79.417358ms from start of poll, 80.763114ms from cluster start. Timeout at 2023-05-16 04:06:10.813538289 +1400 +14 m=+7.616119944, 105.481484ms after stability testing.go:149: cluster: 2023-05-16T04:06:10.813+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.708056825 +1400 +14 m=+7.510638460 (1 nodes), 79.417358ms from start of poll, 80.763114ms from cluster start. Timeout at 2023-05-16 04:06:10.813538289 +1400 +14 m=+7.616119944, 105.481484ms after stability testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.813+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-e6860ba8-8e12-83cd-f247-b2285f13e711 server-addr=e6860ba8-8e12-83cd-f247-b2285f13e711 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711}]" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.813+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-e6860ba8-8e12-83cd-f247-b2285f13e711 server-addr=e6860ba8-8e12-83cd-f247-b2285f13e711 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711}]" testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-e6860ba8-8e12-83cd-f247-b2285f13e711 testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-e6860ba8-8e12-83cd-f247-b2285f13e711 testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.814+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:10.814+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.814+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.814+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" next=1 testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" next=1 testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.919+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.814279458 +1400 +14 m=+7.616861093 (1 nodes), 2.46µs from start of poll, 186.985747ms from cluster start. Timeout at 2023-05-16 04:06:10.919554361 +1400 +14 m=+7.722136016, 105.274923ms after stability testing.go:149: cluster: 2023-05-16T04:06:10.919+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.814279458 +1400 +14 m=+7.616861093 (1 nodes), 2.46µs from start of poll, 186.985747ms from cluster start. Timeout at 2023-05-16 04:06:10.919554361 +1400 +14 m=+7.722136016, 105.274923ms after stability testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.919+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-57141e17-8ef4-1822-30de-a908d71f1065 server-addr=57141e17-8ef4-1822-30de-a908d71f1065 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711} {Suffrage:Voter ID:server-57141e17-8ef4-1822-30de-a908d71f1065 Address:57141e17-8ef4-1822-30de-a908d71f1065}]" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.919+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-57141e17-8ef4-1822-30de-a908d71f1065 server-addr=57141e17-8ef4-1822-30de-a908d71f1065 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711} {Suffrage:Voter ID:server-57141e17-8ef4-1822-30de-a908d71f1065 Address:57141e17-8ef4-1822-30de-a908d71f1065}]" testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-57141e17-8ef4-1822-30de-a908d71f1065 testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-57141e17-8ef4-1822-30de-a908d71f1065 testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.920+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.920+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" next=1 testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" next=1 testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.920+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:10.920+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.026+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.920995618 +1400 +14 m=+7.723577233 (1 nodes), 1.56µs from start of poll, 293.701887ms from cluster start. Timeout at 2023-05-16 04:06:11.026666165 +1400 +14 m=+7.829247800, 105.670567ms after stability testing.go:149: cluster: 2023-05-16T04:06:11.026+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.920995618 +1400 +14 m=+7.723577233 (1 nodes), 1.56µs from start of poll, 293.701887ms from cluster start. Timeout at 2023-05-16 04:06:11.026666165 +1400 +14 m=+7.829247800, 105.670567ms after stability testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" --- PASS: TestRaft_JoinNode_ConfigStore (0.40s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.028+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.028+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testLoggerAdapter verbose: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testing.go:149: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.029+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:11.029+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.029+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering follower state: follower="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Follower]" leader= testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.029+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering follower state: follower="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Follower]" leader= testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering follower state: follower="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Follower]" leader= testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering follower state: follower="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Follower]" leader= testLoggerAdapter verbose: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: entering follower state: follower="Node at b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e [Follower]" leader= testing.go:149: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: entering follower state: follower="Node at b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e [Follower]" leader= testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering candidate state: node="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Candidate]" term=2 testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering candidate state: node="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Candidate]" term=2 testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: election won: tally=2 testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: election won: tally=2 testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering leader state: leader="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Leader]" testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering leader state: leader="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Leader]" testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.208+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.102616161 +1400 +14 m=+7.905197796 (1 nodes), 73.000262ms from start of poll, 74.040334ms from cluster start. Timeout at 2023-05-16 04:06:11.208040805 +1400 +14 m=+8.010622480, 105.424684ms after stability testing.go:149: cluster: 2023-05-16T04:06:11.208+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.102616161 +1400 +14 m=+7.905197796 (1 nodes), 73.000262ms from start of poll, 74.040334ms from cluster start. Timeout at 2023-05-16 04:06:11.208040805 +1400 +14 m=+8.010622480, 105.424684ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.214+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:11.214+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.319+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.214281459 +1400 +14 m=+8.016863094 (2 nodes), 2.98µs from start of poll, 185.705632ms from cluster start. Timeout at 2023-05-16 04:06:11.319536181 +1400 +14 m=+8.122117856, 105.254762ms after stability testing.go:149: cluster: 2023-05-16T04:06:11.319+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.214281459 +1400 +14 m=+8.016863094 (2 nodes), 2.98µs from start of poll, 185.705632ms from cluster start. Timeout at 2023-05-16 04:06:11.319536181 +1400 +14 m=+8.122117856, 105.254762ms after stability testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: updating configuration: command=RemoveServer server-id=server-29ae47ac-1aec-8c65-d250-223c0072191e server-addr= servers="[{Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: updating configuration: command=RemoveServer server-id=server-29ae47ac-1aec-8c65-d250-223c0072191e server-addr= servers="[{Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: removed peer, stopping replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e last-index=3 testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: removed peer, stopping replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e last-index=3 testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.320+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.320+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [WARN] server-29ae47ac-1aec-8c65-d250-223c0072191e: heartbeat timeout reached, starting election: last-leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [WARN] server-29ae47ac-1aec-8c65-d250-223c0072191e: heartbeat timeout reached, starting election: last-leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering candidate state: node="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Candidate]" term=3 testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering candidate state: node="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Candidate]" term=3 testLoggerAdapter verbose: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.373+1400 [WARN] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad testing.go:149: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.373+1400 [WARN] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.373+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.373+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.426+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.426+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" --- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.427+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.427+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.428+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.428+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.428+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:11.428+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.428+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering follower state: follower="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Follower]" leader= testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.428+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering follower state: follower="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Follower]" leader= testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.429+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering follower state: follower="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Follower]" leader= testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.429+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering follower state: follower="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Follower]" leader= testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.481+1400 [WARN] server-a592c42f-3461-e569-9c07-b354084d8ff7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.481+1400 [WARN] server-a592c42f-3461-e569-9c07-b354084d8ff7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering candidate state: node="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Candidate]" term=2 testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering candidate state: node="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Candidate]" term=2 testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: election won: tally=2 testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: election won: tally=2 testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering leader state: leader="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Leader]" testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering leader state: leader="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Leader]" testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.483+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.483+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.589+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.483279713 +1400 +14 m=+8.285861328 (1 nodes), 54.484363ms from start of poll, 55.612616ms from cluster start. Timeout at 2023-05-16 04:06:11.589061442 +1400 +14 m=+8.391643077, 105.781749ms after stability testing.go:149: cluster: 2023-05-16T04:06:11.589+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.483279713 +1400 +14 m=+8.285861328 (1 nodes), 54.484363ms from start of poll, 55.612616ms from cluster start. Timeout at 2023-05-16 04:06:11.589061442 +1400 +14 m=+8.391643077, 105.781749ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.595+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:11.595+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.700+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.595304355 +1400 +14 m=+8.397885970 (2 nodes), 3.44µs from start of poll, 167.637258ms from cluster start. Timeout at 2023-05-16 04:06:11.700311235 +1400 +14 m=+8.502892870, 105.0069ms after stability testing.go:149: cluster: 2023-05-16T04:06:11.700+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.595304355 +1400 +14 m=+8.397885970 (2 nodes), 3.44µs from start of poll, 167.637258ms from cluster start. Timeout at 2023-05-16 04:06:11.700311235 +1400 +14 m=+8.502892870, 105.0069ms after stability testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: updating configuration: command=RemoveServer server-id=server-a592c42f-3461-e569-9c07-b354084d8ff7 server-addr= servers="[{Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: updating configuration: command=RemoveServer server-id=server-a592c42f-3461-e569-9c07-b354084d8ff7 server-addr= servers="[{Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: removed ourself, shutting down testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: removed ourself, shutting down testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [WARN] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [WARN] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering candidate state: node="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Candidate]" term=3 testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering candidate state: node="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Candidate]" term=3 testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.769+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: rejecting vote request since we have a leader: from=8b196570-6b00-d4e2-54d5-d43ed3f56f82 leader=a592c42f-3461-e569-9c07-b354084d8ff7 testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.769+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: rejecting vote request since we have a leader: from=8b196570-6b00-d4e2-54d5-d43ed3f56f82 leader=a592c42f-3461-e569-9c07-b354084d8ff7 testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.780+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.780+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.781+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=3 testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.781+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=3 testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.781+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: duplicate requestVote for same term: term=3 testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.781+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: Election timeout reached, restarting election testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: Election timeout reached, restarting election testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=4 testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=4 testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.836+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.836+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: election won: tally=2 testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: election won: tally=2 testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering leader state: leader="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Leader]" testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering leader state: leader="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Leader]" testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.912+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:11.912+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.018+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.912402318 +1400 +14 m=+8.714983933 (1 nodes), 116.642µs from start of poll, 484.735221ms from cluster start. Timeout at 2023-05-16 04:06:12.018406869 +1400 +14 m=+8.820988524, 106.004591ms after stability testing.go:149: cluster: 2023-05-16T04:06:12.018+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.912402318 +1400 +14 m=+8.714983933 (1 nodes), 116.642µs from start of poll, 484.735221ms from cluster start. Timeout at 2023-05-16 04:06:12.018406869 +1400 +14 m=+8.820988524, 106.004591ms after stability testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:12.018+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:12.018+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.020+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.020+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.020+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.020+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.021+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:12.021+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.021+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.021+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.021+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.021+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering follower state: follower="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Follower]" leader= testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering follower state: follower="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Follower]" leader= testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering candidate state: node="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Candidate]" term=2 testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering candidate state: node="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Candidate]" term=2 testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: election won: tally=2 testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: election won: tally=2 testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering leader state: leader="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Leader]" testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering leader state: leader="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Leader]" testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-83088d1a-625d-a43f-87c7-2e81e437ca2e testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-83088d1a-625d-a43f-87c7-2e81e437ca2e testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.083+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.083+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.188+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:12.083331775 +1400 +14 m=+8.885913390 (2 nodes), 62.175554ms from start of poll, 63.222106ms from cluster start. Timeout at 2023-05-16 04:06:12.188653078 +1400 +14 m=+8.991234713, 105.321323ms after stability testing.go:149: cluster: 2023-05-16T04:06:12.188+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:12.083331775 +1400 +14 m=+8.885913390 (2 nodes), 62.175554ms from start of poll, 63.222106ms from cluster start. Timeout at 2023-05-16 04:06:12.188653078 +1400 +14 m=+8.991234713, 105.321323ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.188+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:12.188+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.294+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.188907501 +1400 +14 m=+8.991489116 (1 nodes), 2.52µs from start of poll, 168.797832ms from cluster start. Timeout at 2023-05-16 04:06:12.293987221 +1400 +14 m=+9.096568956, 105.07984ms after stability testing.go:149: cluster: 2023-05-16T04:06:12.294+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.188907501 +1400 +14 m=+8.991489116 (1 nodes), 2.52µs from start of poll, 168.797832ms from cluster start. Timeout at 2023-05-16 04:06:12.293987221 +1400 +14 m=+9.096568956, 105.07984ms after stability testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: updating configuration: command=RemoveServer server-id=server-abb26b36-c69e-229d-3c5e-3e915fb40726 server-addr= servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: updating configuration: command=RemoveServer server-id=server-abb26b36-c69e-229d-3c5e-3e915fb40726 server-addr= servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: removed ourself, transitioning to follower testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: removed ourself, transitioning to follower testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.300+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.300+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.385+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: not part of stable configuration, aborting election testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.385+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: not part of stable configuration, aborting election testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.406+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:12.406+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=3 testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=3 testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.416+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: rejecting vote request since we have a leader: from=83088d1a-625d-a43f-87c7-2e81e437ca2e leader=abb26b36-c69e-229d-3c5e-3e915fb40726 testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.416+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: rejecting vote request since we have a leader: from=83088d1a-625d-a43f-87c7-2e81e437ca2e leader=abb26b36-c69e-229d-3c5e-3e915fb40726 testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering candidate state: node="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Candidate]" term=3 testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering candidate state: node="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Candidate]" term=3 testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.428+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: duplicate requestVote for same term: term=3 testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.428+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: Election timeout reached, restarting election testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: Election timeout reached, restarting election testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=4 testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=4 testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.512+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.512+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: election won: tally=2 testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: election won: tally=2 testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering leader state: leader="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Leader]" testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering leader state: leader="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Leader]" testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.513+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.513+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.516+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.516+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.620+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.515814319 +1400 +14 m=+9.318395954 (1 nodes), 109.635854ms from start of poll, 495.70467ms from cluster start. Timeout at 2023-05-16 04:06:12.620902039 +1400 +14 m=+9.423483694, 105.08774ms after stability testing.go:149: cluster: 2023-05-16T04:06:12.620+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.515814319 +1400 +14 m=+9.318395954 (1 nodes), 109.635854ms from start of poll, 495.70467ms from cluster start. Timeout at 2023-05-16 04:06:12.620902039 +1400 +14 m=+9.423483694, 105.08774ms after stability testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.726+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.726+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.71s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.728+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testing.go:149: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.728+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.728+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.728+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.728+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.728+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.728+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.728+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.729+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:12.729+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.729+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: entering follower state: follower="Node at baa415f7-9446-b5be-a195-a2bf823f3d5d [Follower]" leader= testing.go:149: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.729+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: entering follower state: follower="Node at baa415f7-9446-b5be-a195-a2bf823f3d5d [Follower]" leader= testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.729+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader= testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.729+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader= testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.729+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.729+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.729+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering follower state: follower="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Follower]" leader= testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.729+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering follower state: follower="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Follower]" leader= testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering candidate state: node="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Candidate]" term=2 testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering candidate state: node="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Candidate]" term=2 testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [WARN] server-f523c432-5146-60d1-40a1-c11499200c01: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [WARN] server-f523c432-5146-60d1-40a1-c11499200c01: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering candidate state: node="Node at f523c432-5146-60d1-40a1-c11499200c01 [Candidate]" term=2 testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering candidate state: node="Node at f523c432-5146-60d1-40a1-c11499200c01 [Candidate]" term=2 testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.793+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: duplicate requestVote for same term: term=2 testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.793+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: election won: tally=3 testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: election won: tally=3 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering leader state: leader="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Leader]" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering leader state: leader="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Leader]" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-baa415f7-9446-b5be-a195-a2bf823f3d5d testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-baa415f7-9446-b5be-a195-a2bf823f3d5d testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-f523c432-5146-60d1-40a1-c11499200c01 testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-f523c432-5146-60d1-40a1-c11499200c01 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: duplicate requestVote for same term: term=2 testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.794+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: duplicate requestVote for same term: term=2 testing.go:149: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.794+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=f523c432-5146-60d1-40a1-c11499200c01 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=f523c432-5146-60d1-40a1-c11499200c01 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.900+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.79474087 +1400 +14 m=+9.597322505 (1 nodes), 65.648874ms from start of poll, 66.904069ms from cluster start. Timeout at 2023-05-16 04:06:12.900398857 +1400 +14 m=+9.702980492, 105.657987ms after stability testing.go:149: cluster: 2023-05-16T04:06:12.900+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.79474087 +1400 +14 m=+9.597322505 (1 nodes), 65.648874ms from start of poll, 66.904069ms from cluster start. Timeout at 2023-05-16 04:06:12.900398857 +1400 +14 m=+9.702980492, 105.657987ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.006+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:13.006+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.112+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.006665751 +1400 +14 m=+9.809247386 (3 nodes), 3.62µs from start of poll, 278.82895ms from cluster start. Timeout at 2023-05-16 04:06:13.112160996 +1400 +14 m=+9.914742611, 105.495225ms after stability testing.go:149: cluster: 2023-05-16T04:06:13.112+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.006665751 +1400 +14 m=+9.809247386 (3 nodes), 3.62µs from start of poll, 278.82895ms from cluster start. Timeout at 2023-05-16 04:06:13.112160996 +1400 +14 m=+9.914742611, 105.495225ms after stability testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: updating configuration: command=RemoveServer server-id=server-7048faf4-9562-b685-9784-5d59b6d871f9 server-addr= servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379}]" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: updating configuration: command=RemoveServer server-id=server-7048faf4-9562-b685-9784-5d59b6d871f9 server-addr= servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379}]" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: removed peer, stopping replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 last-index=3 testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: removed peer, stopping replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 last-index=3 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="append pipeline closed" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="append pipeline closed" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="append pipeline closed" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="append pipeline closed" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.113+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.113+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.121+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.121+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.123+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.123+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.133+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.133+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.135+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.135+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.139+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.139+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.142+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.142+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.150+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.150+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.152+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.152+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.159+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.159+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=50.771519ms testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=50.771519ms testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [WARN] server-7048faf4-9562-b685-9784-5d59b6d871f9: heartbeat timeout reached, starting election: last-leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [WARN] server-7048faf4-9562-b685-9784-5d59b6d871f9: heartbeat timeout reached, starting election: last-leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering candidate state: node="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Candidate]" term=3 testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering candidate state: node="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Candidate]" term=3 testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.166+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=7048faf4-9562-b685-9784-5d59b6d871f9 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.166+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=7048faf4-9562-b685-9784-5d59b6d871f9 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=61.777109ms testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=61.777109ms testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-f523c432-5146-60d1-40a1-c11499200c01 time=59.391621ms testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-f523c432-5146-60d1-40a1-c11499200c01 time=59.391621ms testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact quorum of nodes, stepping down testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.172+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.172+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.173+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.173+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.173+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.173+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testLoggerAdapter verbose: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testing.go:149: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.174+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:13.174+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.174+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader= testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.174+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader= testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.174+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering follower state: follower="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Follower]" leader= testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.174+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering follower state: follower="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Follower]" leader= testLoggerAdapter verbose: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: entering follower state: follower="Node at fb311ca9-cff9-5558-1506-c706834f6a6b [Follower]" leader= testing.go:149: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: entering follower state: follower="Node at fb311ca9-cff9-5558-1506-c706834f6a6b [Follower]" leader= testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.255+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.255+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering candidate state: node="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Candidate]" term=2 testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering candidate state: node="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Candidate]" term=2 testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: election won: tally=2 testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: election won: tally=2 testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering leader state: leader="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Leader]" testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering leader state: leader="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Leader]" testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-369040ce-7c94-1215-9462-9dc9ac78065f testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-369040ce-7c94-1215-9462-9dc9ac78065f testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-fb311ca9-cff9-5558-1506-c706834f6a6b testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-fb311ca9-cff9-5558-1506-c706834f6a6b testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [WARN] server-369040ce-7c94-1215-9462-9dc9ac78065f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [WARN] server-369040ce-7c94-1215-9462-9dc9ac78065f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering candidate state: node="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Candidate]" term=2 testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering candidate state: node="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Candidate]" term=2 testLoggerAdapter verbose: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.257+1400 [WARN] server-fb311ca9-cff9-5558-1506-c706834f6a6b: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc testing.go:149: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.257+1400 [WARN] server-fb311ca9-cff9-5558-1506-c706834f6a6b: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: duplicate requestVote for same term: term=2 testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader=4a6c9f8d-2eae-7808-557e-996d16420ebc testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader=4a6c9f8d-2eae-7808-557e-996d16420ebc testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.363+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.257613513 +1400 +14 m=+10.060195128 (1 nodes), 83.324623ms from start of poll, 84.191414ms from cluster start. Timeout at 2023-05-16 04:06:13.363647564 +1400 +14 m=+10.166229199, 106.034071ms after stability testing.go:149: cluster: 2023-05-16T04:06:13.363+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.257613513 +1400 +14 m=+10.060195128 (1 nodes), 83.324623ms from start of poll, 84.191414ms from cluster start. Timeout at 2023-05-16 04:06:13.363647564 +1400 +14 m=+10.166229199, 106.034071ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.363+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:13.363+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.469+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.363977088 +1400 +14 m=+10.166558723 (2 nodes), 3.06µs from start of poll, 190.555009ms from cluster start. Timeout at 2023-05-16 04:06:13.469851978 +1400 +14 m=+10.272433713, 105.87499ms after stability testing.go:149: cluster: 2023-05-16T04:06:13.469+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.363977088 +1400 +14 m=+10.166558723 (2 nodes), 3.06µs from start of poll, 190.555009ms from cluster start. Timeout at 2023-05-16 04:06:13.469851978 +1400 +14 m=+10.272433713, 105.87499ms after stability testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: updating configuration: command=AddStaging server-id=server-369040ce-7c94-1215-9462-9dc9ac78065f server-addr=369040ce-7c94-1215-9462-9dc9ac78065f servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: updating configuration: command=AddStaging server-id=server-369040ce-7c94-1215-9462-9dc9ac78065f server-addr=369040ce-7c94-1215-9462-9dc9ac78065f servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" --- PASS: TestRaft_AddKnownPeer (0.30s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testing.go:149: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testLoggerAdapter verbose: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: entering follower state: follower="Node at ff128af1-a6a0-9f9b-6060-d13a4bd0a171 [Follower]" leader= testing.go:149: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: entering follower state: follower="Node at ff128af1-a6a0-9f9b-6060-d13a4bd0a171 [Follower]" leader= testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.472+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.472+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testLoggerAdapter verbose: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testing.go:149: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.473+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:13.473+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.473+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering follower state: follower="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Follower]" leader= testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.473+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering follower state: follower="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Follower]" leader= testLoggerAdapter verbose: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: entering follower state: follower="Node at f91e2e08-5cb3-532d-1da6-1712cfd5038c [Follower]" leader= testing.go:149: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: entering follower state: follower="Node at f91e2e08-5cb3-532d-1da6-1712cfd5038c [Follower]" leader= testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [WARN] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [WARN] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering candidate state: node="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Candidate]" term=2 testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering candidate state: node="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Candidate]" term=2 testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: election won: tally=2 testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: election won: tally=2 testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering leader state: leader="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Leader]" testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering leader state: leader="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Leader]" testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-f91e2e08-5cb3-532d-1da6-1712cfd5038c testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-f91e2e08-5cb3-532d-1da6-1712cfd5038c testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.641+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.535896917 +1400 +14 m=+10.338478552 (1 nodes), 62.075272ms from start of poll, 63.59989ms from cluster start. Timeout at 2023-05-16 04:06:13.640939277 +1400 +14 m=+10.443520912, 105.04236ms after stability testing.go:149: cluster: 2023-05-16T04:06:13.641+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.535896917 +1400 +14 m=+10.338478552 (1 nodes), 62.075272ms from start of poll, 63.59989ms from cluster start. Timeout at 2023-05-16 04:06:13.640939277 +1400 +14 m=+10.443520912, 105.04236ms after stability testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: updating configuration: command=RemoveServer server-id=3cb46c98-ecf6-9a7c-cda8-18a095bb4872 server-addr= servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: updating configuration: command=RemoveServer server-id=3cb46c98-ecf6-9a7c-cda8-18a095bb4872 server-addr= servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" --- PASS: TestRaft_RemoveUnknownPeer (0.17s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.643+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:13.643+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering follower state: follower="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Follower]" leader= testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering follower state: follower="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Follower]" leader= testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [WARN] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [WARN] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering candidate state: node="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Candidate]" term=2 testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering candidate state: node="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Candidate]" term=2 testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: election won: tally=1 testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: election won: tally=1 testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering leader state: leader="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Leader]" testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering leader state: leader="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Leader]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.811+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.705729721 +1400 +14 m=+10.508311356 (1 nodes), 62.422996ms from start of poll, 62.808341ms from cluster start. Timeout at 2023-05-16 04:06:13.811703352 +1400 +14 m=+10.614285007, 105.973651ms after stability testing.go:149: cluster: 2023-05-16T04:06:13.811+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.705729721 +1400 +14 m=+10.508311356 (1 nodes), 62.422996ms from start of poll, 62.808341ms from cluster start. Timeout at 2023-05-16 04:06:13.811703352 +1400 +14 m=+10.614285007, 105.973651ms after stability testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.813+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: starting snapshot up to: index=102 testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.813+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: starting snapshot up to: index=102 testing.go:153: 2023-05-16T04:06:13.813+1400 [INFO] creating new snapshot: path=/tmp/raft3765765489/snapshots/2-102-1684159573813.tmp testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: compacting logs: from=1 to=92 testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: snapshot complete up to: index=102 testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: snapshot complete up to: index=102 testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: restored from snapshot: id=2-102-1684159573813 testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: restored from snapshot: id=2-102-1684159573813 testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" --- PASS: TestRaft_SnapshotRestore (0.17s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.816+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.816+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.817+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:13.817+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.817+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering follower state: follower="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Follower]" leader= testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.817+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering follower state: follower="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Follower]" leader= testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.875+1400 [WARN] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.875+1400 [WARN] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering candidate state: node="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Candidate]" term=2 testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering candidate state: node="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Candidate]" term=2 testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: election won: tally=1 testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: election won: tally=1 testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering leader state: leader="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Leader]" testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering leader state: leader="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Leader]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.981+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.876395635 +1400 +14 m=+10.678977270 (1 nodes), 59.288419ms from start of poll, 59.710364ms from cluster start. Timeout at 2023-05-16 04:06:13.981718878 +1400 +14 m=+10.784300513, 105.323243ms after stability testing.go:149: cluster: 2023-05-16T04:06:13.981+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.876395635 +1400 +14 m=+10.678977270 (1 nodes), 59.288419ms from start of poll, 59.710364ms from cluster start. Timeout at 2023-05-16 04:06:13.981718878 +1400 +14 m=+10.784300513, 105.323243ms after stability testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.983+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: starting snapshot up to: index=102 testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.983+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: starting snapshot up to: index=102 testing.go:153: 2023-05-16T04:06:13.983+1400 [INFO] creating new snapshot: path=/tmp/raft2876506391/snapshots/2-102-1684159573983.tmp testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: compacting logs: from=1 to=92 testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: snapshot complete up to: index=102 testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: snapshot complete up to: index=102 testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.985+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.985+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" --- PASS: TestRaft_NoRestoreOnStart (0.17s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.985+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc1c8995-2994-6afa-6868-b168abdfc0aa Address:dc1c8995-2994-6afa-6868-b168abdfc0aa}]" testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.985+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc1c8995-2994-6afa-6868-b168abdfc0aa Address:dc1c8995-2994-6afa-6868-b168abdfc0aa}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.986+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:13.986+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.986+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering follower state: follower="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Follower]" leader= testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.986+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering follower state: follower="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Follower]" leader= testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.986+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering follower state: follower="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Follower]" leader= testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.051+1400 [WARN] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.051+1400 [WARN] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering candidate state: node="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Candidate]" term=2 testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering candidate state: node="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Candidate]" term=2 testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: election won: tally=1 testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: election won: tally=1 testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering leader state: leader="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Leader]" testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering leader state: leader="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Leader]" testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.081+1400 [WARN] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.082+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering candidate state: node="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Candidate]" term=3 testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.082+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: election won: tally=1 testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.082+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering leader state: leader="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Leader]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.158+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.052504874 +1400 +14 m=+10.855086509 (1 nodes), 66.409704ms from start of poll, 66.755148ms from cluster start. Timeout at 2023-05-16 04:06:14.15814252 +1400 +14 m=+10.960724175, 105.637666ms after stability testing.go:149: cluster: 2023-05-16T04:06:14.158+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.052504874 +1400 +14 m=+10.855086509 (1 nodes), 66.409704ms from start of poll, 66.755148ms from cluster start. Timeout at 2023-05-16 04:06:14.15814252 +1400 +14 m=+10.960724175, 105.637666ms after stability testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [ERROR] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [ERROR] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: starting snapshot up to: index=12 testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: starting snapshot up to: index=12 testing.go:153: 2023-05-16T04:06:14.158+1400 [INFO] creating new snapshot: path=/tmp/raft3468561679/snapshots/2-12-1684159574158.tmp testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: compacting logs: from=1 to=2 testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: snapshot complete up to: index=12 testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.17s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.161+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.161+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.161+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.161+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.161+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:14.161+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering follower state: follower="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Follower]" leader= testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering follower state: follower="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Follower]" leader= testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.162+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: entering follower state: follower="Node at f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 [Follower]" leader= testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.162+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: entering follower state: follower="Node at f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 [Follower]" leader= testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.162+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: entering follower state: follower="Node at 594c8d6b-ac40-b588-ec84-4542ea29e002 [Follower]" leader= testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.162+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: entering follower state: follower="Node at 594c8d6b-ac40-b588-ec84-4542ea29e002 [Follower]" leader= testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [WARN] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [WARN] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering candidate state: node="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Candidate]" term=2 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering candidate state: node="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Candidate]" term=2 testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: election won: tally=2 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: election won: tally=2 testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering leader state: leader="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Leader]" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering leader state: leader="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Leader]" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-594c8d6b-ac40-b588-ec84-4542ea29e002 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-594c8d6b-ac40-b588-ec84-4542ea29e002 testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.323+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.217656603 +1400 +14 m=+11.020238218 (1 nodes), 55.752998ms from start of poll, 56.649369ms from cluster start. Timeout at 2023-05-16 04:06:14.322969866 +1400 +14 m=+11.125551501, 105.313283ms after stability testing.go:149: cluster: 2023-05-16T04:06:14.323+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.217656603 +1400 +14 m=+11.020238218 (1 nodes), 55.752998ms from start of poll, 56.649369ms from cluster start. Timeout at 2023-05-16 04:06:14.322969866 +1400 +14 m=+11.125551501, 105.313283ms after stability testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.324+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: starting snapshot up to: index=12 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.324+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: starting snapshot up to: index=12 testing.go:153: 2023-05-16T04:06:14.324+1400 [INFO] creating new snapshot: path=/tmp/raft1344906792/snapshots/2-12-1684159574324.tmp testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.325+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: snapshot complete up to: index=12 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.325+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: snapshot complete up to: index=12 testing.go:153: 2023-05-16T04:06:14.326+1400 [INFO] creating new snapshot: path=/tmp/raft1344906792/snapshots/2-23-1684159574326.tmp testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.326+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: copied to local snapshot: bytes=71 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.326+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: restored user snapshot: index=1 testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: restored user snapshot: index=1 testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.333+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.333+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:14.334+1400 [INFO] creating new snapshot: path=/tmp/raft3505364732/snapshots/2-23-1684159574334.tmp testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: copied to local snapshot: bytes=71 testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: Installed remote snapshot testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: Installed remote snapshot testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [WARN] server-594c8d6b-ac40-b588-ec84-4542ea29e002: skipping application of old log: index=22 testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [WARN] server-594c8d6b-ac40-b588-ec84-4542ea29e002: skipping application of old log: index=22 testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:14.336+1400 [INFO] creating new snapshot: path=/tmp/raft3368977623/snapshots/2-23-1684159574336.tmp testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.338+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: copied to local snapshot: bytes=71 testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.338+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.341+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: Installed remote snapshot testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.341+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: Installed remote snapshot testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.342+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.342+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.361+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.361+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.361+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.361+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.361+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.361+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.361+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:14.361+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.362+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: entering follower state: follower="Node at 472e3a07-d252-55b6-b13c-14eee42fc623 [Follower]" leader= testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.362+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: entering follower state: follower="Node at 472e3a07-d252-55b6-b13c-14eee42fc623 [Follower]" leader= testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.362+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: entering follower state: follower="Node at 8b64f623-830c-0393-f155-d64c94c92e98 [Follower]" leader= testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.362+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering follower state: follower="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Follower]" leader= testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.362+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: entering follower state: follower="Node at 8b64f623-830c-0393-f155-d64c94c92e98 [Follower]" leader= testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.362+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering follower state: follower="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Follower]" leader= testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [WARN] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [WARN] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering candidate state: node="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Candidate]" term=2 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering candidate state: node="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Candidate]" term=2 testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: election won: tally=2 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: election won: tally=2 testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering leader state: leader="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Leader]" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering leader state: leader="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Leader]" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-472e3a07-d252-55b6-b13c-14eee42fc623 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-472e3a07-d252-55b6-b13c-14eee42fc623 testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-8b64f623-830c-0393-f155-d64c94c92e98 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-8b64f623-830c-0393-f155-d64c94c92e98 testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.420+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.420+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.420445716 +1400 +14 m=+11.223027351 (1 nodes), 58.391689ms from start of poll, 59.498602ms from cluster start. Timeout at 2023-05-16 04:06:14.525960801 +1400 +14 m=+11.328542776, 105.515425ms after stability testing.go:149: cluster: 2023-05-16T04:06:14.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.420445716 +1400 +14 m=+11.223027351 (1 nodes), 58.391689ms from start of poll, 59.498602ms from cluster start. Timeout at 2023-05-16 04:06:14.525960801 +1400 +14 m=+11.328542776, 105.515425ms after stability testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.526+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: starting snapshot up to: index=12 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.526+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: starting snapshot up to: index=12 testing.go:153: 2023-05-16T04:06:14.527+1400 [INFO] creating new snapshot: path=/tmp/raft1633987355/snapshots/2-12-1684159574527.tmp testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.528+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: snapshot complete up to: index=12 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.528+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: snapshot complete up to: index=12 testing.go:153: 2023-05-16T04:06:14.529+1400 [INFO] creating new snapshot: path=/tmp/raft1633987355/snapshots/2-23-1684159574529.tmp testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.529+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: copied to local snapshot: bytes=71 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.529+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: restored user snapshot: index=1 testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: restored user snapshot: index=1 testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.537+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.537+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.538+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.538+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:14.538+1400 [INFO] creating new snapshot: path=/tmp/raft685787767/snapshots/2-23-1684159574538.tmp testing.go:153: 2023-05-16T04:06:14.538+1400 [INFO] creating new snapshot: path=/tmp/raft1580458597/snapshots/2-23-1684159574538.tmp testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: copied to local snapshot: bytes=71 testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: Installed remote snapshot testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: Installed remote snapshot testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: copied to local snapshot: bytes=71 testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [WARN] server-472e3a07-d252-55b6-b13c-14eee42fc623: skipping application of old log: index=22 testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [WARN] server-472e3a07-d252-55b6-b13c-14eee42fc623: skipping application of old log: index=22 testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.539+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.539+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: Installed remote snapshot testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: Installed remote snapshot testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.540+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.540+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.557+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.557+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.558+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.558+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.559+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.559+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.560+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:14.560+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.560+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering follower state: follower="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Follower]" leader= testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.560+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering follower state: follower="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Follower]" leader= testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: entering follower state: follower="Node at f477a1fe-85b0-ac5c-0300-95e12f7640e0 [Follower]" leader= testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: entering follower state: follower="Node at f477a1fe-85b0-ac5c-0300-95e12f7640e0 [Follower]" leader= testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: entering follower state: follower="Node at 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 [Follower]" leader= testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: entering follower state: follower="Node at 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 [Follower]" leader= testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [WARN] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [WARN] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering candidate state: node="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Candidate]" term=2 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering candidate state: node="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Candidate]" term=2 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: election won: tally=2 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: election won: tally=2 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering leader state: leader="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Leader]" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering leader state: leader="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Leader]" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.614+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.614+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.719+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.614204503 +1400 +14 m=+11.416786138 (1 nodes), 53.635153ms from start of poll, 54.593725ms from cluster start. Timeout at 2023-05-16 04:06:14.719279883 +1400 +14 m=+11.521861518, 105.07538ms after stability testing.go:149: cluster: 2023-05-16T04:06:14.719+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.614204503 +1400 +14 m=+11.416786138 (1 nodes), 53.635153ms from start of poll, 54.593725ms from cluster start. Timeout at 2023-05-16 04:06:14.719279883 +1400 +14 m=+11.521861518, 105.07538ms after stability testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.720+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: starting snapshot up to: index=12 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.720+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: starting snapshot up to: index=12 testing.go:153: 2023-05-16T04:06:14.720+1400 [INFO] creating new snapshot: path=/tmp/raft2598925801/snapshots/2-12-1684159574720.tmp testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.721+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: snapshot complete up to: index=12 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.721+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: snapshot complete up to: index=12 testing.go:153: 2023-05-16T04:06:14.722+1400 [INFO] creating new snapshot: path=/tmp/raft2598925801/snapshots/2-23-1684159574722.tmp testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: copied to local snapshot: bytes=71 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: restored user snapshot: index=1 testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: restored user snapshot: index=1 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.733+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.733+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:14.734+1400 [INFO] creating new snapshot: path=/tmp/raft1270729232/snapshots/2-23-1684159574734.tmp testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.735+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: copied to local snapshot: bytes=71 testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.735+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.735+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.735+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:14.735+1400 [INFO] creating new snapshot: path=/tmp/raft4082407490/snapshots/2-23-1684159574735.tmp testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.736+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: copied to local snapshot: bytes=71 testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.736+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: Installed remote snapshot testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: Installed remote snapshot testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [WARN] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: skipping application of old log: index=22 testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [WARN] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: skipping application of old log: index=22 testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.740+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.740+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.742+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: Installed remote snapshot testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.742+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: Installed remote snapshot testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.742+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.742+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.755+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:14.755+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering follower state: follower="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Follower]" leader= testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering follower state: follower="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Follower]" leader= testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: entering follower state: follower="Node at 3dea05ed-8854-a0de-4b55-250f9c5e0d6c [Follower]" leader= testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: entering follower state: follower="Node at 3dea05ed-8854-a0de-4b55-250f9c5e0d6c [Follower]" leader= testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: entering follower state: follower="Node at 0cb1df11-83a6-892b-b1d6-679df7901410 [Follower]" leader= testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: entering follower state: follower="Node at 0cb1df11-83a6-892b-b1d6-679df7901410 [Follower]" leader= testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.808+1400 [WARN] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.808+1400 [WARN] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering candidate state: node="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Candidate]" term=2 testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering candidate state: node="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Candidate]" term=2 testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: election won: tally=2 testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: election won: tally=2 testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering leader state: leader="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Leader]" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering leader state: leader="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Leader]" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-0cb1df11-83a6-892b-b1d6-679df7901410 testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-0cb1df11-83a6-892b-b1d6-679df7901410 testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.915+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.810114095 +1400 +14 m=+11.612695730 (1 nodes), 54.456203ms from start of poll, 55.295393ms from cluster start. Timeout at 2023-05-16 04:06:14.915266715 +1400 +14 m=+11.717848350, 105.15262ms after stability testing.go:149: cluster: 2023-05-16T04:06:14.915+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.810114095 +1400 +14 m=+11.612695730 (1 nodes), 54.456203ms from start of poll, 55.295393ms from cluster start. Timeout at 2023-05-16 04:06:14.915266715 +1400 +14 m=+11.717848350, 105.15262ms after stability testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.916+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: starting snapshot up to: index=12 testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.916+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: starting snapshot up to: index=12 testing.go:153: 2023-05-16T04:06:14.916+1400 [INFO] creating new snapshot: path=/tmp/raft3176389591/snapshots/2-12-1684159574916.tmp testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.917+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: snapshot complete up to: index=12 testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.917+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: snapshot complete up to: index=12 testing.go:153: 2023-05-16T04:06:14.918+1400 [INFO] creating new snapshot: path=/tmp/raft3176389591/snapshots/2-113-1684159574918.tmp testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.918+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: copied to local snapshot: bytes=71 testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.918+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: restored user snapshot: index=1 testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: restored user snapshot: index=1 testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.926+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.926+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:14.927+1400 [INFO] creating new snapshot: path=/tmp/raft670186945/snapshots/2-113-1684159574927.tmp testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.927+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: copied to local snapshot: bytes=71 testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.927+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: Installed remote snapshot testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: Installed remote snapshot testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [WARN] server-0cb1df11-83a6-892b-b1d6-679df7901410: skipping application of old log: index=22 testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [WARN] server-0cb1df11-83a6-892b-b1d6-679df7901410: skipping application of old log: index=22 testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.928+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.928+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.929+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.929+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:14.930+1400 [INFO] creating new snapshot: path=/tmp/raft2096542586/snapshots/2-113-1684159574930.tmp testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: copied to local snapshot: bytes=71 testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: Installed remote snapshot testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: Installed remote snapshot testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.931+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.931+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.948+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.948+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.949+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:14.949+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.949+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: entering follower state: follower="Node at a6f3b0b9-5799-c956-00ee-89e057d28452 [Follower]" leader= testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.949+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: entering follower state: follower="Node at a6f3b0b9-5799-c956-00ee-89e057d28452 [Follower]" leader= testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering follower state: follower="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Follower]" leader= testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering follower state: follower="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Follower]" leader= testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: entering follower state: follower="Node at e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f [Follower]" leader= testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: entering follower state: follower="Node at e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f [Follower]" leader= testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [WARN] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [WARN] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering candidate state: node="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Candidate]" term=2 testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering candidate state: node="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Candidate]" term=2 testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: election won: tally=2 testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: election won: tally=2 testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering leader state: leader="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Leader]" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering leader state: leader="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Leader]" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-a6f3b0b9-5799-c956-00ee-89e057d28452 testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-a6f3b0b9-5799-c956-00ee-89e057d28452 testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.007+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.007+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.112+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.007343222 +1400 +14 m=+11.809924837 (1 nodes), 57.850923ms from start of poll, 58.828354ms from cluster start. Timeout at 2023-05-16 04:06:15.112641365 +1400 +14 m=+11.915223020, 105.298183ms after stability testing.go:149: cluster: 2023-05-16T04:06:15.112+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.007343222 +1400 +14 m=+11.809924837 (1 nodes), 57.850923ms from start of poll, 58.828354ms from cluster start. Timeout at 2023-05-16 04:06:15.112641365 +1400 +14 m=+11.915223020, 105.298183ms after stability testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.113+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: starting snapshot up to: index=12 testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.113+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: starting snapshot up to: index=12 testing.go:153: 2023-05-16T04:06:15.113+1400 [INFO] creating new snapshot: path=/tmp/raft4245320530/snapshots/2-12-1684159575113.tmp testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.114+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: snapshot complete up to: index=12 testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.114+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: snapshot complete up to: index=12 testing.go:153: 2023-05-16T04:06:15.115+1400 [INFO] creating new snapshot: path=/tmp/raft4245320530/snapshots/2-1013-1684159575115.tmp testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.116+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: copied to local snapshot: bytes=71 testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.116+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: restored user snapshot: index=1 testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: restored user snapshot: index=1 testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.124+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.124+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:15.124+1400 [INFO] creating new snapshot: path=/tmp/raft4044288030/snapshots/2-1013-1684159575124.tmp testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.125+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: copied to local snapshot: bytes=71 testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.125+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: Installed remote snapshot testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: Installed remote snapshot testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [WARN] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: skipping application of old log: index=22 testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [WARN] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: skipping application of old log: index=22 testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:15.127+1400 [INFO] creating new snapshot: path=/tmp/raft549836052/snapshots/2-1013-1684159575127.tmp testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.127+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: copied to local snapshot: bytes=71 testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.127+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.128+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: Installed remote snapshot testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.128+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: Installed remote snapshot testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.128+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.128+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.146+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.146+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.146+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.146+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.146+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.146+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.147+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:15.147+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.147+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: entering follower state: follower="Node at e53c871b-99d0-0c11-33a7-7f1952c12ec0 [Follower]" leader= testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.147+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: entering follower state: follower="Node at e53c871b-99d0-0c11-33a7-7f1952c12ec0 [Follower]" leader= testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.147+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering follower state: follower="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Follower]" leader= testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.147+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering follower state: follower="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Follower]" leader= testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.147+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: entering follower state: follower="Node at c539e53a-1c75-cfe2-70eb-898edcf8d51c [Follower]" leader= testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.147+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: entering follower state: follower="Node at c539e53a-1c75-cfe2-70eb-898edcf8d51c [Follower]" leader= testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [WARN] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [WARN] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering candidate state: node="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Candidate]" term=2 testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering candidate state: node="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Candidate]" term=2 testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: election won: tally=2 testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: election won: tally=2 testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering leader state: leader="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Leader]" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering leader state: leader="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Leader]" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-c539e53a-1c75-cfe2-70eb-898edcf8d51c testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-c539e53a-1c75-cfe2-70eb-898edcf8d51c testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.210+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.210+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.316+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.210383358 +1400 +14 m=+12.012964993 (1 nodes), 63.304507ms from start of poll, 64.284438ms from cluster start. Timeout at 2023-05-16 04:06:15.315950844 +1400 +14 m=+12.118532479, 105.567486ms after stability testing.go:149: cluster: 2023-05-16T04:06:15.316+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.210383358 +1400 +14 m=+12.012964993 (1 nodes), 63.304507ms from start of poll, 64.284438ms from cluster start. Timeout at 2023-05-16 04:06:15.315950844 +1400 +14 m=+12.118532479, 105.567486ms after stability testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.316+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: starting snapshot up to: index=12 testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.316+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: starting snapshot up to: index=12 testing.go:153: 2023-05-16T04:06:15.317+1400 [INFO] creating new snapshot: path=/tmp/raft198941943/snapshots/2-12-1684159575317.tmp testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.317+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: snapshot complete up to: index=12 testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.317+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: snapshot complete up to: index=12 testing.go:153: 2023-05-16T04:06:15.318+1400 [INFO] creating new snapshot: path=/tmp/raft198941943/snapshots/2-10013-1684159575318.tmp testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.320+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: copied to local snapshot: bytes=71 testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.320+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: restored user snapshot: index=1 testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: restored user snapshot: index=1 testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.331+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.331+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testing.go:153: 2023-05-16T04:06:15.332+1400 [INFO] creating new snapshot: path=/tmp/raft2486864240/snapshots/2-10013-1684159575332.tmp testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: copied to local snapshot: bytes=71 testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: Installed remote snapshot testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: Installed remote snapshot testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [WARN] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: skipping application of old log: index=22 testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [WARN] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: skipping application of old log: index=22 testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" testing.go:153: 2023-05-16T04:06:15.334+1400 [INFO] creating new snapshot: path=/tmp/raft688072939/snapshots/2-10013-1684159575334.tmp testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.334+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: copied to local snapshot: bytes=71 testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.334+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.335+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: Installed remote snapshot testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.335+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: Installed remote snapshot testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.335+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.335+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" --- PASS: TestRaft_UserRestore (1.19s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.354+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.354+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.354+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.354+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.354+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.354+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.355+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:15.355+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.355+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: entering follower state: follower="Node at 03622733-cd5d-ccfb-64e0-3a8c894869be [Follower]" leader= testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.355+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: entering follower state: follower="Node at 03622733-cd5d-ccfb-64e0-3a8c894869be [Follower]" leader= testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.355+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering follower state: follower="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Follower]" leader= testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.355+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering follower state: follower="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Follower]" leader= testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.355+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: entering follower state: follower="Node at e7cba1d4-896d-0553-c5ee-41509e938548 [Follower]" leader= testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.355+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: entering follower state: follower="Node at e7cba1d4-896d-0553-c5ee-41509e938548 [Follower]" leader= testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [WARN] server-1ed42e07-fa6f-872e-16cd-268f263532c0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [WARN] server-1ed42e07-fa6f-872e-16cd-268f263532c0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering candidate state: node="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Candidate]" term=2 testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering candidate state: node="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Candidate]" term=2 testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: election won: tally=2 testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: election won: tally=2 testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering leader state: leader="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Leader]" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering leader state: leader="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Leader]" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-03622733-cd5d-ccfb-64e0-3a8c894869be testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-03622733-cd5d-ccfb-64e0-3a8c894869be testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-e7cba1d4-896d-0553-c5ee-41509e938548 testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-e7cba1d4-896d-0553-c5ee-41509e938548 testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.522+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.416699793 +1400 +14 m=+12.219281408 (2 nodes), 61.577767ms from start of poll, 62.463157ms from cluster start. Timeout at 2023-05-16 04:06:15.5224056 +1400 +14 m=+12.324987235, 105.705827ms after stability testing.go:149: cluster: 2023-05-16T04:06:15.522+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.416699793 +1400 +14 m=+12.219281408 (2 nodes), 61.577767ms from start of poll, 62.463157ms from cluster start. Timeout at 2023-05-16 04:06:15.5224056 +1400 +14 m=+12.324987235, 105.705827ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.522+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:15.522+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.628+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.522731144 +1400 +14 m=+12.325312779 (1 nodes), 2.82µs from start of poll, 168.494528ms from cluster start. Timeout at 2023-05-16 04:06:15.628689015 +1400 +14 m=+12.431270670, 105.957891ms after stability testing.go:149: cluster: 2023-05-16T04:06:15.628+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.522731144 +1400 +14 m=+12.325312779 (1 nodes), 2.82µs from start of poll, 168.494528ms from cluster start. Timeout at 2023-05-16 04:06:15.628689015 +1400 +14 m=+12.431270670, 105.957891ms after stability testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to heartbeat to: peer=03622733-cd5d-ccfb-64e0-3a8c894869be error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to heartbeat to: peer=03622733-cd5d-ccfb-64e0-3a8c894869be error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to pipeline appendEntries: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="append pipeline closed" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to pipeline appendEntries: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="append pipeline closed" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to appendEntries to: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to appendEntries to: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.631+1400 [ERROR] server-03622733-cd5d-ccfb-64e0-3a8c894869be: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.631+1400 [ERROR] server-03622733-cd5d-ccfb-64e0-3a8c894869be: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.631+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: starting snapshot up to: index=102 testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.631+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: starting snapshot up to: index=102 testing.go:153: 2023-05-16T04:06:15.631+1400 [INFO] creating new snapshot: path=/tmp/raft4236312094/snapshots/2-102-1684159575631.tmp testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: compacting logs: from=1 to=92 testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: snapshot complete up to: index=102 testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: snapshot complete up to: index=102 testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.632+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: starting snapshot up to: index=30 testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.632+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: starting snapshot up to: index=30 testing.go:153: 2023-05-16T04:06:15.632+1400 [INFO] creating new snapshot: path=/tmp/raft720526378/snapshots/2-30-1684159575632.tmp testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: compacting logs: from=1 to=30 testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: compacting logs: from=1 to=30 testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: snapshot complete up to: index=30 testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: snapshot complete up to: index=30 testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.639+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to get log: index=2 error="log not found" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.639+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to get log: index=2 error="log not found" testing.go:153: 2023-05-16T04:06:15.640+1400 [INFO] creating new snapshot: path=/tmp/raft1387116560/snapshots/2-102-1684159575640.tmp testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: copied to local snapshot: bytes=693 testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: Installed remote snapshot testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: Installed remote snapshot testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [WARN] server-03622733-cd5d-ccfb-64e0-3a8c894869be: skipping application of old log: index=102 testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [WARN] server-03622733-cd5d-ccfb-64e0-3a8c894869be: skipping application of old log: index=102 testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.641+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.641+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" --- PASS: TestRaft_SendSnapshotFollower (0.29s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.648+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.648+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.648+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.648+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.649+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:15.649+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.649+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: entering follower state: follower="Node at bcae9302-e6da-e3b6-97fa-5a309038ac87 [Follower]" leader= testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.649+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: entering follower state: follower="Node at bcae9302-e6da-e3b6-97fa-5a309038ac87 [Follower]" leader= testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.649+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: entering follower state: follower="Node at 6b3433b5-5490-c2d5-6dad-8df73d1d7285 [Follower]" leader= testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.649+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: entering follower state: follower="Node at 6b3433b5-5490-c2d5-6dad-8df73d1d7285 [Follower]" leader= testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering follower state: follower="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Follower]" leader= testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering follower state: follower="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Follower]" leader= testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.709+1400 [WARN] server-95aabfe7-fa96-a83a-9d30-011608429910: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.709+1400 [WARN] server-95aabfe7-fa96-a83a-9d30-011608429910: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering candidate state: node="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Candidate]" term=2 testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering candidate state: node="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Candidate]" term=2 testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: election won: tally=2 testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: election won: tally=2 testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering leader state: leader="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Leader]" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering leader state: leader="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Leader]" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-bcae9302-e6da-e3b6-97fa-5a309038ac87 testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-bcae9302-e6da-e3b6-97fa-5a309038ac87 testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.817+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.711053027 +1400 +14 m=+12.513634642 (2 nodes), 61.85243ms from start of poll, 62.786761ms from cluster start. Timeout at 2023-05-16 04:06:15.816984217 +1400 +14 m=+12.619565852, 105.93121ms after stability testing.go:149: cluster: 2023-05-16T04:06:15.817+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.711053027 +1400 +14 m=+12.513634642 (2 nodes), 61.85243ms from start of poll, 62.786761ms from cluster start. Timeout at 2023-05-16 04:06:15.816984217 +1400 +14 m=+12.619565852, 105.93121ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.817+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:15.817+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.923+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.81723556 +1400 +14 m=+12.619817175 (1 nodes), 3.24µs from start of poll, 168.969294ms from cluster start. Timeout at 2023-05-16 04:06:15.923215331 +1400 +14 m=+12.725796966, 105.979791ms after stability testing.go:149: cluster: 2023-05-16T04:06:15.923+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.81723556 +1400 +14 m=+12.619817175 (1 nodes), 3.24µs from start of poll, 168.969294ms from cluster start. Timeout at 2023-05-16 04:06:15.923215331 +1400 +14 m=+12.725796966, 105.979791ms after stability testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to pipeline appendEntries: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="append pipeline closed" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to pipeline appendEntries: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="append pipeline closed" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to appendEntries to: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to appendEntries to: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.925+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=bcae9302-e6da-e3b6-97fa-5a309038ac87 error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.925+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=bcae9302-e6da-e3b6-97fa-5a309038ac87 error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.927+1400 [ERROR] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.927+1400 [ERROR] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.927+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: starting snapshot up to: index=101 testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.927+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: starting snapshot up to: index=101 testing.go:153: 2023-05-16T04:06:15.927+1400 [INFO] creating new snapshot: path=/tmp/raft303909923/snapshots/2-101-1684159575927.tmp testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: compacting logs: from=1 to=92 testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: snapshot complete up to: index=101 testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: snapshot complete up to: index=101 testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: starting snapshot up to: index=102 testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: starting snapshot up to: index=102 testing.go:153: 2023-05-16T04:06:15.929+1400 [INFO] creating new snapshot: path=/tmp/raft419147949/snapshots/2-102-1684159575929.tmp testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: compacting logs: from=1 to=92 testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: snapshot complete up to: index=102 testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: snapshot complete up to: index=102 raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.934+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to get log: index=2 error="log not found" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.934+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to get log: index=2 error="log not found" testing.go:153: 2023-05-16T04:06:15.938+1400 [INFO] creating new snapshot: path=/tmp/raft4039688190/snapshots/2-102-1684159575938.tmp testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.939+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: copied to local snapshot: bytes=693 testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.939+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.940+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: Installed remote snapshot testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.940+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: Installed remote snapshot testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.941+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.941+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:16.000+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=6b3433b5-5490-c2d5-6dad-8df73d1d7285 error="command timed out" testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:16.000+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=6b3433b5-5490-c2d5-6dad-8df73d1d7285 error="command timed out" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.35s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.002+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.002+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.002+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.002+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.002+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:16.002+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.003+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: entering follower state: follower="Node at 3018a7db-32dc-5433-aac3-a047171aacd6 [Follower]" leader= testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.003+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: entering follower state: follower="Node at 3018a7db-32dc-5433-aac3-a047171aacd6 [Follower]" leader= testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.003+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.003+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.057+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.057+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=2 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=2 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.058+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.058+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=2 testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=2 testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: duplicate requestVote for same term: term=2 testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.059+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.059+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.165+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.059768842 +1400 +14 m=+12.862350457 (1 nodes), 56.825271ms from start of poll, 57.867983ms from cluster start. Timeout at 2023-05-16 04:06:16.165852534 +1400 +14 m=+12.968434189, 106.083732ms after stability testing.go:149: cluster: 2023-05-16T04:06:16.165+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.059768842 +1400 +14 m=+12.862350457 (1 nodes), 56.825271ms from start of poll, 57.867983ms from cluster start. Timeout at 2023-05-16 04:06:16.165852534 +1400 +14 m=+12.968434189, 106.083732ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.171+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:16.171+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.277+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.171784324 +1400 +14 m=+12.974365959 (2 nodes), 4.58µs from start of poll, 169.883485ms from cluster start. Timeout at 2023-05-16 04:06:16.277510992 +1400 +14 m=+13.080092627, 105.726668ms after stability testing.go:149: cluster: 2023-05-16T04:06:16.277+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.171784324 +1400 +14 m=+12.974365959 (2 nodes), 4.58µs from start of poll, 169.883485ms from cluster start. Timeout at 2023-05-16 04:06:16.277510992 +1400 +14 m=+13.080092627, 105.726668ms after stability testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=RemoveServer server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr= servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=RemoveServer server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr= servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: removed peer, stopping replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=3 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: removed peer, stopping replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=3 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.278+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.278+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.340+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.340+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.341+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=3 testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.341+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=3 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.341+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.341+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.341+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.341+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=AddStaging server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr=7e074892-d43a-9b83-058c-4c6dc15f2751 servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7} {Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751}]" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=AddStaging server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr=7e074892-d43a-9b83-058c-4c6dc15f2751 servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7} {Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751}]" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [ERROR] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: peer has newer term, stopping replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [ERROR] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: peer has newer term, stopping replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.385+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:16.385+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: Election timeout reached, restarting election testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: Election timeout reached, restarting election testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=4 testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=4 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.441+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since our last index is greater: candidate=7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=4 last-candidate-index=3 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.441+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since our last index is greater: candidate=7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=4 last-candidate-index=3 testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.441+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.441+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=5 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=5 testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.455+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.455+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.456+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.456+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: appendEntries rejected, sending older logs: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" next=4 testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: appendEntries rejected, sending older logs: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" next=4 testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.562+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.456772007 +1400 +14 m=+13.259353642 (1 nodes), 71.589925ms from start of poll, 454.871168ms from cluster start. Timeout at 2023-05-16 04:06:16.562175091 +1400 +14 m=+13.364756706, 105.403064ms after stability testing.go:149: cluster: 2023-05-16T04:06:16.562+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.456772007 +1400 +14 m=+13.259353642 (1 nodes), 71.589925ms from start of poll, 454.871168ms from cluster start. Timeout at 2023-05-16 04:06:16.562175091 +1400 +14 m=+13.364756706, 105.403064ms after stability testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" --- PASS: TestRaft_ReJoinFollower (0.56s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.563+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.563+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.564+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:16.564+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.564+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.564+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.623+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=2 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.623+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=2 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=2 testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=2 testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.624+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: duplicate requestVote for same term: term=2 testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.624+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: duplicate requestVote for same term: term=2 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.676+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.676+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=3 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=3 testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.677+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.677+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: election won: tally=2 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: election won: tally=2 testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering leader state: leader="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Leader]" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering leader state: leader="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Leader]" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: added peer, starting replication: peer=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: added peer, starting replication: peer=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: pipelining replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: pipelining replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.783+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.677785455 +1400 +14 m=+13.480367070 (1 nodes), 113.373698ms from start of poll, 114.065046ms from cluster start. Timeout at 2023-05-16 04:06:16.783698645 +1400 +14 m=+13.586280320, 105.91325ms after stability testing.go:149: cluster: 2023-05-16T04:06:16.783+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.677785455 +1400 +14 m=+13.480367070 (1 nodes), 113.373698ms from start of poll, 114.065046ms from cluster start. Timeout at 2023-05-16 04:06:16.783698645 +1400 +14 m=+13.586280320, 105.91325ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.789+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:16.789+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.895+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.789630715 +1400 +14 m=+13.592212330 (1 nodes), 1.94µs from start of poll, 225.910306ms from cluster start. Timeout at 2023-05-16 04:06:16.895638966 +1400 +14 m=+13.698220641, 106.008311ms after stability testing.go:149: cluster: 2023-05-16T04:06:16.895+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.789630715 +1400 +14 m=+13.592212330 (1 nodes), 1.94µs from start of poll, 225.910306ms from cluster start. Timeout at 2023-05-16 04:06:16.895638966 +1400 +14 m=+13.698220641, 106.008311ms after stability raft_test.go:1492: [INFO] Disconnecting Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower] testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to pipeline appendEntries: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="append pipeline closed" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to pipeline appendEntries: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="append pipeline closed" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: aborting pipeline replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: aborting pipeline replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.901+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.901+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.906+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.906+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.921+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.921+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.926+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.926+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.939+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.939+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact: server-id=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 time=51.267705ms testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact: server-id=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 time=51.267705ms testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact quorum of nodes, stepping down testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.944+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:16.944+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader=e56a0d25-53a3-dde4-b8d8-e63211ee6e5d testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader=e56a0d25-53a3-dde4-b8d8-e63211ee6e5d testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=4 testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=4 testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=4 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=4 testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=5 testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=5 testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=5 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=5 testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.077+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.077+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=6 testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=6 testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.110+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.004736014 +1400 +14 m=+13.807317649 (0 nodes), 59.848267ms from start of poll, 441.015625ms from cluster start. Timeout at 2023-05-16 04:06:17.110044056 +1400 +14 m=+13.912625751, 105.308102ms after stability testing.go:149: cluster: 2023-05-16T04:06:17.110+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.004736014 +1400 +14 m=+13.807317649 (0 nodes), 59.848267ms from start of poll, 441.015625ms from cluster start. Timeout at 2023-05-16 04:06:17.110044056 +1400 +14 m=+13.912625751, 105.308102ms after stability testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=6 testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=6 testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=7 testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=7 testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" --- PASS: TestRaft_LeaderLeaseExpire (0.65s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.217+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" testing.go:149: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.217+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" testLoggerAdapter verbose: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" testing.go:149: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.218+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:17.218+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.218+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: entering follower state: follower="Node at 2741c739-3eb5-496c-763f-89d501309423 [Follower]" leader= testing.go:149: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.218+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: entering follower state: follower="Node at 2741c739-3eb5-496c-763f-89d501309423 [Follower]" leader= testLoggerAdapter verbose: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: entering follower state: follower="Node at 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 [Follower]" leader= testing.go:149: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: entering follower state: follower="Node at 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 [Follower]" leader= testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering follower state: follower="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Follower]" leader= testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering follower state: follower="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Follower]" leader= testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [WARN] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [WARN] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering candidate state: node="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Candidate]" term=2 testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering candidate state: node="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Candidate]" term=2 testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: election won: tally=2 testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: election won: tally=2 testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering leader state: leader="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Leader]" testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering leader state: leader="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Leader]" testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-2741c739-3eb5-496c-763f-89d501309423 testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-2741c739-3eb5-496c-763f-89d501309423 testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.399+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.294569114 +1400 +14 m=+14.097150769 (1 nodes), 76.045137ms from start of poll, 76.993669ms from cluster start. Timeout at 2023-05-16 04:06:17.399828396 +1400 +14 m=+14.202410031, 105.259262ms after stability testing.go:149: cluster: 2023-05-16T04:06:17.399+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.294569114 +1400 +14 m=+14.097150769 (1 nodes), 76.045137ms from start of poll, 76.993669ms from cluster start. Timeout at 2023-05-16 04:06:17.399828396 +1400 +14 m=+14.202410031, 105.259262ms after stability testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.468+1400 [ERROR] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: failed to heartbeat to: peer=5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 error="command timed out" testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.468+1400 [ERROR] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: failed to heartbeat to: peer=5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 error="command timed out" --- PASS: TestRaft_Barrier (0.25s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" testing.go:149: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" testLoggerAdapter verbose: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: entering follower state: follower="Node at b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d [Follower]" leader= testing.go:149: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: entering follower state: follower="Node at b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d [Follower]" leader= testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering follower state: follower="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Follower]" leader= testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering follower state: follower="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Follower]" leader= testLoggerAdapter verbose: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" testing.go:149: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.471+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:17.471+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: entering follower state: follower="Node at 2db3bca4-7544-c836-6f8a-94ac1a7a8c40 [Follower]" leader= testing.go:149: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: entering follower state: follower="Node at 2db3bca4-7544-c836-6f8a-94ac1a7a8c40 [Follower]" leader= testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [WARN] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [WARN] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering candidate state: node="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Candidate]" term=2 testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering candidate state: node="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Candidate]" term=2 testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: election won: tally=2 testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: election won: tally=2 testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering leader state: leader="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Leader]" testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering leader state: leader="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Leader]" testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.655+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.550496914 +1400 +14 m=+14.353078529 (1 nodes), 79.265655ms from start of poll, 80.374028ms from cluster start. Timeout at 2023-05-16 04:06:17.655839017 +1400 +14 m=+14.458420652, 105.342123ms after stability testing.go:149: cluster: 2023-05-16T04:06:17.655+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.550496914 +1400 +14 m=+14.353078529 (1 nodes), 79.265655ms from start of poll, 80.374028ms from cluster start. Timeout at 2023-05-16 04:06:17.655839017 +1400 +14 m=+14.458420652, 105.342123ms after stability testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.707+1400 [ERROR] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: failed to heartbeat to: peer=b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d error="command timed out" testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.707+1400 [ERROR] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: failed to heartbeat to: peer=b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d error="command timed out" --- PASS: TestRaft_VerifyLeader (0.24s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11a64167-65eb-1084-fa9c-f340e70b44b0 Address:11a64167-65eb-1084-fa9c-f340e70b44b0}]" testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11a64167-65eb-1084-fa9c-f340e70b44b0 Address:11a64167-65eb-1084-fa9c-f340e70b44b0}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.708+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:17.708+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering follower state: follower="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Follower]" leader= testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering follower state: follower="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Follower]" leader= testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [WARN] server-11a64167-65eb-1084-fa9c-f340e70b44b0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [WARN] server-11a64167-65eb-1084-fa9c-f340e70b44b0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering candidate state: node="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Candidate]" term=2 testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering candidate state: node="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Candidate]" term=2 testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: election won: tally=1 testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: election won: tally=1 testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering leader state: leader="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Leader]" testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering leader state: leader="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Leader]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.911+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.804093487 +1400 +14 m=+14.606675102 (1 nodes), 95.358865ms from start of poll, 95.79595ms from cluster start. Timeout at 2023-05-16 04:06:17.911871299 +1400 +14 m=+14.714453014, 107.777912ms after stability testing.go:149: cluster: 2023-05-16T04:06:17.911+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.804093487 +1400 +14 m=+14.606675102 (1 nodes), 95.358865ms from start of poll, 95.79595ms from cluster start. Timeout at 2023-05-16 04:06:17.911871299 +1400 +14 m=+14.714453014, 107.777912ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.20s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" testing.go:149: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.913+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:17.913+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: entering follower state: follower="Node at 041ad9e3-9828-91b6-adeb-e58df252c453 [Follower]" leader= testing.go:149: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: entering follower state: follower="Node at 041ad9e3-9828-91b6-adeb-e58df252c453 [Follower]" leader= testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.966+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.966+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering candidate state: node="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Candidate]" term=2 testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering candidate state: node="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Candidate]" term=2 testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: election won: tally=2 testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: election won: tally=2 testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering leader state: leader="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Leader]" testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering leader state: leader="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Leader]" testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: added peer, starting replication: peer=server-041ad9e3-9828-91b6-adeb-e58df252c453 testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: added peer, starting replication: peer=server-041ad9e3-9828-91b6-adeb-e58df252c453 testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: pipelining replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: pipelining replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.073+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.967825459 +1400 +14 m=+14.770407094 (1 nodes), 54.048998ms from start of poll, 54.718045ms from cluster start. Timeout at 2023-05-16 04:06:18.073493106 +1400 +14 m=+14.876074761, 105.667667ms after stability testing.go:149: cluster: 2023-05-16T04:06:18.073+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.967825459 +1400 +14 m=+14.770407094 (1 nodes), 54.048998ms from start of poll, 54.718045ms from cluster start. Timeout at 2023-05-16 04:06:18.073493106 +1400 +14 m=+14.876074761, 105.667667ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.073+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:18.073+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.179+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.073756829 +1400 +14 m=+14.876338484 (1 nodes), 2.5µs from start of poll, 160.649435ms from cluster start. Timeout at 2023-05-16 04:06:18.179544818 +1400 +14 m=+14.982126433, 105.787949ms after stability testing.go:149: cluster: 2023-05-16T04:06:18.179+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.073756829 +1400 +14 m=+14.876338484 (1 nodes), 2.5µs from start of poll, 160.649435ms from cluster start. Timeout at 2023-05-16 04:06:18.179544818 +1400 +14 m=+14.982126433, 105.787949ms after stability testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: new leader elected, stepping down testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: new leader elected, stepping down testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.180+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: aborting pipeline replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.180+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: aborting pipeline replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" --- PASS: TestRaft_VerifyLeader_Fail (0.27s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.181+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" testing.go:149: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.181+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.181+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.181+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" testLoggerAdapter verbose: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" testing.go:149: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.182+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:18.182+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.182+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: entering follower state: follower="Node at 898c8559-a733-964d-1704-e94acb5b2708 [Follower]" leader= testing.go:149: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.182+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: entering follower state: follower="Node at 898c8559-a733-964d-1704-e94acb5b2708 [Follower]" leader= testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.182+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering follower state: follower="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Follower]" leader= testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.182+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering follower state: follower="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Follower]" leader= testLoggerAdapter verbose: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: entering follower state: follower="Node at 9f38cc7c-059a-daae-bbea-8063509e276f [Follower]" leader= testing.go:149: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: entering follower state: follower="Node at 9f38cc7c-059a-daae-bbea-8063509e276f [Follower]" leader= testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [WARN] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [WARN] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering candidate state: node="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Candidate]" term=2 testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering candidate state: node="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Candidate]" term=2 testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: election won: tally=2 testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: election won: tally=2 testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering leader state: leader="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Leader]" testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering leader state: leader="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Leader]" testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-898c8559-a733-964d-1704-e94acb5b2708 testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-898c8559-a733-964d-1704-e94acb5b2708 testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-9f38cc7c-059a-daae-bbea-8063509e276f testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-9f38cc7c-059a-daae-bbea-8063509e276f testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.347+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.241539909 +1400 +14 m=+15.044121544 (1 nodes), 59.299559ms from start of poll, 60.246931ms from cluster start. Timeout at 2023-05-16 04:06:18.347027614 +1400 +14 m=+15.149609249, 105.487705ms after stability testing.go:149: cluster: 2023-05-16T04:06:18.347+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.241539909 +1400 +14 m=+15.044121544 (1 nodes), 59.299559ms from start of poll, 60.246931ms from cluster start. Timeout at 2023-05-16 04:06:18.347027614 +1400 +14 m=+15.149609249, 105.487705ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.353+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:18.353+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.459+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.353335469 +1400 +14 m=+15.155917104 (2 nodes), 3.26µs from start of poll, 172.042491ms from cluster start. Timeout at 2023-05-16 04:06:18.459060116 +1400 +14 m=+15.261641751, 105.724647ms after stability testing.go:149: cluster: 2023-05-16T04:06:18.459+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.353335469 +1400 +14 m=+15.155917104 (2 nodes), 3.26µs from start of poll, 172.042491ms from cluster start. Timeout at 2023-05-16 04:06:18.459060116 +1400 +14 m=+15.261641751, 105.724647ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 898c8559-a733-964d-1704-e94acb5b2708 [Follower] testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [ERROR] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: failed to heartbeat to: peer=898c8559-a733-964d-1704-e94acb5b2708 error="failed to connect to peer: 898c8559-a733-964d-1704-e94acb5b2708" testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [ERROR] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: failed to heartbeat to: peer=898c8559-a733-964d-1704-e94acb5b2708 error="failed to connect to peer: 898c8559-a733-964d-1704-e94acb5b2708" testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.460+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 Address:2af8cc22-51eb-7b07-74b5-19ad6d1eefa1}]" testing.go:149: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.460+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 Address:2af8cc22-51eb-7b07-74b5-19ad6d1eefa1}]" testLoggerAdapter verbose: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.461+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: entering leader state: leader="Node at 2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 [Leader]" testing.go:149: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.461+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: entering leader state: leader="Node at 2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.461+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ba23a44-267a-81b6-1805-e686655a6c81 Address:3ba23a44-267a-81b6-1805-e686655a6c81}]" testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.461+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ba23a44-267a-81b6-1805-e686655a6c81 Address:3ba23a44-267a-81b6-1805-e686655a6c81}]" testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.462+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering follower state: follower="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Follower]" leader= testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.462+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering follower state: follower="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Follower]" leader= testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [WARN] server-3ba23a44-267a-81b6-1805-e686655a6c81: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [WARN] server-3ba23a44-267a-81b6-1805-e686655a6c81: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering candidate state: node="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Candidate]" term=2 testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering candidate state: node="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Candidate]" term=2 testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: election won: tally=1 testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: election won: tally=1 testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering leader state: leader="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Leader]" testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering leader state: leader="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Leader]" --- PASS: TestRaft_NotifyCh (0.06s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" testLoggerAdapter verbose: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" testing.go:149: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" testLoggerAdapter verbose: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.518+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" testing.go:149: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.518+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.518+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:18.518+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering follower state: follower="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Follower]" leader= testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering follower state: follower="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Follower]" leader= testLoggerAdapter verbose: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: entering follower state: follower="Node at 9ebbf41e-b993-2ba7-fd13-b4f512eb226a [Follower]" leader= testing.go:149: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: entering follower state: follower="Node at 9ebbf41e-b993-2ba7-fd13-b4f512eb226a [Follower]" leader= testLoggerAdapter verbose: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.519+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: entering follower state: follower="Node at 97226467-e0a6-5686-538f-707a3f5ac009 [Follower]" leader= testing.go:149: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.519+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: entering follower state: follower="Node at 97226467-e0a6-5686-538f-707a3f5ac009 [Follower]" leader= testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [WARN] server-77f13ea4-1642-0b64-dcd1-eef457780c43: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [WARN] server-77f13ea4-1642-0b64-dcd1-eef457780c43: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering candidate state: node="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Candidate]" term=2 testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering candidate state: node="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Candidate]" term=2 testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: election won: tally=2 testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: election won: tally=2 testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering leader state: leader="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Leader]" testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering leader state: leader="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Leader]" testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-97226467-e0a6-5686-538f-707a3f5ac009 testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-97226467-e0a6-5686-538f-707a3f5ac009 testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.583+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.583+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.688+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.583331563 +1400 +14 m=+15.385913178 (2 nodes), 64.546282ms from start of poll, 65.561254ms from cluster start. Timeout at 2023-05-16 04:06:18.688423363 +1400 +14 m=+15.491004998, 105.09182ms after stability testing.go:149: cluster: 2023-05-16T04:06:18.688+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.583331563 +1400 +14 m=+15.385913178 (2 nodes), 64.546282ms from start of poll, 65.561254ms from cluster start. Timeout at 2023-05-16 04:06:18.688423363 +1400 +14 m=+15.491004998, 105.09182ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.688+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:18.688+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.794+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.688694086 +1400 +14 m=+15.491275721 (1 nodes), 2.06µs from start of poll, 170.923797ms from cluster start. Timeout at 2023-05-16 04:06:18.79405295 +1400 +14 m=+15.596634605, 105.358884ms after stability testing.go:149: cluster: 2023-05-16T04:06:18.794+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.688694086 +1400 +14 m=+15.491275721 (1 nodes), 2.06µs from start of poll, 170.923797ms from cluster start. Timeout at 2023-05-16 04:06:18.79405295 +1400 +14 m=+15.596634605, 105.358884ms after stability testLoggerAdapter verbose: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.794+1400 [WARN] server-97226467-e0a6-5686-538f-707a3f5ac009: rejecting vote request since we have a leader: from=9ebbf41e-b993-2ba7-fd13-b4f512eb226a leader=77f13ea4-1642-0b64-dcd1-eef457780c43 testing.go:149: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.794+1400 [WARN] server-97226467-e0a6-5686-538f-707a3f5ac009: rejecting vote request since we have a leader: from=9ebbf41e-b993-2ba7-fd13-b4f512eb226a leader=77f13ea4-1642-0b64-dcd1-eef457780c43 testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" --- PASS: TestRaft_Voting (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" testing.go:149: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" testLoggerAdapter verbose: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" testing.go:149: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.796+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:18.796+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: entering follower state: follower="Node at 030f13fa-8e96-6dd9-facf-6ac36a7aac3d [Follower]" leader= testing.go:149: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: entering follower state: follower="Node at 030f13fa-8e96-6dd9-facf-6ac36a7aac3d [Follower]" leader= testLoggerAdapter verbose: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: entering follower state: follower="Node at 3a357a48-8a72-f73f-468d-5ba6197fa8a3 [Follower]" leader= testing.go:149: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: entering follower state: follower="Node at 3a357a48-8a72-f73f-468d-5ba6197fa8a3 [Follower]" leader= testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering follower state: follower="Node at be946513-49ca-a889-e985-34e5143e9042 [Follower]" leader= testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering follower state: follower="Node at be946513-49ca-a889-e985-34e5143e9042 [Follower]" leader= testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [WARN] server-be946513-49ca-a889-e985-34e5143e9042: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [WARN] server-be946513-49ca-a889-e985-34e5143e9042: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering candidate state: node="Node at be946513-49ca-a889-e985-34e5143e9042 [Candidate]" term=2 testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering candidate state: node="Node at be946513-49ca-a889-e985-34e5143e9042 [Candidate]" term=2 testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: election won: tally=2 testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: election won: tally=2 testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering leader state: leader="Node at be946513-49ca-a889-e985-34e5143e9042 [Leader]" testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering leader state: leader="Node at be946513-49ca-a889-e985-34e5143e9042 [Leader]" testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.957+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.851592209 +1400 +14 m=+15.654173844 (2 nodes), 54.812587ms from start of poll, 55.718078ms from cluster start. Timeout at 2023-05-16 04:06:18.957174035 +1400 +14 m=+15.759755670, 105.581826ms after stability testing.go:149: cluster: 2023-05-16T04:06:18.957+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.851592209 +1400 +14 m=+15.654173844 (2 nodes), 54.812587ms from start of poll, 55.718078ms from cluster start. Timeout at 2023-05-16 04:06:18.957174035 +1400 +14 m=+15.759755670, 105.581826ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.957+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:18.957+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.062+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.957402257 +1400 +14 m=+15.759983892 (1 nodes), 1.96µs from start of poll, 161.528126ms from cluster start. Timeout at 2023-05-16 04:06:19.06268426 +1400 +14 m=+15.865265895, 105.282003ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.062+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.957402257 +1400 +14 m=+15.759983892 (1 nodes), 1.96µs from start of poll, 161.528126ms from cluster start. Timeout at 2023-05-16 04:06:19.06268426 +1400 +14 m=+15.865265895, 105.282003ms after stability testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.064+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" testing.go:149: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.064+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: initial configuration: index=0 servers=[] testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.065+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:19.065+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.065+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: entering follower state: follower="Node at 9e1a7b19-2472-077e-8876-67055c18b3a4 [Follower]" leader= testing.go:149: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.065+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: entering follower state: follower="Node at 9e1a7b19-2472-077e-8876-67055c18b3a4 [Follower]" leader= testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering follower state: follower="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Follower]" leader= testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering follower state: follower="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Follower]" leader= testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: entering follower state: follower="Node at c6eed6c2-5993-16be-5926-f5d4aba279d8 [Follower]" leader= testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: entering follower state: follower="Node at c6eed6c2-5993-16be-5926-f5d4aba279d8 [Follower]" leader= testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.119+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: no known peers, aborting election testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.119+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: no known peers, aborting election testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: heartbeat timeout reached, starting election: last-leader= testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering candidate state: node="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Candidate]" term=2 testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering candidate state: node="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Candidate]" term=2 testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: election won: tally=2 testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: election won: tally=2 testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering leader state: leader="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Leader]" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering leader state: leader="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Leader]" testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=9e1a7b19-2472-077e-8876-67055c18b3a4 testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=9e1a7b19-2472-077e-8876-67055c18b3a4 testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.246+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.140613419 +1400 +14 m=+15.943195034 (1 nodes), 75.052085ms from start of poll, 76.090098ms from cluster start. Timeout at 2023-05-16 04:06:19.24658299 +1400 +14 m=+16.049164605, 105.969571ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.246+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.140613419 +1400 +14 m=+15.943195034 (1 nodes), 75.052085ms from start of poll, 76.090098ms from cluster start. Timeout at 2023-05-16 04:06:19.24658299 +1400 +14 m=+16.049164605, 105.969571ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.246+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:19.246+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.352+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.246853153 +1400 +14 m=+16.049434768 (1 nodes), 2.14µs from start of poll, 182.329832ms from cluster start. Timeout at 2023-05-16 04:06:19.352708602 +1400 +14 m=+16.155290237, 105.855469ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.352+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.246853153 +1400 +14 m=+16.049434768 (1 nodes), 2.14µs from start of poll, 182.329832ms from cluster start. Timeout at 2023-05-16 04:06:19.352708602 +1400 +14 m=+16.155290237, 105.855469ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.352+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:19.352+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.459+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.352971085 +1400 +14 m=+16.155552720 (1 nodes), 1.84µs from start of poll, 288.447784ms from cluster start. Timeout at 2023-05-16 04:06:19.458985397 +1400 +14 m=+16.261567052, 106.014332ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.459+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.352971085 +1400 +14 m=+16.155552720 (1 nodes), 1.84µs from start of poll, 288.447784ms from cluster start. Timeout at 2023-05-16 04:06:19.458985397 +1400 +14 m=+16.261567052, 106.014332ms after stability testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=AddStaging server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr=c6eed6c2-5993-16be-5926-f5d4aba279d8 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2} {Suffrage:Voter ID:c6eed6c2-5993-16be-5926-f5d4aba279d8 Address:c6eed6c2-5993-16be-5926-f5d4aba279d8}]" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=AddStaging server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr=c6eed6c2-5993-16be-5926-f5d4aba279d8 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2} {Suffrage:Voter ID:c6eed6c2-5993-16be-5926-f5d4aba279d8 Address:c6eed6c2-5993-16be-5926-f5d4aba279d8}]" testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.459+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.459+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: appendEntries rejected, sending older logs: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" next=1 testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: appendEntries rejected, sending older logs: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" next=1 testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.460+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:19.460+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.565+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.460254932 +1400 +14 m=+16.262836567 (1 nodes), 1.36µs from start of poll, 395.731631ms from cluster start. Timeout at 2023-05-16 04:06:19.565259331 +1400 +14 m=+16.367840946, 105.004379ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.565+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.460254932 +1400 +14 m=+16.262836567 (1 nodes), 1.36µs from start of poll, 395.731631ms from cluster start. Timeout at 2023-05-16 04:06:19.565259331 +1400 +14 m=+16.367840946, 105.004379ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.565+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:19.565+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.671+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.565492793 +1400 +14 m=+16.368074428 (1 nodes), 1.56µs from start of poll, 500.969492ms from cluster start. Timeout at 2023-05-16 04:06:19.671050599 +1400 +14 m=+16.473632374, 105.557946ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.671+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.565492793 +1400 +14 m=+16.368074428 (1 nodes), 1.56µs from start of poll, 500.969492ms from cluster start. Timeout at 2023-05-16 04:06:19.671050599 +1400 +14 m=+16.473632374, 105.557946ms after stability testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=RemoveServer server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr= servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=RemoveServer server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr= servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: removed peer, stopping replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 last-index=4 testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: removed peer, stopping replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 last-index=4 testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.673+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" testing.go:149: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.673+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.673+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.673+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.673+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:19.673+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.674+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: entering follower state: follower="Node at 75b2fd5b-075b-1749-1248-85b03602aac3 [Follower]" leader= testing.go:149: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.674+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: entering follower state: follower="Node at 75b2fd5b-075b-1749-1248-85b03602aac3 [Follower]" leader= testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.674+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering follower state: follower="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Follower]" leader= testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.674+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering follower state: follower="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Follower]" leader= testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: heartbeat timeout reached, starting election: last-leader= testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering candidate state: node="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Candidate]" term=2 testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering candidate state: node="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Candidate]" term=2 testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: election won: tally=2 testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: election won: tally=2 testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering leader state: leader="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Leader]" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering leader state: leader="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Leader]" testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.834+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:19.72875002 +1400 +14 m=+16.531331635 (1 nodes), 54.765886ms from start of poll, 55.415354ms from cluster start. Timeout at 2023-05-16 04:06:19.834214265 +1400 +14 m=+16.636796020, 105.464385ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.834+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:19.72875002 +1400 +14 m=+16.531331635 (1 nodes), 54.765886ms from start of poll, 55.415354ms from cluster start. Timeout at 2023-05-16 04:06:19.834214265 +1400 +14 m=+16.636796020, 105.464385ms after stability testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.834+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: initial configuration: index=0 servers=[] testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.834+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.834+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:19.834+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.835+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: entering follower state: follower="Node at 038e407e-9062-3d91-552a-1f5650a48253 [Follower]" leader= testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.835+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: entering follower state: follower="Node at 038e407e-9062-3d91-552a-1f5650a48253 [Follower]" leader= testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.890+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: no known peers, aborting election testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.890+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: no known peers, aborting election testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.940+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.835011634 +1400 +14 m=+16.637593249 (1 nodes), 2.06µs from start of poll, 161.676968ms from cluster start. Timeout at 2023-05-16 04:06:19.940104674 +1400 +14 m=+16.742686309, 105.09306ms after stability testing.go:149: cluster: 2023-05-16T04:06:19.940+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.835011634 +1400 +14 m=+16.637593249 (1 nodes), 2.06µs from start of poll, 161.676968ms from cluster start. Timeout at 2023-05-16 04:06:19.940104674 +1400 +14 m=+16.742686309, 105.09306ms after stability testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=AddStaging server-id=server-038e407e-9062-3d91-552a-1f5650a48253 server-addr=038e407e-9062-3d91-552a-1f5650a48253 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=AddStaging server-id=server-038e407e-9062-3d91-552a-1f5650a48253 server-addr=038e407e-9062-3d91-552a-1f5650a48253 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=server-038e407e-9062-3d91-552a-1f5650a48253 testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=server-038e407e-9062-3d91-552a-1f5650a48253 testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.940+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.940+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: appendEntries rejected, sending older logs: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" next=1 testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: appendEntries rejected, sending older logs: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" next=1 testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.941+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.941+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.941+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:19.941+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.047+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.94140807 +1400 +14 m=+16.743989685 (1 nodes), 1.3µs from start of poll, 268.073404ms from cluster start. Timeout at 2023-05-16 04:06:20.047218838 +1400 +14 m=+16.849800493, 105.810808ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.047+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.94140807 +1400 +14 m=+16.743989685 (1 nodes), 1.3µs from start of poll, 268.073404ms from cluster start. Timeout at 2023-05-16 04:06:20.047218838 +1400 +14 m=+16.849800493, 105.810808ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.047+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:20.047+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.153+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.047557142 +1400 +14 m=+16.850138777 (1 nodes), 1.94µs from start of poll, 374.222496ms from cluster start. Timeout at 2023-05-16 04:06:20.153336711 +1400 +14 m=+16.955918326, 105.779549ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.153+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.047557142 +1400 +14 m=+16.850138777 (1 nodes), 1.94µs from start of poll, 374.222496ms from cluster start. Timeout at 2023-05-16 04:06:20.153336711 +1400 +14 m=+16.955918326, 105.779549ms after stability testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=RemoveServer server-id=75b2fd5b-075b-1749-1248-85b03602aac3 server-addr= servers="[{Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=RemoveServer server-id=75b2fd5b-075b-1749-1248-85b03602aac3 server-addr= servers="[{Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: removed peer, stopping replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 last-index=4 testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: removed peer, stopping replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 last-index=4 testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.154+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.154+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.48s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.155+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.155+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" testLoggerAdapter verbose: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" testing.go:149: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.156+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:20.156+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.156+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering follower state: follower="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Follower]" leader= testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.156+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering follower state: follower="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Follower]" leader= testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= testLoggerAdapter verbose: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: entering follower state: follower="Node at b6401cfe-e422-54c7-68c4-fa38095f64d8 [Follower]" leader= testing.go:149: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: entering follower state: follower="Node at b6401cfe-e422-54c7-68c4-fa38095f64d8 [Follower]" leader= testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [WARN] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [WARN] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering candidate state: node="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Candidate]" term=2 testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering candidate state: node="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Candidate]" term=2 testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: election won: tally=2 testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: election won: tally=2 testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering leader state: leader="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Leader]" testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering leader state: leader="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Leader]" testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.226+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.226+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.226303772 +1400 +14 m=+17.028885407 (1 nodes), 69.662622ms from start of poll, 70.586653ms from cluster start. Timeout at 2023-05-16 04:06:20.332152501 +1400 +14 m=+17.134734156, 105.848749ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.226303772 +1400 +14 m=+17.028885407 (1 nodes), 69.662622ms from start of poll, 70.586653ms from cluster start. Timeout at 2023-05-16 04:06:20.332152501 +1400 +14 m=+17.134734156, 105.848749ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.332+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:20.332+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.437+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.332473485 +1400 +14 m=+17.135055100 (1 nodes), 2.1µs from start of poll, 176.756346ms from cluster start. Timeout at 2023-05-16 04:06:20.437477144 +1400 +14 m=+17.240058779, 105.003679ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.437+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.332473485 +1400 +14 m=+17.135055100 (1 nodes), 2.1µs from start of poll, 176.756346ms from cluster start. Timeout at 2023-05-16 04:06:20.437477144 +1400 +14 m=+17.240058779, 105.003679ms after stability testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.437+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering candidate state: node="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Candidate]" term=3 testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.437+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering candidate state: node="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Candidate]" term=3 testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: election won: tally=2 testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: election won: tally=2 testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering leader state: leader="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Leader]" testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering leader state: leader="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Leader]" testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-6249adaf-7fa0-9a24-f9f2-179cb0877210 testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-6249adaf-7fa0-9a24-f9f2-179cb0877210 testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.438+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:20.438+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.439+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.439+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.545+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.439133283 +1400 +14 m=+17.241714918 (1 nodes), 467.545µs from start of poll, 283.416164ms from cluster start. Timeout at 2023-05-16 04:06:20.545061653 +1400 +14 m=+17.347643288, 105.92837ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.545+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.439133283 +1400 +14 m=+17.241714918 (1 nodes), 467.545µs from start of poll, 283.416164ms from cluster start. Timeout at 2023-05-16 04:06:20.545061653 +1400 +14 m=+17.347643288, 105.92837ms after stability testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" --- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fe047cd-368f-42d7-8337-4924b08f6fe8 Address:2fe047cd-368f-42d7-8337-4924b08f6fe8}]" testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fe047cd-368f-42d7-8337-4924b08f6fe8 Address:2fe047cd-368f-42d7-8337-4924b08f6fe8}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.546+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:20.546+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering follower state: follower="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Follower]" leader= testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering follower state: follower="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Follower]" leader= testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [WARN] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [WARN] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering candidate state: node="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Candidate]" term=2 testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering candidate state: node="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Candidate]" term=2 testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: election won: tally=1 testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: election won: tally=1 testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering leader state: leader="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Leader]" testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering leader state: leader="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Leader]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.727+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.621952741 +1400 +14 m=+17.424534356 (1 nodes), 75.040446ms from start of poll, 75.458011ms from cluster start. Timeout at 2023-05-16 04:06:20.727542187 +1400 +14 m=+17.530124022, 105.589666ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.727+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.621952741 +1400 +14 m=+17.424534356 (1 nodes), 75.040446ms from start of poll, 75.458011ms from cluster start. Timeout at 2023-05-16 04:06:20.727542187 +1400 +14 m=+17.530124022, 105.589666ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.730+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.730+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testLoggerAdapter verbose: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.730+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testing.go:149: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.730+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testLoggerAdapter verbose: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.730+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testing.go:149: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.730+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testLoggerAdapter verbose: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.731+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testing.go:149: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.731+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testLoggerAdapter verbose: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testing.go:149: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testLoggerAdapter verbose: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.732+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testing.go:149: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.732+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.732+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:20.732+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.732+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: entering follower state: follower="Node at dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 [Follower]" leader= testing.go:149: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.732+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: entering follower state: follower="Node at dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 [Follower]" leader= testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.732+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering follower state: follower="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Follower]" leader= testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.732+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering follower state: follower="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Follower]" leader= testLoggerAdapter verbose: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.732+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: entering follower state: follower="Node at b81d5f4b-ba2e-5649-2138-d348eeb0f60e [Follower]" leader= testing.go:149: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.732+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: entering follower state: follower="Node at b81d5f4b-ba2e-5649-2138-d348eeb0f60e [Follower]" leader= testLoggerAdapter verbose: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.732+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: entering follower state: follower="Node at bc478a63-a6e6-28f5-2d04-37c72331e776 [Follower]" leader= testing.go:149: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.732+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: entering follower state: follower="Node at bc478a63-a6e6-28f5-2d04-37c72331e776 [Follower]" leader= testLoggerAdapter verbose: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: entering follower state: follower="Node at 5b6bca9a-f4a0-7870-ba64-7f33be112495 [Follower]" leader= testing.go:149: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: entering follower state: follower="Node at 5b6bca9a-f4a0-7870-ba64-7f33be112495 [Follower]" leader= testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= testLoggerAdapter verbose: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.733+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: entering follower state: follower="Node at c9924079-5f27-61ca-26f7-4c35362c26f4 [Follower]" leader= testing.go:149: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.733+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: entering follower state: follower="Node at c9924079-5f27-61ca-26f7-4c35362c26f4 [Follower]" leader= testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [WARN] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [WARN] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering candidate state: node="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Candidate]" term=2 testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering candidate state: node="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Candidate]" term=2 testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: election won: tally=4 testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: election won: tally=4 testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering leader state: leader="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Leader]" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering leader state: leader="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Leader]" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.891+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.786153778 +1400 +14 m=+17.588735413 (1 nodes), 52.726162ms from start of poll, 56.512586ms from cluster start. Timeout at 2023-05-16 04:06:20.891425021 +1400 +14 m=+17.694006656, 105.271243ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.891+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.786153778 +1400 +14 m=+17.588735413 (1 nodes), 52.726162ms from start of poll, 56.512586ms from cluster start. Timeout at 2023-05-16 04:06:20.891425021 +1400 +14 m=+17.694006656, 105.271243ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.891+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:20.891+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.997+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:20.891784305 +1400 +14 m=+17.694365940 (6 nodes), 7.2µs from start of poll, 162.143113ms from cluster start. Timeout at 2023-05-16 04:06:20.997572193 +1400 +14 m=+17.800153808, 105.787868ms after stability testing.go:149: cluster: 2023-05-16T04:06:20.997+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:20.891784305 +1400 +14 m=+17.694365940 (6 nodes), 7.2µs from start of poll, 162.143113ms from cluster start. Timeout at 2023-05-16 04:06:20.997572193 +1400 +14 m=+17.800153808, 105.787868ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.997+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:20.997+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.103+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.997866677 +1400 +14 m=+17.800448292 (1 nodes), 3.9µs from start of poll, 268.225465ms from cluster start. Timeout at 2023-05-16 04:06:21.102998178 +1400 +14 m=+17.905579813, 105.131521ms after stability testing.go:149: cluster: 2023-05-16T04:06:21.103+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.997866677 +1400 +14 m=+17.800448292 (1 nodes), 3.9µs from start of poll, 268.225465ms from cluster start. Timeout at 2023-05-16 04:06:21.102998178 +1400 +14 m=+17.905579813, 105.131521ms after stability testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering candidate state: node="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Candidate]" term=3 testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering candidate state: node="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.103+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:21.103+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: election won: tally=4 testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: election won: tally=4 testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering leader state: leader="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Leader]" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering leader state: leader="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Leader]" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.210+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.105051342 +1400 +14 m=+17.907632977 (1 nodes), 1.332976ms from start of poll, 375.41015ms from cluster start. Timeout at 2023-05-16 04:06:21.210573427 +1400 +14 m=+18.013155042, 105.522065ms after stability testing.go:149: cluster: 2023-05-16T04:06:21.210+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.105051342 +1400 +14 m=+17.907632977 (1 nodes), 1.332976ms from start of poll, 375.41015ms from cluster start. Timeout at 2023-05-16 04:06:21.210573427 +1400 +14 m=+18.013155042, 105.522065ms after stability testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.210+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.210+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.261+1400 [ERROR] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: failed to heartbeat to: peer=b81d5f4b-ba2e-5649-2138-d348eeb0f60e error="command timed out" testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.261+1400 [ERROR] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: failed to heartbeat to: peer=b81d5f4b-ba2e-5649-2138-d348eeb0f60e error="command timed out" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.53s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.263+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" testing.go:149: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.263+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.263+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.263+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" testLoggerAdapter verbose: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" testing.go:149: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.264+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:21.264+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.264+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: entering follower state: follower="Node at c3a2c95d-cbc0-7303-8574-ba9850b6768c [Follower]" leader= testing.go:149: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.264+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: entering follower state: follower="Node at c3a2c95d-cbc0-7303-8574-ba9850b6768c [Follower]" leader= testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.264+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering follower state: follower="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Follower]" leader= testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.264+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering follower state: follower="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Follower]" leader= testLoggerAdapter verbose: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: entering follower state: follower="Node at 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 [Follower]" leader= testing.go:149: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: entering follower state: follower="Node at 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 [Follower]" leader= testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [WARN] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [WARN] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering candidate state: node="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Candidate]" term=2 testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering candidate state: node="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Candidate]" term=2 testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: election won: tally=2 testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: election won: tally=2 testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering leader state: leader="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Leader]" testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering leader state: leader="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Leader]" testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-c3a2c95d-cbc0-7303-8574-ba9850b6768c testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-c3a2c95d-cbc0-7303-8574-ba9850b6768c testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.430+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.325458723 +1400 +14 m=+18.128040338 (1 nodes), 61.03042ms from start of poll, 62.077193ms from cluster start. Timeout at 2023-05-16 04:06:21.430471862 +1400 +14 m=+18.233053497, 105.013159ms after stability testing.go:149: cluster: 2023-05-16T04:06:21.430+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.325458723 +1400 +14 m=+18.128040338 (1 nodes), 61.03042ms from start of poll, 62.077193ms from cluster start. Timeout at 2023-05-16 04:06:21.430471862 +1400 +14 m=+18.233053497, 105.013159ms after stability testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.430+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.430+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.431+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.431+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.432+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" testing.go:149: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.432+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" testLoggerAdapter verbose: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" testing.go:149: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.433+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:21.433+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.433+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: entering follower state: follower="Node at 63d1e415-1f05-330d-1b02-64bd6757dcbf [Follower]" leader= testing.go:149: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.433+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: entering follower state: follower="Node at 63d1e415-1f05-330d-1b02-64bd6757dcbf [Follower]" leader= testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering follower state: follower="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Follower]" leader= testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering follower state: follower="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Follower]" leader= testLoggerAdapter verbose: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: entering follower state: follower="Node at 3e142577-3fcc-adc5-50c4-612303809486 [Follower]" leader= testing.go:149: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: entering follower state: follower="Node at 3e142577-3fcc-adc5-50c4-612303809486 [Follower]" leader= testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [WARN] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [WARN] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering candidate state: node="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Candidate]" term=2 testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering candidate state: node="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Candidate]" term=2 testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: election won: tally=2 testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: election won: tally=2 testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering leader state: leader="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Leader]" testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering leader state: leader="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Leader]" testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-63d1e415-1f05-330d-1b02-64bd6757dcbf testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-63d1e415-1f05-330d-1b02-64bd6757dcbf testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-3e142577-3fcc-adc5-50c4-612303809486 testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-3e142577-3fcc-adc5-50c4-612303809486 testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.599+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.493601167 +1400 +14 m=+18.296182782 (2 nodes), 60.17511ms from start of poll, 61.198502ms from cluster start. Timeout at 2023-05-16 04:06:21.599120372 +1400 +14 m=+18.401702007, 105.519225ms after stability testing.go:149: cluster: 2023-05-16T04:06:21.599+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.493601167 +1400 +14 m=+18.296182782 (2 nodes), 60.17511ms from start of poll, 61.198502ms from cluster start. Timeout at 2023-05-16 04:06:21.599120372 +1400 +14 m=+18.401702007, 105.519225ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.599+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:21.599+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.704+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.599468656 +1400 +14 m=+18.402050351 (1 nodes), 2.7µs from start of poll, 167.066071ms from cluster start. Timeout at 2023-05-16 04:06:21.704729178 +1400 +14 m=+18.507310933, 105.260582ms after stability testing.go:149: cluster: 2023-05-16T04:06:21.704+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.599468656 +1400 +14 m=+18.402050351 (1 nodes), 2.7µs from start of poll, 167.066071ms from cluster start. Timeout at 2023-05-16 04:06:21.704729178 +1400 +14 m=+18.507310933, 105.260582ms after stability testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.706+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.706+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" testLoggerAdapter verbose: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" testing.go:149: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.707+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:21.707+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.707+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.707+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering follower state: follower="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Follower]" leader= testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering follower state: follower="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Follower]" leader= testLoggerAdapter verbose: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: entering follower state: follower="Node at e6c9f727-3007-3e26-19fe-1764cdaedea9 [Follower]" leader= testing.go:149: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: entering follower state: follower="Node at e6c9f727-3007-3e26-19fe-1764cdaedea9 [Follower]" leader= testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [WARN] server-313389c2-f60a-9532-2f79-5beec9a54ec8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [WARN] server-313389c2-f60a-9532-2f79-5beec9a54ec8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering candidate state: node="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Candidate]" term=2 testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering candidate state: node="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Candidate]" term=2 testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: election won: tally=2 testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: election won: tally=2 testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering leader state: leader="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Leader]" testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering leader state: leader="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Leader]" testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.872+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.76672115 +1400 +14 m=+18.569302785 (1 nodes), 59.121958ms from start of poll, 60.108429ms from cluster start. Timeout at 2023-05-16 04:06:21.872259455 +1400 +14 m=+18.674841110, 105.538325ms after stability testing.go:149: cluster: 2023-05-16T04:06:21.872+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.76672115 +1400 +14 m=+18.569302785 (1 nodes), 59.121958ms from start of poll, 60.108429ms from cluster start. Timeout at 2023-05-16 04:06:21.872259455 +1400 +14 m=+18.674841110, 105.538325ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.872+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:21.872+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.978+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.872559219 +1400 +14 m=+18.675140854 (2 nodes), 2.58µs from start of poll, 165.946498ms from cluster start. Timeout at 2023-05-16 04:06:21.978340427 +1400 +14 m=+18.780922042, 105.781188ms after stability testing.go:149: cluster: 2023-05-16T04:06:21.978+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.872559219 +1400 +14 m=+18.675140854 (2 nodes), 2.58µs from start of poll, 165.946498ms from cluster start. Timeout at 2023-05-16 04:06:21.978340427 +1400 +14 m=+18.780922042, 105.781188ms after stability testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering candidate state: node="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Candidate]" term=3 testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering candidate state: node="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.045+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:22.045+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: election won: tally=2 testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: election won: tally=2 testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.045+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.045+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering leader state: leader="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Leader]" testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering leader state: leader="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Leader]" testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-313389c2-f60a-9532-2f79-5beec9a54ec8 testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-313389c2-f60a-9532-2f79-5beec9a54ec8 testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.151+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.046016226 +1400 +14 m=+18.848597861 (1 nodes), 123.961µs from start of poll, 339.403505ms from cluster start. Timeout at 2023-05-16 04:06:22.151655153 +1400 +14 m=+18.954236788, 105.638927ms after stability testing.go:149: cluster: 2023-05-16T04:06:22.151+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.046016226 +1400 +14 m=+18.848597861 (1 nodes), 123.961µs from start of poll, 339.403505ms from cluster start. Timeout at 2023-05-16 04:06:22.151655153 +1400 +14 m=+18.954236788, 105.638927ms after stability testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.45s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.153+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" testing.go:149: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.153+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" testLoggerAdapter verbose: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" testing.go:149: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.154+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.154+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.154+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:22.154+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.154+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: entering follower state: follower="Node at daa8a26e-5b8f-d054-8241-34f612eba224 [Follower]" leader= testing.go:149: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.154+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: entering follower state: follower="Node at daa8a26e-5b8f-d054-8241-34f612eba224 [Follower]" leader= testLoggerAdapter verbose: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: entering follower state: follower="Node at 21827d4e-199d-cef6-9f4c-689713aed83c [Follower]" leader= testing.go:149: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: entering follower state: follower="Node at 21827d4e-199d-cef6-9f4c-689713aed83c [Follower]" leader= testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.155+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering follower state: follower="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Follower]" leader= testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.155+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering follower state: follower="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Follower]" leader= testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [WARN] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [WARN] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering candidate state: node="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Candidate]" term=2 testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering candidate state: node="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Candidate]" term=2 testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: election won: tally=2 testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: election won: tally=2 testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering leader state: leader="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Leader]" testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering leader state: leader="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Leader]" testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-daa8a26e-5b8f-d054-8241-34f612eba224 testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-daa8a26e-5b8f-d054-8241-34f612eba224 testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-21827d4e-199d-cef6-9f4c-689713aed83c testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-21827d4e-199d-cef6-9f4c-689713aed83c testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.328+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.222709611 +1400 +14 m=+19.025291246 (1 nodes), 67.980582ms from start of poll, 69.033015ms from cluster start. Timeout at 2023-05-16 04:06:22.328822203 +1400 +14 m=+19.131403838, 106.112592ms after stability testing.go:149: cluster: 2023-05-16T04:06:22.328+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.222709611 +1400 +14 m=+19.025291246 (1 nodes), 67.980582ms from start of poll, 69.033015ms from cluster start. Timeout at 2023-05-16 04:06:22.328822203 +1400 +14 m=+19.131403838, 106.112592ms after stability testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: cannot transfer leadership to itself testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: cannot transfer leadership to itself testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" --- PASS: TestRaft_LeadershipTransferToItself (0.18s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.331+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" testing.go:149: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.331+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.331+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.331+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" testLoggerAdapter verbose: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.331+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" testing.go:149: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.331+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.331+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:22.331+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.332+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: entering follower state: follower="Node at 33a955cb-f8ac-72fc-2126-42378abd615c [Follower]" leader= testing.go:149: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.332+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: entering follower state: follower="Node at 33a955cb-f8ac-72fc-2126-42378abd615c [Follower]" leader= testLoggerAdapter verbose: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.332+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: entering follower state: follower="Node at a9bf173c-fced-bf88-7997-c052452235f4 [Follower]" leader= testing.go:149: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.332+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: entering follower state: follower="Node at a9bf173c-fced-bf88-7997-c052452235f4 [Follower]" leader= testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.332+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering follower state: follower="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Follower]" leader= testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.332+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering follower state: follower="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Follower]" leader= testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [WARN] server-58827ced-5746-ffdb-b21d-b7ff77619d86: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [WARN] server-58827ced-5746-ffdb-b21d-b7ff77619d86: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering candidate state: node="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Candidate]" term=2 testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering candidate state: node="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Candidate]" term=2 testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: election won: tally=2 testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: election won: tally=2 testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering leader state: leader="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Leader]" testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering leader state: leader="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Leader]" testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-33a955cb-f8ac-72fc-2126-42378abd615c testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-33a955cb-f8ac-72fc-2126-42378abd615c testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-a9bf173c-fced-bf88-7997-c052452235f4 testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-a9bf173c-fced-bf88-7997-c052452235f4 testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.512+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.40651346 +1400 +14 m=+19.209095095 (1 nodes), 74.535519ms from start of poll, 75.593492ms from cluster start. Timeout at 2023-05-16 04:06:22.512216688 +1400 +14 m=+19.314798383, 105.703288ms after stability testing.go:149: cluster: 2023-05-16T04:06:22.512+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.40651346 +1400 +14 m=+19.209095095 (1 nodes), 74.535519ms from start of poll, 75.593492ms from cluster start. Timeout at 2023-05-16 04:06:22.512216688 +1400 +14 m=+19.314798383, 105.703288ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.512+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:22.512+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.618+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:22.512506651 +1400 +14 m=+19.315088266 (2 nodes), 3.14µs from start of poll, 181.586663ms from cluster start. Timeout at 2023-05-16 04:06:22.61838064 +1400 +14 m=+19.420962295, 105.874029ms after stability testing.go:149: cluster: 2023-05-16T04:06:22.618+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:22.512506651 +1400 +14 m=+19.315088266 (2 nodes), 3.14µs from start of poll, 181.586663ms from cluster start. Timeout at 2023-05-16 04:06:22.61838064 +1400 +14 m=+19.420962295, 105.874029ms after stability testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.163+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.163+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.164+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.164+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.215+1400 [ERROR] server-58827ced-5746-ffdb-b21d-b7ff77619d86: failed to heartbeat to: peer=33a955cb-f8ac-72fc-2126-42378abd615c error="command timed out" testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.215+1400 [ERROR] server-58827ced-5746-ffdb-b21d-b7ff77619d86: failed to heartbeat to: peer=33a955cb-f8ac-72fc-2126-42378abd615c error="command timed out" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.89s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering follower state: follower="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Follower]" leader= testLoggerAdapter verbose: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering follower state: follower="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Follower]" leader= testing.go:149: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.217+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-05-16T04:06:23.217+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: entering follower state: follower="Node at ee56d808-044e-5016-7ae3-7840711d323f [Follower]" leader= testing.go:149: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: entering follower state: follower="Node at ee56d808-044e-5016-7ae3-7840711d323f [Follower]" leader= testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [WARN] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [WARN] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering candidate state: node="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Candidate]" term=2 testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering candidate state: node="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Candidate]" term=2 testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: election won: tally=2 testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: election won: tally=2 testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering leader state: leader="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Leader]" testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering leader state: leader="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Leader]" testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: added peer, starting replication: peer=server-ee56d808-044e-5016-7ae3-7840711d323f testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: added peer, starting replication: peer=server-ee56d808-044e-5016-7ae3-7840711d323f testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: pipelining replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: pipelining replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.383+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:23.278537431 +1400 +14 m=+20.081119046 (1 nodes), 60.948019ms from start of poll, 61.760229ms from cluster start. Timeout at 2023-05-16 04:06:23.383737832 +1400 +14 m=+20.186319487, 105.200441ms after stability testing.go:149: cluster: 2023-05-16T04:06:23.383+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:23.278537431 +1400 +14 m=+20.081119046 (1 nodes), 60.948019ms from start of poll, 61.760229ms from cluster start. Timeout at 2023-05-16 04:06:23.383737832 +1400 +14 m=+20.186319487, 105.200441ms after stability testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.383+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-05-16T04:06:23.383+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.489+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:23.384060536 +1400 +14 m=+20.186642171 (1 nodes), 2.74µs from start of poll, 167.283354ms from cluster start. Timeout at 2023-05-16 04:06:23.48944092 +1400 +14 m=+20.292022555, 105.380384ms after stability testing.go:149: cluster: 2023-05-16T04:06:23.489+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:23.384060536 +1400 +14 m=+20.186642171 (1 nodes), 2.74µs from start of poll, 167.283354ms from cluster start. Timeout at 2023-05-16 04:06:23.48944092 +1400 +14 m=+20.292022555, 105.380384ms after stability testLoggerAdapter verbose: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.501+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" testing.go:149: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.501+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.501+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: aborting pipeline replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.501+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: aborting pipeline replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 20.344s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_arm64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: user script /srv/workspace/pbuilder/12871/tmp/hooks/B01_cleanup starting I: user script /srv/workspace/pbuilder/12871/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/12871 and its subdirectories I: Current time: Tue May 16 04:06:36 +14 2023 I: pbuilder-time-stamp: 1684159596