I: pbuilder: network access will be disabled during build
I: Current time: Tue Jan  9 00:29:30 +14 2024
I: pbuilder-time-stamp: 1704709770
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/bookworm-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz]
I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz]
I: Extracting source
gpgv: Signature made Sun Jan 19 07:24:28 2020 
gpgv:                using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.1.2-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch
dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch
dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch
dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch
dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch
I: using fakeroot in build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/D01_modify_environment starting
debug: Running on ionos1-amd64.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 Jan  8 10:30 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="15" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
  BASH_VERSION='5.2.15(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=amd64
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=15 '
  DIRSTACK=()
  DISTRIBUTION=bookworm
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=x86_64
  HOST_ARCH=amd64
  IFS=' 	
  '
  INVOCATION_ID=1cdb63706b6348dd87576a53b76f11d8
  LANG=C
  LANGUAGE=et_EE:et
  LC_ALL=C
  MACHTYPE=x86_64-pc-linux-gnu
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnu
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=1225125
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/pbuilderrc_B0r6 --distribution bookworm --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.5KHihXX9/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.1.2-1.dsc'
  SUDO_GID=110
  SUDO_UID=105
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
  http_proxy=http://78.137.99.97:3128
I: uname -a
  Linux i-capture-the-hostname 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
I: ls -l /bin
  total 5632
  -rwxr-xr-x 1 root root 1265648 Apr 23  2023 bash
  -rwxr-xr-x 3 root root   39224 Sep 19  2022 bunzip2
  -rwxr-xr-x 3 root root   39224 Sep 19  2022 bzcat
  lrwxrwxrwx 1 root root       6 Sep 19  2022 bzcmp -> bzdiff
  -rwxr-xr-x 1 root root    2225 Sep 19  2022 bzdiff
  lrwxrwxrwx 1 root root       6 Sep 19  2022 bzegrep -> bzgrep
  -rwxr-xr-x 1 root root    4893 Nov 27  2021 bzexe
  lrwxrwxrwx 1 root root       6 Sep 19  2022 bzfgrep -> bzgrep
  -rwxr-xr-x 1 root root    3775 Sep 19  2022 bzgrep
  -rwxr-xr-x 3 root root   39224 Sep 19  2022 bzip2
  -rwxr-xr-x 1 root root   14568 Sep 19  2022 bzip2recover
  lrwxrwxrwx 1 root root       6 Sep 19  2022 bzless -> bzmore
  -rwxr-xr-x 1 root root    1297 Sep 19  2022 bzmore
  -rwxr-xr-x 1 root root   44016 Sep 20  2022 cat
  -rwxr-xr-x 1 root root   68656 Sep 20  2022 chgrp
  -rwxr-xr-x 1 root root   64496 Sep 20  2022 chmod
  -rwxr-xr-x 1 root root   72752 Sep 20  2022 chown
  -rwxr-xr-x 1 root root  151152 Sep 20  2022 cp
  -rwxr-xr-x 1 root root  125640 Jan  5  2023 dash
  -rwxr-xr-x 1 root root  121904 Sep 20  2022 date
  -rwxr-xr-x 1 root root   89240 Sep 20  2022 dd
  -rwxr-xr-x 1 root root  102200 Sep 20  2022 df
  -rwxr-xr-x 1 root root  151344 Sep 20  2022 dir
  -rwxr-xr-x 1 root root   88656 Mar 23  2023 dmesg
  lrwxrwxrwx 1 root root       8 Dec 19  2022 dnsdomainname -> hostname
  lrwxrwxrwx 1 root root       8 Dec 19  2022 domainname -> hostname
  -rwxr-xr-x 1 root root   43856 Sep 20  2022 echo
  -rwxr-xr-x 1 root root      41 Jan 24  2023 egrep
  -rwxr-xr-x 1 root root   35664 Sep 20  2022 false
  -rwxr-xr-x 1 root root      41 Jan 24  2023 fgrep
  -rwxr-xr-x 1 root root   85600 Mar 23  2023 findmnt
  -rwsr-xr-x 1 root root   35128 Mar 23  2023 fusermount
  -rwxr-xr-x 1 root root  203152 Jan 24  2023 grep
  -rwxr-xr-x 2 root root    2346 Apr 10  2022 gunzip
  -rwxr-xr-x 1 root root    6447 Apr 10  2022 gzexe
  -rwxr-xr-x 1 root root   98136 Apr 10  2022 gzip
  -rwxr-xr-x 1 root root   22680 Dec 19  2022 hostname
  -rwxr-xr-x 1 root root   72824 Sep 20  2022 ln
  -rwxr-xr-x 1 root root   53024 Mar 23  2023 login
  -rwxr-xr-x 1 root root  151344 Sep 20  2022 ls
  -rwxr-xr-x 1 root root  207168 Mar 23  2023 lsblk
  -rwxr-xr-x 1 root root   97552 Sep 20  2022 mkdir
  -rwxr-xr-x 1 root root   72912 Sep 20  2022 mknod
  -rwxr-xr-x 1 root root   43952 Sep 20  2022 mktemp
  -rwxr-xr-x 1 root root   59712 Mar 23  2023 more
  -rwsr-xr-x 1 root root   59704 Mar 23  2023 mount
  -rwxr-xr-x 1 root root   18744 Mar 23  2023 mountpoint
  -rwxr-xr-x 1 root root  142968 Sep 20  2022 mv
  lrwxrwxrwx 1 root root       8 Dec 19  2022 nisdomainname -> hostname
  lrwxrwxrwx 1 root root      14 Apr  3  2023 pidof -> /sbin/killall5
  -rwxr-xr-x 1 root root   43952 Sep 20  2022 pwd
  lrwxrwxrwx 1 root root       4 Apr 23  2023 rbash -> bash
  -rwxr-xr-x 1 root root   52112 Sep 20  2022 readlink
  -rwxr-xr-x 1 root root   72752 Sep 20  2022 rm
  -rwxr-xr-x 1 root root   56240 Sep 20  2022 rmdir
  -rwxr-xr-x 1 root root   27560 Jul 28 23:46 run-parts
  -rwxr-xr-x 1 root root  126424 Jan  5  2023 sed
  lrwxrwxrwx 1 root root       9 Jan  8 10:30 sh -> /bin/bash
  -rwxr-xr-x 1 root root   43888 Sep 20  2022 sleep
  -rwxr-xr-x 1 root root   85008 Sep 20  2022 stty
  -rwsr-xr-x 1 root root   72000 Mar 23  2023 su
  -rwxr-xr-x 1 root root   39824 Sep 20  2022 sync
  -rwxr-xr-x 1 root root  531984 Apr  6  2023 tar
  -rwxr-xr-x 1 root root   14520 Jul 28 23:46 tempfile
  -rwxr-xr-x 1 root root  109616 Sep 20  2022 touch
  -rwxr-xr-x 1 root root   35664 Sep 20  2022 true
  -rwxr-xr-x 1 root root   14568 Mar 23  2023 ulockmgr_server
  -rwsr-xr-x 1 root root   35128 Mar 23  2023 umount
  -rwxr-xr-x 1 root root   43888 Sep 20  2022 uname
  -rwxr-xr-x 2 root root    2346 Apr 10  2022 uncompress
  -rwxr-xr-x 1 root root  151344 Sep 20  2022 vdir
  -rwxr-xr-x 1 root root   72024 Mar 23  2023 wdctl
  lrwxrwxrwx 1 root root       8 Dec 19  2022 ypdomainname -> hostname
  -rwxr-xr-x 1 root root    1984 Apr 10  2022 zcat
  -rwxr-xr-x 1 root root    1678 Apr 10  2022 zcmp
  -rwxr-xr-x 1 root root    6460 Apr 10  2022 zdiff
  -rwxr-xr-x 1 root root      29 Apr 10  2022 zegrep
  -rwxr-xr-x 1 root root      29 Apr 10  2022 zfgrep
  -rwxr-xr-x 1 root root    2081 Apr 10  2022 zforce
  -rwxr-xr-x 1 root root    8103 Apr 10  2022 zgrep
  -rwxr-xr-x 1 root root    2206 Apr 10  2022 zless
  -rwxr-xr-x 1 root root    1842 Apr 10  2022 zmore
  -rwxr-xr-x 1 root root    4577 Apr 10  2022 znew
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: amd64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~)
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 18148 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however:
  Package debhelper is not installed.
 pbuilder-satisfydepends-dummy depends on dh-golang; however:
  Package dh-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.19-go{a} golang-1.19-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-3-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-diff-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 
0 packages upgraded, 98 newly installed, 0 to remove and 0 not upgraded.
Need to get 118 MB of archives. After unpacking 785 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian bookworm/main amd64 sensible-utils all 0.0.17+nmu1 [19.0 kB]
Get: 2 http://deb.debian.org/debian bookworm/main amd64 libmagic-mgc amd64 1:5.44-3 [305 kB]
Get: 3 http://deb.debian.org/debian bookworm/main amd64 libmagic1 amd64 1:5.44-3 [104 kB]
Get: 4 http://deb.debian.org/debian bookworm/main amd64 file amd64 1:5.44-3 [42.5 kB]
Get: 5 http://deb.debian.org/debian bookworm/main amd64 gettext-base amd64 0.21-12 [160 kB]
Get: 6 http://deb.debian.org/debian bookworm/main amd64 libuchardet0 amd64 0.0.7-1 [67.8 kB]
Get: 7 http://deb.debian.org/debian bookworm/main amd64 groff-base amd64 1.22.4-10 [916 kB]
Get: 8 http://deb.debian.org/debian bookworm/main amd64 bsdextrautils amd64 2.38.1-5+b1 [86.6 kB]
Get: 9 http://deb.debian.org/debian bookworm/main amd64 libpipeline1 amd64 1.5.7-1 [38.5 kB]
Get: 10 http://deb.debian.org/debian bookworm/main amd64 man-db amd64 2.11.2-2 [1386 kB]
Get: 11 http://deb.debian.org/debian bookworm/main amd64 m4 amd64 1.4.19-3 [287 kB]
Get: 12 http://deb.debian.org/debian bookworm/main amd64 autoconf all 2.71-3 [332 kB]
Get: 13 http://deb.debian.org/debian bookworm/main amd64 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian bookworm/main amd64 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian bookworm/main amd64 autopoint all 0.21-12 [495 kB]
Get: 16 http://deb.debian.org/debian bookworm/main amd64 libdebhelper-perl all 13.11.4 [81.2 kB]
Get: 17 http://deb.debian.org/debian bookworm/main amd64 libtool all 2.4.7-5 [517 kB]
Get: 18 http://deb.debian.org/debian bookworm/main amd64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian bookworm/main amd64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian bookworm/main amd64 libsub-override-perl all 0.09-4 [9304 B]
Get: 21 http://deb.debian.org/debian bookworm/main amd64 libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB]
Get: 22 http://deb.debian.org/debian bookworm/main amd64 dh-strip-nondeterminism all 1.13.1-1 [8620 B]
Get: 23 http://deb.debian.org/debian bookworm/main amd64 libelf1 amd64 0.188-2.1 [174 kB]
Get: 24 http://deb.debian.org/debian bookworm/main amd64 dwz amd64 0.15-1 [109 kB]
Get: 25 http://deb.debian.org/debian bookworm/main amd64 libicu72 amd64 72.1-3 [9376 kB]
Get: 26 http://deb.debian.org/debian bookworm/main amd64 libxml2 amd64 2.9.14+dfsg-1.3~deb12u1 [687 kB]
Get: 27 http://deb.debian.org/debian bookworm/main amd64 gettext amd64 0.21-12 [1300 kB]
Get: 28 http://deb.debian.org/debian bookworm/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian bookworm/main amd64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian bookworm/main amd64 debhelper all 13.11.4 [942 kB]
Get: 31 http://deb.debian.org/debian bookworm/main amd64 dh-golang all 1.59 [27.8 kB]
Get: 32 http://deb.debian.org/debian bookworm/main amd64 golang-1.19-src all 1.19.8-2 [18.3 MB]
Get: 33 http://deb.debian.org/debian bookworm/main amd64 golang-1.19-go amd64 1.19.8-2 [62.7 MB]
Get: 34 http://deb.debian.org/debian bookworm/main amd64 golang-src all 2:1.19~1 [5756 B]
Get: 35 http://deb.debian.org/debian bookworm/main amd64 golang-go amd64 2:1.19~1 [45.0 kB]
Get: 36 http://deb.debian.org/debian bookworm/main amd64 golang-any amd64 2:1.19~1 [5876 B]
Get: 37 http://deb.debian.org/debian bookworm/main amd64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B]
Get: 38 http://deb.debian.org/debian bookworm/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-sys-dev all 0.3.0-1 [393 kB]
Get: 41 http://deb.debian.org/debian bookworm/main amd64 golang-github-mattn-go-isatty-dev all 0.0.17-1 [6016 B]
Get: 42 http://deb.debian.org/debian bookworm/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian bookworm/main amd64 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB]
Get: 44 http://deb.debian.org/debian bookworm/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian bookworm/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian bookworm/main amd64 golang-github-stretchr-objx-dev all 0.5.0-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian bookworm/main amd64 golang-github-stretchr-testify-dev all 1.8.1-1 [67.9 kB]
Get: 49 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian bookworm/main amd64 golang-github-pkg-errors-dev all 0.9.1-2 [13.2 kB]
Get: 52 http://deb.debian.org/debian bookworm/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian bookworm/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian bookworm/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.2-2 [8480 B]
Get: 56 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 57 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 58 http://deb.debian.org/debian bookworm/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 59 http://deb.debian.org/debian bookworm/main amd64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 60 http://deb.debian.org/debian bookworm/main amd64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B]
Get: 61 http://deb.debian.org/debian bookworm/main amd64 golang-github-golang-protobuf-1-3-dev all 1.3.5-4 [234 kB]
Get: 62 http://deb.debian.org/debian bookworm/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 63 http://deb.debian.org/debian bookworm/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 64 http://deb.debian.org/debian bookworm/main amd64 golang-github-json-iterator-go-dev all 1.1.12-1 [62.2 kB]
Get: 65 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-client-model-dev all 0.3.0-3 [10.5 kB]
Get: 66 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 67 http://deb.debian.org/debian bookworm/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 68 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-text-dev all 0.7.0-1 [3863 kB]
Get: 69 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-net-dev all 1:0.7.0+dfsg-1 [731 kB]
Get: 70 http://deb.debian.org/debian bookworm/main amd64 golang-github-aws-aws-sdk-go-dev all 1.44.133-1 [10.5 MB]
Get: 71 http://deb.debian.org/debian bookworm/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 72 http://deb.debian.org/debian bookworm/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 73 http://deb.debian.org/debian bookworm/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 74 http://deb.debian.org/debian bookworm/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 75 http://deb.debian.org/debian bookworm/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 76 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-oauth2-dev all 0.3.0-1 [35.0 kB]
Get: 77 http://deb.debian.org/debian bookworm/main amd64 golang-google-protobuf-dev all 1.28.1-3 [542 kB]
Get: 78 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 79 http://deb.debian.org/debian bookworm/main amd64 golang-protobuf-extensions-dev all 1.0.1-3 [29.5 kB]
Get: 80 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-common-dev all 0.39.0-2 [110 kB]
Get: 81 http://deb.debian.org/debian bookworm/main amd64 golang-golang-x-sync-dev all 0.1.0-1 [19.7 kB]
Get: 82 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-procfs-dev all 0.8.0-3 [148 kB]
Get: 83 http://deb.debian.org/debian bookworm/main amd64 golang-github-prometheus-client-golang-dev all 1.14.0-3 [174 kB]
Get: 84 http://deb.debian.org/debian bookworm/main amd64 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB]
Get: 85 http://deb.debian.org/debian bookworm/main amd64 golang-github-creack-pty-dev all 1.1.18-1 [9796 B]
Get: 86 http://deb.debian.org/debian bookworm/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 87 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 88 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 89 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10 [20.3 kB]
Get: 90 http://deb.debian.org/debian bookworm/main amd64 libsasl2-2 amd64 2.1.28+dfsg-10 [59.7 kB]
Get: 91 http://deb.debian.org/debian bookworm/main amd64 libsasl2-dev amd64 2.1.28+dfsg-10 [235 kB]
Get: 92 http://deb.debian.org/debian bookworm/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 93 http://deb.debian.org/debian bookworm/main amd64 golang-github-pkg-diff-dev all 0.0~git20210226.20ebb0f-1 [93.6 kB]
Get: 94 http://deb.debian.org/debian bookworm/main amd64 golang-github-rogpeppe-go-internal-dev all 1.9.0-1 [117 kB]
Get: 95 http://deb.debian.org/debian bookworm/main amd64 golang-github-kr-pretty-dev all 0.3.0-1 [14.9 kB]
Get: 96 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 97 http://deb.debian.org/debian bookworm/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 98 http://deb.debian.org/debian bookworm/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 118 MB in 12s (10.1 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 18148 files and directories currently installed.)
Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ...
Unpacking sensible-utils (0.0.17+nmu1) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../01-libmagic-mgc_1%3a5.44-3_amd64.deb ...
Unpacking libmagic-mgc (1:5.44-3) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../02-libmagic1_1%3a5.44-3_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.44-3) ...
Selecting previously unselected package file.
Preparing to unpack .../03-file_1%3a5.44-3_amd64.deb ...
Unpacking file (1:5.44-3) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../04-gettext-base_0.21-12_amd64.deb ...
Unpacking gettext-base (0.21-12) ...
Selecting previously unselected package libuchardet0:amd64.
Preparing to unpack .../05-libuchardet0_0.0.7-1_amd64.deb ...
Unpacking libuchardet0:amd64 (0.0.7-1) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../06-groff-base_1.22.4-10_amd64.deb ...
Unpacking groff-base (1.22.4-10) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../07-bsdextrautils_2.38.1-5+b1_amd64.deb ...
Unpacking bsdextrautils (2.38.1-5+b1) ...
Selecting previously unselected package libpipeline1:amd64.
Preparing to unpack .../08-libpipeline1_1.5.7-1_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.7-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../09-man-db_2.11.2-2_amd64.deb ...
Unpacking man-db (2.11.2-2) ...
Selecting previously unselected package m4.
Preparing to unpack .../10-m4_1.4.19-3_amd64.deb ...
Unpacking m4 (1.4.19-3) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../11-autoconf_2.71-3_all.deb ...
Unpacking autoconf (2.71-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../12-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../13-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../14-autopoint_0.21-12_all.deb ...
Unpacking autopoint (0.21-12) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../15-libdebhelper-perl_13.11.4_all.deb ...
Unpacking libdebhelper-perl (13.11.4) ...
Selecting previously unselected package libtool.
Preparing to unpack .../16-libtool_2.4.7-5_all.deb ...
Unpacking libtool (2.4.7-5) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../17-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../18-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libsub-override-perl.
Preparing to unpack .../19-libsub-override-perl_0.09-4_all.deb ...
Unpacking libsub-override-perl (0.09-4) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../20-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../21-dh-strip-nondeterminism_1.13.1-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.13.1-1) ...
Selecting previously unselected package libelf1:amd64.
Preparing to unpack .../22-libelf1_0.188-2.1_amd64.deb ...
Unpacking libelf1:amd64 (0.188-2.1) ...
Selecting previously unselected package dwz.
Preparing to unpack .../23-dwz_0.15-1_amd64.deb ...
Unpacking dwz (0.15-1) ...
Selecting previously unselected package libicu72:amd64.
Preparing to unpack .../24-libicu72_72.1-3_amd64.deb ...
Unpacking libicu72:amd64 (72.1-3) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../25-libxml2_2.9.14+dfsg-1.3~deb12u1_amd64.deb ...
Unpacking libxml2:amd64 (2.9.14+dfsg-1.3~deb12u1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../26-gettext_0.21-12_amd64.deb ...
Unpacking gettext (0.21-12) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../27-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../28-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../29-debhelper_13.11.4_all.deb ...
Unpacking debhelper (13.11.4) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../30-dh-golang_1.59_all.deb ...
Unpacking dh-golang (1.59) ...
Selecting previously unselected package golang-1.19-src.
Preparing to unpack .../31-golang-1.19-src_1.19.8-2_all.deb ...
Unpacking golang-1.19-src (1.19.8-2) ...
Selecting previously unselected package golang-1.19-go.
Preparing to unpack .../32-golang-1.19-go_1.19.8-2_amd64.deb ...
Unpacking golang-1.19-go (1.19.8-2) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../33-golang-src_2%3a1.19~1_all.deb ...
Unpacking golang-src (2:1.19~1) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../34-golang-go_2%3a1.19~1_amd64.deb ...
Unpacking golang-go:amd64 (2:1.19~1) ...
Selecting previously unselected package golang-any:amd64.
Preparing to unpack .../35-golang-any_2%3a1.19~1_amd64.deb ...
Unpacking golang-any:amd64 (2:1.19~1) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../36-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../37-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../38-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../39-golang-golang-x-sys-dev_0.3.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.3.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../40-golang-github-mattn-go-isatty-dev_0.0.17-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.17-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../41-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../42-golang-github-fatih-color-dev_1.7.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.7.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../43-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../44-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../45-golang-github-stretchr-objx-dev_0.5.0-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../46-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../47-golang-github-stretchr-testify-dev_1.8.1-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.8.1-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../48-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../49-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../50-golang-github-pkg-errors-dev_0.9.1-2_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-2) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../51-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../52-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../53-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../54-golang-github-hashicorp-go-uuid-dev_1.0.2-2_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../55-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../56-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../57-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../58-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../59-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ...
Selecting previously unselected package golang-github-golang-protobuf-1-3-dev.
Preparing to unpack .../60-golang-github-golang-protobuf-1-3-dev_1.3.5-4_all.deb ...
Unpacking golang-github-golang-protobuf-1-3-dev (1.3.5-4) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../61-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../62-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../63-golang-github-json-iterator-go-dev_1.1.12-1_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-1) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../64-golang-github-prometheus-client-model-dev_0.3.0-3_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.3.0-3) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../65-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../66-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../67-golang-golang-x-text-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.7.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../68-golang-golang-x-net-dev_1%3a0.7.0+dfsg-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.7.0+dfsg-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../69-golang-github-aws-aws-sdk-go-dev_1.44.133-1_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.44.133-1) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../70-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../71-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../72-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../73-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../74-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../75-golang-golang-x-oauth2-dev_0.3.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.3.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../76-golang-google-protobuf-dev_1.28.1-3_all.deb ...
Unpacking golang-google-protobuf-dev (1.28.1-3) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../77-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-protobuf-extensions-dev.
Preparing to unpack .../78-golang-protobuf-extensions-dev_1.0.1-3_all.deb ...
Unpacking golang-protobuf-extensions-dev (1.0.1-3) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../79-golang-github-prometheus-common-dev_0.39.0-2_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.39.0-2) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../80-golang-golang-x-sync-dev_0.1.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.1.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../81-golang-github-prometheus-procfs-dev_0.8.0-3_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.8.0-3) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../82-golang-github-prometheus-client-golang-dev_1.14.0-3_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.14.0-3) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../83-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../84-golang-github-creack-pty-dev_1.1.18-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.18-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../85-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../86-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../87-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../88-libsasl2-modules-db_2.1.28+dfsg-10_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg-10) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../89-libsasl2-2_2.1.28+dfsg-10_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg-10) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../90-libsasl2-dev_2.1.28+dfsg-10_amd64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg-10) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../91-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-github-pkg-diff-dev.
Preparing to unpack .../92-golang-github-pkg-diff-dev_0.0~git20210226.20ebb0f-1_all.deb ...
Unpacking golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../93-golang-github-rogpeppe-go-internal-dev_1.9.0-1_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../94-golang-github-kr-pretty-dev_0.3.0-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.0-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../95-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../96-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../97-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:amd64 (1.5.7-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-2) ...
Setting up libicu72:amd64 (72.1-3) ...
Setting up bsdextrautils (2.38.1-5+b1) ...
Setting up golang-github-creack-pty-dev (1.1.18-1) ...
Setting up golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.0-1) ...
Setting up libmagic-mgc (1:5.44-3) ...
Setting up golang-github-pkg-errors-dev (0.9.1-2) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.11.4) ...
Setting up golang-golang-x-sys-dev (0.3.0-1) ...
Setting up libmagic1:amd64 (1:5.44-3) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.21-12) ...
Setting up golang-1.19-src (1.19.8-2) ...
Setting up m4 (1.4.19-3) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-github-golang-protobuf-1-3-dev (1.3.5-4) ...
Setting up file (1:5.44-3) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg-10) ...
Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.28.1-3) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.8.1-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-golang-x-sync-dev (0.1.0-1) ...
Setting up autopoint (0.21-12) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg-10) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.71-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up sensible-utils (0.0.17+nmu1) ...
Setting up golang-golang-x-text-dev (0.7.0-1) ...
Setting up libuchardet0:amd64 (0.0.7-1) ...
Setting up golang-github-prometheus-procfs-dev (0.8.0-3) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up libsub-override-perl (0.09-4) ...
Setting up golang-github-prometheus-client-model-dev (0.3.0-3) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.9.0-1) ...
Setting up golang-src (2:1.19~1) ...
Setting up libelf1:amd64 (0.188-2.1) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:amd64 (2.9.14+dfsg-1.3~deb12u1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up golang-1.19-go (1.19.8-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up golang-github-kr-pretty-dev (0.3.0-1) ...
Setting up libfile-stripnondeterminism-perl (1.13.1-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.17-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up gettext (0.21-12) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.4.7-5) ...
Setting up golang-go:amd64 (2:1.19~1) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-protobuf-extensions-dev (1.0.1-3) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-any:amd64 (2:1.19~1) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg-10) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-golang-x-net-dev (1:0.7.0+dfsg-1) ...
Setting up dh-strip-nondeterminism (1.13.1-1) ...
Setting up dwz (0.15-1) ...
Setting up groff-base (1.22.4-10) ...
Setting up golang-golang-x-oauth2-dev (0.3.0-1) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.44.133-1) ...
Setting up golang-github-fatih-color-dev (1.7.0-1) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up man-db (2.11.2-2) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up debhelper (13.11.4) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.59) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.39.0-2) ...
Setting up golang-github-prometheus-client-golang-dev (1.14.0-3) ...
Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ...
Processing triggers for libc-bin (2.36-9+deb12u3) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
Reading package lists...
Building dependency tree...
Reading state information...
fakeroot is already the newest version (1.31-1.2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
I: Building the package
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for bookworm
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.1.2-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.1.2-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Dmitry Smirnov <onlyjob@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
 fakeroot debian/rules clean
dh clean --buildsystem=golang --with=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules build
dh build --buildsystem=golang --with=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go install -trimpath -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/goarch
internal/unsafeheader
internal/goexperiment
internal/goos
internal/race
runtime/internal/syscall
unicode/utf8
unicode
internal/itoa
crypto/internal/subtle
sync/atomic
container/list
math/bits
runtime/internal/atomic
internal/cpu
internal/abi
runtime/internal/math
crypto/subtle
runtime/internal/sys
crypto/internal/boring/sig
encoding
unicode/utf16
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/subtle
internal/bytealg
math
runtime
internal/reflectlite
sync
internal/testlog
internal/singleflight
internal/sysinfo
math/rand
runtime/cgo
errors
sort
internal/oserror
path
github.com/hashicorp/golang-lru/simplelru
io
internal/safefilepath
vendor/golang.org/x/net/dns/dnsmessage
strconv
crypto/internal/nistec/fiat
syscall
hash
bytes
crypto/internal/randutil
strings
hash/crc32
hash/crc64
crypto
crypto/rc4
reflect
net/http/internal/ascii
regexp/syntax
html
github.com/hashicorp/go-immutable-radix
vendor/golang.org/x/text/transform
bufio
regexp
time
internal/syscall/execenv
internal/syscall/unix
context
internal/poll
io/fs
encoding/binary
internal/fmtsort
embed
crypto/internal/edwards25519/field
crypto/md5
vendor/golang.org/x/crypto/internal/poly1305
vendor/golang.org/x/crypto/curve25519/internal/field
encoding/base64
crypto/cipher
golang.org/x/sys/unix
crypto/internal/nistec
os
encoding/pem
crypto/internal/edwards25519
crypto/internal/boring
vendor/golang.org/x/crypto/chacha20
crypto/des
crypto/hmac
crypto/sha256
crypto/aes
crypto/sha512
crypto/sha1
vendor/golang.org/x/crypto/hkdf
internal/godebug
os/signal
path/filepath
io/ioutil
fmt
internal/intern
net/netip
vendor/golang.org/x/sys/cpu
vendor/golang.org/x/crypto/chacha20poly1305
net
math/big
log
encoding/hex
vendor/golang.org/x/crypto/curve25519
net/url
mime/quotedprintable
net/http/internal
mime
compress/flate
vendor/golang.org/x/net/http2/hpack
encoding/json
encoding/gob
vendor/golang.org/x/text/unicode/norm
go/token
github.com/mattn/go-isatty
text/template/parse
flag
runtime/debug
vendor/golang.org/x/text/unicode/bidi
github.com/mattn/go-colorable
github.com/fatih/color
runtime/trace
compress/gzip
vendor/golang.org/x/text/secure/bidirule
testing
text/template
github.com/hashicorp/go-hclog
crypto/rand
crypto/internal/boring/bbig
crypto/elliptic
crypto/dsa
encoding/asn1
vendor/golang.org/x/net/idna
crypto/ed25519
crypto/rsa
html/template
crypto/x509/pkix
vendor/golang.org/x/crypto/cryptobyte
crypto/ecdsa
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   dh_auto_test -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go test -vet=off -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
    testing.go:153: 2024-01-08T10:37:51.853Z [INFO]  creating new snapshot: path=/tmp/raft4256912193/raft1767633831/snapshots/3-10-1704710271853.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
    testing.go:153: 2024-01-08T10:37:51.854Z [INFO]  creating new snapshot: path=/tmp/raft2187074017/snapshots/3-10-1704710271854.tmp
    testing.go:153: 2024-01-08T10:37:51.854Z [WARN]  found temporary snapshot: name=3-10-1704710271854.tmp
--- PASS: TestFileSS_CreateSnapshot (0.00s)
=== RUN   TestFileSS_CancelSnapshot
    testing.go:153: 2024-01-08T10:37:51.855Z [INFO]  creating new snapshot: path=/tmp/raft2847088997/snapshots/3-10-1704710271855.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
    testing.go:153: 2024-01-08T10:37:51.855Z [INFO]  creating new snapshot: path=/tmp/raft1856832098/snapshots/3-10-1704710271855.tmp
    testing.go:153: 2024-01-08T10:37:51.855Z [INFO]  creating new snapshot: path=/tmp/raft1856832098/snapshots/3-11-1704710271855.tmp
    testing.go:153: 2024-01-08T10:37:51.856Z [INFO]  creating new snapshot: path=/tmp/raft1856832098/snapshots/3-12-1704710271856.tmp
    testing.go:153: 2024-01-08T10:37:51.856Z [INFO]  reaping snapshot: path=/tmp/raft1856832098/snapshots/3-10-1704710271855
    testing.go:153: 2024-01-08T10:37:51.856Z [INFO]  creating new snapshot: path=/tmp/raft1856832098/snapshots/3-13-1704710271856.tmp
    testing.go:153: 2024-01-08T10:37:51.862Z [INFO]  reaping snapshot: path=/tmp/raft1856832098/snapshots/3-11-1704710271855
    testing.go:153: 2024-01-08T10:37:51.862Z [INFO]  creating new snapshot: path=/tmp/raft1856832098/snapshots/3-14-1704710271862.tmp
    testing.go:153: 2024-01-08T10:37:51.863Z [INFO]  reaping snapshot: path=/tmp/raft1856832098/snapshots/3-12-1704710271856
--- PASS: TestFileSS_Retention (0.01s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
    testing.go:153: 2024-01-08T10:37:51.864Z [INFO]  creating new snapshot: path=/tmp/raft3062085856/snapshots/5-130350-1704710271864.tmp
    testing.go:153: 2024-01-08T10:37:51.864Z [INFO]  creating new snapshot: path=/tmp/raft3062085856/snapshots/36-204917-1704710271864.tmp
--- PASS: TestFileSS_Ordering (0.00s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.01s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.01s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
    net_transport_test.go:339: Disabled unreliable test
--- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.01s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.01s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.01s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:51.969Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:51.977Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:51.989Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:52.013Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:52.055Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:52.136Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:52.298Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:52.618Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:53.262Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:54.270Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:718: countingWriter: 2024-01-08T10:37:55.274Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.01s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.01s)
=== RUN   TestRaft_StartStop
testLoggerAdapter verbose: server-afc95e39-c2f8-b04c-3bbb-18a71963eab1: 2024-01-08T10:37:55.983Z [INFO]  server-afc95e39-c2f8-b04c-3bbb-18a71963eab1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afc95e39-c2f8-b04c-3bbb-18a71963eab1 Address:afc95e39-c2f8-b04c-3bbb-18a71963eab1}]"
    testing.go:149: server-afc95e39-c2f8-b04c-3bbb-18a71963eab1: 2024-01-08T10:37:55.983Z [INFO]  server-afc95e39-c2f8-b04c-3bbb-18a71963eab1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afc95e39-c2f8-b04c-3bbb-18a71963eab1 Address:afc95e39-c2f8-b04c-3bbb-18a71963eab1}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
testLoggerAdapter verbose: server-53234915-4d71-e881-0775-3a6b5c7093ab: 2024-01-08T10:37:55.985Z [INFO]  server-53234915-4d71-e881-0775-3a6b5c7093ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53234915-4d71-e881-0775-3a6b5c7093ab Address:53234915-4d71-e881-0775-3a6b5c7093ab}]"
    testing.go:149: server-53234915-4d71-e881-0775-3a6b5c7093ab: 2024-01-08T10:37:55.985Z [INFO]  server-53234915-4d71-e881-0775-3a6b5c7093ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53234915-4d71-e881-0775-3a6b5c7093ab Address:53234915-4d71-e881-0775-3a6b5c7093ab}]"
--- PASS: TestRaft_AfterShutdown (0.01s)
=== RUN   TestRaft_LiveBootstrap
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:55.992Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: initial configuration: index=0 servers=[]
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:55.992Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: 2024-01-08T10:37:55.992Z [INFO]  server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: initial configuration: index=0 servers=[]
    testing.go:149: server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: 2024-01-08T10:37:55.992Z [INFO]  server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: 2024-01-08T10:37:55.993Z [INFO]  server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: initial configuration: index=0 servers=[]
    testing.go:149: server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: 2024-01-08T10:37:55.993Z [INFO]  server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:55.993Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: entering follower state: follower="Node at ee2d09f7-1dd2-dffc-9717-f0d84195b395 [Follower]" leader=
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:55.993Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: entering follower state: follower="Node at ee2d09f7-1dd2-dffc-9717-f0d84195b395 [Follower]" leader=
testLoggerAdapter verbose: server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: 2024-01-08T10:37:55.994Z [INFO]  server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: entering follower state: follower="Node at ce8ed5a8-725c-4397-6bda-bc594fbdbeaa [Follower]" leader=
testLoggerAdapter verbose: server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: 2024-01-08T10:37:55.994Z [INFO]  server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: entering follower state: follower="Node at e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 [Follower]" leader=
    testing.go:149: server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: 2024-01-08T10:37:55.994Z [INFO]  server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: entering follower state: follower="Node at ce8ed5a8-725c-4397-6bda-bc594fbdbeaa [Follower]" leader=
    testing.go:149: server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: 2024-01-08T10:37:55.994Z [INFO]  server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: entering follower state: follower="Node at e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 [Follower]" leader=
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:55.998Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:37:55.998Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.074Z [WARN]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.074Z [WARN]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.074Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: entering candidate state: node="Node at ee2d09f7-1dd2-dffc-9717-f0d84195b395 [Candidate]" term=2
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.074Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: entering candidate state: node="Node at ee2d09f7-1dd2-dffc-9717-f0d84195b395 [Candidate]" term=2
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.074Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: election won: tally=2
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.074Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: election won: tally=2
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: entering leader state: leader="Node at ee2d09f7-1dd2-dffc-9717-f0d84195b395 [Leader]"
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: entering leader state: leader="Node at ee2d09f7-1dd2-dffc-9717-f0d84195b395 [Leader]"
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: added peer, starting replication: peer=server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: added peer, starting replication: peer=server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: added peer, starting replication: peer=server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: added peer, starting replication: peer=server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782
testLoggerAdapter verbose: server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: 2024-01-08T10:37:56.075Z [WARN]  server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: failed to get previous log: previous-index=1 last-index=0 error="log not found"
    testing.go:149: server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: 2024-01-08T10:37:56.075Z [WARN]  server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782: failed to get previous log: previous-index=1 last-index=0 error="log not found"
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [WARN]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: appendEntries rejected, sending older logs: peer="{Voter server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782}" next=1
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.075Z [WARN]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: appendEntries rejected, sending older logs: peer="{Voter server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782}" next=1
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.076Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: pipelining replication: peer="{Voter server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782}"
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.076Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: pipelining replication: peer="{Voter server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782}"
testLoggerAdapter verbose: server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: 2024-01-08T10:37:56.076Z [WARN]  server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: failed to get previous log: previous-index=1 last-index=0 error="log not found"
    testing.go:149: server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: 2024-01-08T10:37:56.076Z [WARN]  server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa: failed to get previous log: previous-index=1 last-index=0 error="log not found"
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.077Z [WARN]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: appendEntries rejected, sending older logs: peer="{Voter server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa ce8ed5a8-725c-4397-6bda-bc594fbdbeaa}" next=1
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.077Z [WARN]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: appendEntries rejected, sending older logs: peer="{Voter server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa ce8ed5a8-725c-4397-6bda-bc594fbdbeaa}" next=1
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.077Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: pipelining replication: peer="{Voter server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa ce8ed5a8-725c-4397-6bda-bc594fbdbeaa}"
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.077Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: pipelining replication: peer="{Voter server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa ce8ed5a8-725c-4397-6bda-bc594fbdbeaa}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.186Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:56.078142024 +0000 UTC m=+4.229936175 (2 nodes), 79.602853ms from start of poll, 85.983975ms from cluster start. Timeout at 2024-01-08 10:37:56.186516728 +0000 UTC m=+4.338310876, 108.374701ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:56.186Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:56.078142024 +0000 UTC m=+4.229936175 (2 nodes), 79.602853ms from start of poll, 85.983975ms from cluster start. Timeout at 2024-01-08 10:37:56.186516728 +0000 UTC m=+4.338310876, 108.374701ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.186Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:56.186Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.294Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.187110724 +0000 UTC m=+4.338904838 (1 nodes), 2.747µs from start of poll, 194.952638ms from cluster start. Timeout at 2024-01-08 10:37:56.294288044 +0000 UTC m=+4.446082170, 107.177332ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:56.294Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.187110724 +0000 UTC m=+4.338904838 (1 nodes), 2.747µs from start of poll, 194.952638ms from cluster start. Timeout at 2024-01-08 10:37:56.294288044 +0000 UTC m=+4.446082170, 107.177332ms after stability
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.310Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: aborting pipeline replication: peer="{Voter server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782}"
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.310Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: aborting pipeline replication: peer="{Voter server-e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782 e4dd4a3f-a69e-6d69-abe5-9dec2fc5d782}"
testLoggerAdapter verbose: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.310Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: aborting pipeline replication: peer="{Voter server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa ce8ed5a8-725c-4397-6bda-bc594fbdbeaa}"
    testing.go:149: server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: 2024-01-08T10:37:56.310Z [INFO]  server-ee2d09f7-1dd2-dffc-9717-f0d84195b395: aborting pipeline replication: peer="{Voter server-ce8ed5a8-725c-4397-6bda-bc594fbdbeaa ce8ed5a8-725c-4397-6bda-bc594fbdbeaa}"
--- PASS: TestRaft_LiveBootstrap (0.32s)
=== RUN   TestRaft_RecoverCluster_NoState
testLoggerAdapter verbose: server-64024754-4ad1-8e74-e0ee-02947c9a98cd: 2024-01-08T10:37:56.313Z [INFO]  server-64024754-4ad1-8e74-e0ee-02947c9a98cd: initial configuration: index=0 servers=[]
    testing.go:149: server-64024754-4ad1-8e74-e0ee-02947c9a98cd: 2024-01-08T10:37:56.313Z [INFO]  server-64024754-4ad1-8e74-e0ee-02947c9a98cd: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.01s)
=== RUN   TestRaft_RecoverCluster
    raft_test.go:119: Disabled unreliable test
--- SKIP: TestRaft_RecoverCluster (0.00s)
=== RUN   TestRaft_HasExistingState
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.319Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52ec5894-fbca-fa47-0487-be8f97dfa858 Address:52ec5894-fbca-fa47-0487-be8f97dfa858} {Suffrage:Voter ID:server-85ba9891-6b1c-fefe-6043-c3341ded65ee Address:85ba9891-6b1c-fefe-6043-c3341ded65ee}]"
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.319Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52ec5894-fbca-fa47-0487-be8f97dfa858 Address:52ec5894-fbca-fa47-0487-be8f97dfa858} {Suffrage:Voter ID:server-85ba9891-6b1c-fefe-6043-c3341ded65ee Address:85ba9891-6b1c-fefe-6043-c3341ded65ee}]"
testLoggerAdapter verbose: server-85ba9891-6b1c-fefe-6043-c3341ded65ee: 2024-01-08T10:37:56.320Z [INFO]  server-85ba9891-6b1c-fefe-6043-c3341ded65ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52ec5894-fbca-fa47-0487-be8f97dfa858 Address:52ec5894-fbca-fa47-0487-be8f97dfa858} {Suffrage:Voter ID:server-85ba9891-6b1c-fefe-6043-c3341ded65ee Address:85ba9891-6b1c-fefe-6043-c3341ded65ee}]"
    testing.go:149: server-85ba9891-6b1c-fefe-6043-c3341ded65ee: 2024-01-08T10:37:56.320Z [INFO]  server-85ba9891-6b1c-fefe-6043-c3341ded65ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52ec5894-fbca-fa47-0487-be8f97dfa858 Address:52ec5894-fbca-fa47-0487-be8f97dfa858} {Suffrage:Voter ID:server-85ba9891-6b1c-fefe-6043-c3341ded65ee Address:85ba9891-6b1c-fefe-6043-c3341ded65ee}]"
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.320Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: entering follower state: follower="Node at 52ec5894-fbca-fa47-0487-be8f97dfa858 [Follower]" leader=
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.320Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: entering follower state: follower="Node at 52ec5894-fbca-fa47-0487-be8f97dfa858 [Follower]" leader=
testLoggerAdapter verbose: server-85ba9891-6b1c-fefe-6043-c3341ded65ee: 2024-01-08T10:37:56.321Z [INFO]  server-85ba9891-6b1c-fefe-6043-c3341ded65ee: entering follower state: follower="Node at 85ba9891-6b1c-fefe-6043-c3341ded65ee [Follower]" leader=
testLoggerAdapter verbose: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.321Z [INFO]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: initial configuration: index=0 servers=[]
    testing.go:149: server-85ba9891-6b1c-fefe-6043-c3341ded65ee: 2024-01-08T10:37:56.321Z [INFO]  server-85ba9891-6b1c-fefe-6043-c3341ded65ee: entering follower state: follower="Node at 85ba9891-6b1c-fefe-6043-c3341ded65ee [Follower]" leader=
    testing.go:149: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.321Z [INFO]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.321Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:56.321Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.329Z [INFO]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: entering follower state: follower="Node at 732bb4ac-de8e-e246-f305-da93d9b517f2 [Follower]" leader=
    testing.go:149: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.329Z [INFO]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: entering follower state: follower="Node at 732bb4ac-de8e-e246-f305-da93d9b517f2 [Follower]" leader=
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.414Z [WARN]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.414Z [WARN]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.414Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: entering candidate state: node="Node at 52ec5894-fbca-fa47-0487-be8f97dfa858 [Candidate]" term=2
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.414Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: entering candidate state: node="Node at 52ec5894-fbca-fa47-0487-be8f97dfa858 [Candidate]" term=2
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: election won: tally=2
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: election won: tally=2
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: entering leader state: leader="Node at 52ec5894-fbca-fa47-0487-be8f97dfa858 [Leader]"
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: entering leader state: leader="Node at 52ec5894-fbca-fa47-0487-be8f97dfa858 [Leader]"
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: added peer, starting replication: peer=server-85ba9891-6b1c-fefe-6043-c3341ded65ee
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: added peer, starting replication: peer=server-85ba9891-6b1c-fefe-6043-c3341ded65ee
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: pipelining replication: peer="{Voter server-85ba9891-6b1c-fefe-6043-c3341ded65ee 85ba9891-6b1c-fefe-6043-c3341ded65ee}"
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.415Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: pipelining replication: peer="{Voter server-85ba9891-6b1c-fefe-6043-c3341ded65ee 85ba9891-6b1c-fefe-6043-c3341ded65ee}"
testLoggerAdapter verbose: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.425Z [WARN]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: no known peers, aborting election
    testing.go:149: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.425Z [WARN]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: no known peers, aborting election
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.522Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.416165559 +0000 UTC m=+4.567959713 (1 nodes), 94.447145ms from start of poll, 96.756469ms from cluster start. Timeout at 2024-01-08 10:37:56.521882172 +0000 UTC m=+4.673676374, 105.716661ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:56.522Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.416165559 +0000 UTC m=+4.567959713 (1 nodes), 94.447145ms from start of poll, 96.756469ms from cluster start. Timeout at 2024-01-08 10:37:56.521882172 +0000 UTC m=+4.673676374, 105.716661ms after stability
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.522Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: updating configuration: command=AddStaging server-id=server-732bb4ac-de8e-e246-f305-da93d9b517f2 server-addr=732bb4ac-de8e-e246-f305-da93d9b517f2 servers="[{Suffrage:Voter ID:server-52ec5894-fbca-fa47-0487-be8f97dfa858 Address:52ec5894-fbca-fa47-0487-be8f97dfa858} {Suffrage:Voter ID:server-85ba9891-6b1c-fefe-6043-c3341ded65ee Address:85ba9891-6b1c-fefe-6043-c3341ded65ee} {Suffrage:Voter ID:server-732bb4ac-de8e-e246-f305-da93d9b517f2 Address:732bb4ac-de8e-e246-f305-da93d9b517f2}]"
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.522Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: updating configuration: command=AddStaging server-id=server-732bb4ac-de8e-e246-f305-da93d9b517f2 server-addr=732bb4ac-de8e-e246-f305-da93d9b517f2 servers="[{Suffrage:Voter ID:server-52ec5894-fbca-fa47-0487-be8f97dfa858 Address:52ec5894-fbca-fa47-0487-be8f97dfa858} {Suffrage:Voter ID:server-85ba9891-6b1c-fefe-6043-c3341ded65ee Address:85ba9891-6b1c-fefe-6043-c3341ded65ee} {Suffrage:Voter ID:server-732bb4ac-de8e-e246-f305-da93d9b517f2 Address:732bb4ac-de8e-e246-f305-da93d9b517f2}]"
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.523Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: added peer, starting replication: peer=server-732bb4ac-de8e-e246-f305-da93d9b517f2
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.523Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: added peer, starting replication: peer=server-732bb4ac-de8e-e246-f305-da93d9b517f2
testLoggerAdapter verbose: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.523Z [WARN]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: failed to get previous log: previous-index=3 last-index=0 error="log not found"
    testing.go:149: server-732bb4ac-de8e-e246-f305-da93d9b517f2: 2024-01-08T10:37:56.523Z [WARN]  server-732bb4ac-de8e-e246-f305-da93d9b517f2: failed to get previous log: previous-index=3 last-index=0 error="log not found"
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.523Z [WARN]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: appendEntries rejected, sending older logs: peer="{Voter server-732bb4ac-de8e-e246-f305-da93d9b517f2 732bb4ac-de8e-e246-f305-da93d9b517f2}" next=1
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.523Z [WARN]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: appendEntries rejected, sending older logs: peer="{Voter server-732bb4ac-de8e-e246-f305-da93d9b517f2 732bb4ac-de8e-e246-f305-da93d9b517f2}" next=1
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.524Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: pipelining replication: peer="{Voter server-732bb4ac-de8e-e246-f305-da93d9b517f2 732bb4ac-de8e-e246-f305-da93d9b517f2}"
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.524Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: pipelining replication: peer="{Voter server-732bb4ac-de8e-e246-f305-da93d9b517f2 732bb4ac-de8e-e246-f305-da93d9b517f2}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.524Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:56.524Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.630Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.524905502 +0000 UTC m=+4.676699630 (1 nodes), 2.218µs from start of poll, 205.496386ms from cluster start. Timeout at 2024-01-08 10:37:56.630021434 +0000 UTC m=+4.781815556, 105.115926ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:56.630Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.524905502 +0000 UTC m=+4.676699630 (1 nodes), 2.218µs from start of poll, 205.496386ms from cluster start. Timeout at 2024-01-08 10:37:56.630021434 +0000 UTC m=+4.781815556, 105.115926ms after stability
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.630Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: aborting pipeline replication: peer="{Voter server-732bb4ac-de8e-e246-f305-da93d9b517f2 732bb4ac-de8e-e246-f305-da93d9b517f2}"
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.630Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: aborting pipeline replication: peer="{Voter server-732bb4ac-de8e-e246-f305-da93d9b517f2 732bb4ac-de8e-e246-f305-da93d9b517f2}"
testLoggerAdapter verbose: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.630Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: aborting pipeline replication: peer="{Voter server-85ba9891-6b1c-fefe-6043-c3341ded65ee 85ba9891-6b1c-fefe-6043-c3341ded65ee}"
    testing.go:149: server-52ec5894-fbca-fa47-0487-be8f97dfa858: 2024-01-08T10:37:56.630Z [INFO]  server-52ec5894-fbca-fa47-0487-be8f97dfa858: aborting pipeline replication: peer="{Voter server-85ba9891-6b1c-fefe-6043-c3341ded65ee 85ba9891-6b1c-fefe-6043-c3341ded65ee}"
--- PASS: TestRaft_HasExistingState (0.31s)
=== RUN   TestRaft_SingleNode
testLoggerAdapter verbose: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.680Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4f8eeb6-c545-2699-5b54-dc7260cda68d Address:c4f8eeb6-c545-2699-5b54-dc7260cda68d}]"
    testing.go:149: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.680Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4f8eeb6-c545-2699-5b54-dc7260cda68d Address:c4f8eeb6-c545-2699-5b54-dc7260cda68d}]"
testLoggerAdapter verbose: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.684Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: entering follower state: follower="Node at c4f8eeb6-c545-2699-5b54-dc7260cda68d [Follower]" leader=
    testing.go:149: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.684Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: entering follower state: follower="Node at c4f8eeb6-c545-2699-5b54-dc7260cda68d [Follower]" leader=
testLoggerAdapter verbose: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.778Z [WARN]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.778Z [WARN]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.778Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: entering candidate state: node="Node at c4f8eeb6-c545-2699-5b54-dc7260cda68d [Candidate]" term=2
    testing.go:149: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.778Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: entering candidate state: node="Node at c4f8eeb6-c545-2699-5b54-dc7260cda68d [Candidate]" term=2
testLoggerAdapter verbose: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.779Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: election won: tally=1
    testing.go:149: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.779Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: election won: tally=1
testLoggerAdapter verbose: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.779Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: entering leader state: leader="Node at c4f8eeb6-c545-2699-5b54-dc7260cda68d [Leader]"
    testing.go:149: server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: 2024-01-08T10:37:56.779Z [INFO]  server-c4f8eeb6-c545-2699-5b54-dc7260cda68d: entering leader state: leader="Node at c4f8eeb6-c545-2699-5b54-dc7260cda68d [Leader]"
--- PASS: TestRaft_SingleNode (0.15s)
=== RUN   TestRaft_TripleNode
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.783Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef15228e-5fb1-958d-b7f9-66dedbb7121d Address:ef15228e-5fb1-958d-b7f9-66dedbb7121d} {Suffrage:Voter ID:server-8db61a4d-3ca5-56ee-860c-c752b982adac Address:8db61a4d-3ca5-56ee-860c-c752b982adac} {Suffrage:Voter ID:server-445e5caf-3316-85ac-f3f6-9c49471c7373 Address:445e5caf-3316-85ac-f3f6-9c49471c7373}]"
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.783Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef15228e-5fb1-958d-b7f9-66dedbb7121d Address:ef15228e-5fb1-958d-b7f9-66dedbb7121d} {Suffrage:Voter ID:server-8db61a4d-3ca5-56ee-860c-c752b982adac Address:8db61a4d-3ca5-56ee-860c-c752b982adac} {Suffrage:Voter ID:server-445e5caf-3316-85ac-f3f6-9c49471c7373 Address:445e5caf-3316-85ac-f3f6-9c49471c7373}]"
testLoggerAdapter verbose: server-8db61a4d-3ca5-56ee-860c-c752b982adac: 2024-01-08T10:37:56.784Z [INFO]  server-8db61a4d-3ca5-56ee-860c-c752b982adac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef15228e-5fb1-958d-b7f9-66dedbb7121d Address:ef15228e-5fb1-958d-b7f9-66dedbb7121d} {Suffrage:Voter ID:server-8db61a4d-3ca5-56ee-860c-c752b982adac Address:8db61a4d-3ca5-56ee-860c-c752b982adac} {Suffrage:Voter ID:server-445e5caf-3316-85ac-f3f6-9c49471c7373 Address:445e5caf-3316-85ac-f3f6-9c49471c7373}]"
    testing.go:149: server-8db61a4d-3ca5-56ee-860c-c752b982adac: 2024-01-08T10:37:56.784Z [INFO]  server-8db61a4d-3ca5-56ee-860c-c752b982adac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef15228e-5fb1-958d-b7f9-66dedbb7121d Address:ef15228e-5fb1-958d-b7f9-66dedbb7121d} {Suffrage:Voter ID:server-8db61a4d-3ca5-56ee-860c-c752b982adac Address:8db61a4d-3ca5-56ee-860c-c752b982adac} {Suffrage:Voter ID:server-445e5caf-3316-85ac-f3f6-9c49471c7373 Address:445e5caf-3316-85ac-f3f6-9c49471c7373}]"
testLoggerAdapter verbose: server-445e5caf-3316-85ac-f3f6-9c49471c7373: 2024-01-08T10:37:56.784Z [INFO]  server-445e5caf-3316-85ac-f3f6-9c49471c7373: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef15228e-5fb1-958d-b7f9-66dedbb7121d Address:ef15228e-5fb1-958d-b7f9-66dedbb7121d} {Suffrage:Voter ID:server-8db61a4d-3ca5-56ee-860c-c752b982adac Address:8db61a4d-3ca5-56ee-860c-c752b982adac} {Suffrage:Voter ID:server-445e5caf-3316-85ac-f3f6-9c49471c7373 Address:445e5caf-3316-85ac-f3f6-9c49471c7373}]"
    testing.go:149: server-445e5caf-3316-85ac-f3f6-9c49471c7373: 2024-01-08T10:37:56.784Z [INFO]  server-445e5caf-3316-85ac-f3f6-9c49471c7373: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef15228e-5fb1-958d-b7f9-66dedbb7121d Address:ef15228e-5fb1-958d-b7f9-66dedbb7121d} {Suffrage:Voter ID:server-8db61a4d-3ca5-56ee-860c-c752b982adac Address:8db61a4d-3ca5-56ee-860c-c752b982adac} {Suffrage:Voter ID:server-445e5caf-3316-85ac-f3f6-9c49471c7373 Address:445e5caf-3316-85ac-f3f6-9c49471c7373}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.785Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:37:56.785Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.785Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: entering follower state: follower="Node at ef15228e-5fb1-958d-b7f9-66dedbb7121d [Follower]" leader=
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.785Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: entering follower state: follower="Node at ef15228e-5fb1-958d-b7f9-66dedbb7121d [Follower]" leader=
testLoggerAdapter verbose: server-8db61a4d-3ca5-56ee-860c-c752b982adac: 2024-01-08T10:37:56.785Z [INFO]  server-8db61a4d-3ca5-56ee-860c-c752b982adac: entering follower state: follower="Node at 8db61a4d-3ca5-56ee-860c-c752b982adac [Follower]" leader=
    testing.go:149: server-8db61a4d-3ca5-56ee-860c-c752b982adac: 2024-01-08T10:37:56.785Z [INFO]  server-8db61a4d-3ca5-56ee-860c-c752b982adac: entering follower state: follower="Node at 8db61a4d-3ca5-56ee-860c-c752b982adac [Follower]" leader=
testLoggerAdapter verbose: server-445e5caf-3316-85ac-f3f6-9c49471c7373: 2024-01-08T10:37:56.786Z [INFO]  server-445e5caf-3316-85ac-f3f6-9c49471c7373: entering follower state: follower="Node at 445e5caf-3316-85ac-f3f6-9c49471c7373 [Follower]" leader=
    testing.go:149: server-445e5caf-3316-85ac-f3f6-9c49471c7373: 2024-01-08T10:37:56.786Z [INFO]  server-445e5caf-3316-85ac-f3f6-9c49471c7373: entering follower state: follower="Node at 445e5caf-3316-85ac-f3f6-9c49471c7373 [Follower]" leader=
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.862Z [WARN]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.862Z [WARN]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.862Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: entering candidate state: node="Node at ef15228e-5fb1-958d-b7f9-66dedbb7121d [Candidate]" term=2
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.862Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: entering candidate state: node="Node at ef15228e-5fb1-958d-b7f9-66dedbb7121d [Candidate]" term=2
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: election won: tally=2
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: election won: tally=2
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: entering leader state: leader="Node at ef15228e-5fb1-958d-b7f9-66dedbb7121d [Leader]"
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: entering leader state: leader="Node at ef15228e-5fb1-958d-b7f9-66dedbb7121d [Leader]"
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: added peer, starting replication: peer=server-8db61a4d-3ca5-56ee-860c-c752b982adac
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: added peer, starting replication: peer=server-8db61a4d-3ca5-56ee-860c-c752b982adac
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: added peer, starting replication: peer=server-445e5caf-3316-85ac-f3f6-9c49471c7373
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: added peer, starting replication: peer=server-445e5caf-3316-85ac-f3f6-9c49471c7373
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: pipelining replication: peer="{Voter server-445e5caf-3316-85ac-f3f6-9c49471c7373 445e5caf-3316-85ac-f3f6-9c49471c7373}"
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.863Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: pipelining replication: peer="{Voter server-445e5caf-3316-85ac-f3f6-9c49471c7373 445e5caf-3316-85ac-f3f6-9c49471c7373}"
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.864Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: pipelining replication: peer="{Voter server-8db61a4d-3ca5-56ee-860c-c752b982adac 8db61a4d-3ca5-56ee-860c-c752b982adac}"
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:56.864Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: pipelining replication: peer="{Voter server-8db61a4d-3ca5-56ee-860c-c752b982adac 8db61a4d-3ca5-56ee-860c-c752b982adac}"
testLoggerAdapter verbose: server-8db61a4d-3ca5-56ee-860c-c752b982adac: 2024-01-08T10:37:56.864Z [WARN]  server-8db61a4d-3ca5-56ee-860c-c752b982adac: rejecting vote request since our last term is greater: candidate=ef15228e-5fb1-958d-b7f9-66dedbb7121d last-term=2 last-candidate-term=1
    testing.go:149: server-8db61a4d-3ca5-56ee-860c-c752b982adac: 2024-01-08T10:37:56.864Z [WARN]  server-8db61a4d-3ca5-56ee-860c-c752b982adac: rejecting vote request since our last term is greater: candidate=ef15228e-5fb1-958d-b7f9-66dedbb7121d last-term=2 last-candidate-term=1
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.980Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:56.86515134 +0000 UTC m=+5.016945490 (2 nodes), 79.731262ms from start of poll, 83.505891ms from cluster start. Timeout at 2024-01-08 10:37:56.980222053 +0000 UTC m=+5.132016343, 115.070853ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:56.980Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:56.86515134 +0000 UTC m=+5.016945490 (2 nodes), 79.731262ms from start of poll, 83.505891ms from cluster start. Timeout at 2024-01-08 10:37:56.980222053 +0000 UTC m=+5.132016343, 115.070853ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:56.980Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:56.980Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.086Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.980922148 +0000 UTC m=+5.132716268 (1 nodes), 3.322µs from start of poll, 199.276669ms from cluster start. Timeout at 2024-01-08 10:37:57.086417141 +0000 UTC m=+5.238211261, 105.494993ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:57.086Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:56.980922148 +0000 UTC m=+5.132716268 (1 nodes), 3.322µs from start of poll, 199.276669ms from cluster start. Timeout at 2024-01-08 10:37:57.086417141 +0000 UTC m=+5.238211261, 105.494993ms after stability
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:57.103Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: aborting pipeline replication: peer="{Voter server-445e5caf-3316-85ac-f3f6-9c49471c7373 445e5caf-3316-85ac-f3f6-9c49471c7373}"
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:57.103Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: aborting pipeline replication: peer="{Voter server-445e5caf-3316-85ac-f3f6-9c49471c7373 445e5caf-3316-85ac-f3f6-9c49471c7373}"
testLoggerAdapter verbose: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:57.103Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: aborting pipeline replication: peer="{Voter server-8db61a4d-3ca5-56ee-860c-c752b982adac 8db61a4d-3ca5-56ee-860c-c752b982adac}"
    testing.go:149: server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: 2024-01-08T10:37:57.103Z [INFO]  server-ef15228e-5fb1-958d-b7f9-66dedbb7121d: aborting pipeline replication: peer="{Voter server-8db61a4d-3ca5-56ee-860c-c752b982adac 8db61a4d-3ca5-56ee-860c-c752b982adac}"
--- PASS: TestRaft_TripleNode (0.33s)
=== RUN   TestRaft_LeaderFail
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.122Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 Address:ccc441d5-5ed3-b0ff-1334-1c1f12e83de1} {Suffrage:Voter ID:server-294f74a8-d626-23aa-d7df-0123d027628c Address:294f74a8-d626-23aa-d7df-0123d027628c} {Suffrage:Voter ID:server-fff0e3e6-f861-221f-5da4-b30db566d071 Address:fff0e3e6-f861-221f-5da4-b30db566d071}]"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.122Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 Address:ccc441d5-5ed3-b0ff-1334-1c1f12e83de1} {Suffrage:Voter ID:server-294f74a8-d626-23aa-d7df-0123d027628c Address:294f74a8-d626-23aa-d7df-0123d027628c} {Suffrage:Voter ID:server-fff0e3e6-f861-221f-5da4-b30db566d071 Address:fff0e3e6-f861-221f-5da4-b30db566d071}]"
testLoggerAdapter verbose: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.123Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 Address:ccc441d5-5ed3-b0ff-1334-1c1f12e83de1} {Suffrage:Voter ID:server-294f74a8-d626-23aa-d7df-0123d027628c Address:294f74a8-d626-23aa-d7df-0123d027628c} {Suffrage:Voter ID:server-fff0e3e6-f861-221f-5da4-b30db566d071 Address:fff0e3e6-f861-221f-5da4-b30db566d071}]"
    testing.go:149: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.123Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 Address:ccc441d5-5ed3-b0ff-1334-1c1f12e83de1} {Suffrage:Voter ID:server-294f74a8-d626-23aa-d7df-0123d027628c Address:294f74a8-d626-23aa-d7df-0123d027628c} {Suffrage:Voter ID:server-fff0e3e6-f861-221f-5da4-b30db566d071 Address:fff0e3e6-f861-221f-5da4-b30db566d071}]"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.123Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 Address:ccc441d5-5ed3-b0ff-1334-1c1f12e83de1} {Suffrage:Voter ID:server-294f74a8-d626-23aa-d7df-0123d027628c Address:294f74a8-d626-23aa-d7df-0123d027628c} {Suffrage:Voter ID:server-fff0e3e6-f861-221f-5da4-b30db566d071 Address:fff0e3e6-f861-221f-5da4-b30db566d071}]"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.123Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 Address:ccc441d5-5ed3-b0ff-1334-1c1f12e83de1} {Suffrage:Voter ID:server-294f74a8-d626-23aa-d7df-0123d027628c Address:294f74a8-d626-23aa-d7df-0123d027628c} {Suffrage:Voter ID:server-fff0e3e6-f861-221f-5da4-b30db566d071 Address:fff0e3e6-f861-221f-5da4-b30db566d071}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.123Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.123Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering follower state: follower="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Follower]" leader=
    testing.go:149: cluster: 2024-01-08T10:37:57.123Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.123Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering follower state: follower="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Follower]" leader=
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.124Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering follower state: follower="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Follower]" leader=
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.124Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering follower state: follower="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Follower]" leader=
testLoggerAdapter verbose: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.124Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: entering follower state: follower="Node at 294f74a8-d626-23aa-d7df-0123d027628c [Follower]" leader=
    testing.go:149: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.124Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: entering follower state: follower="Node at 294f74a8-d626-23aa-d7df-0123d027628c [Follower]" leader=
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering candidate state: node="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Candidate]" term=2
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering candidate state: node="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Candidate]" term=2
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: election won: tally=2
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: election won: tally=2
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering leader state: leader="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Leader]"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering leader state: leader="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Leader]"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: added peer, starting replication: peer=server-294f74a8-d626-23aa-d7df-0123d027628c
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: added peer, starting replication: peer=server-294f74a8-d626-23aa-d7df-0123d027628c
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: added peer, starting replication: peer=server-fff0e3e6-f861-221f-5da4-b30db566d071
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: added peer, starting replication: peer=server-fff0e3e6-f861-221f-5da4-b30db566d071
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: pipelining replication: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: pipelining replication: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: pipelining replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.199Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: pipelining replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.306Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:57.199874863 +0000 UTC m=+5.351668714 (2 nodes), 75.837938ms from start of poll, 78.532471ms from cluster start. Timeout at 2024-01-08 10:37:57.306042094 +0000 UTC m=+5.457836216, 106.167502ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:57.306Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:57.199874863 +0000 UTC m=+5.351668714 (2 nodes), 75.837938ms from start of poll, 78.532471ms from cluster start. Timeout at 2024-01-08 10:37:57.306042094 +0000 UTC m=+5.457836216, 106.167502ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.306Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:57.306Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.414Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:57.306776009 +0000 UTC m=+5.458570127 (1 nodes), 2.854µs from start of poll, 185.433884ms from cluster start. Timeout at 2024-01-08 10:37:57.413910525 +0000 UTC m=+5.565704653, 107.134526ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:57.414Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:57.306776009 +0000 UTC m=+5.458570127 (1 nodes), 2.854µs from start of poll, 185.433884ms from cluster start. Timeout at 2024-01-08 10:37:57.413910525 +0000 UTC m=+5.565704653, 107.134526ms after stability
    raft_test.go:336: [INFO] Disconnecting Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Leader]
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.426Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=fff0e3e6-f861-221f-5da4-b30db566d071 error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.426Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=fff0e3e6-f861-221f-5da4-b30db566d071 error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.427Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=294f74a8-d626-23aa-d7df-0123d027628c error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.427Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=294f74a8-d626-23aa-d7df-0123d027628c error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.433Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:57.433Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.433Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to pipeline appendEntries: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="append pipeline closed"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.433Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to pipeline appendEntries: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="append pipeline closed"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.433Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: aborting pipeline replication: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.433Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: aborting pipeline replication: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.435Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to pipeline appendEntries: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="append pipeline closed"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.435Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to pipeline appendEntries: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="append pipeline closed"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.435Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: aborting pipeline replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.435Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: aborting pipeline replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.442Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.442Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.443Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.443Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.444Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=294f74a8-d626-23aa-d7df-0123d027628c error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.444Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=294f74a8-d626-23aa-d7df-0123d027628c error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.447Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=fff0e3e6-f861-221f-5da4-b30db566d071 error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.447Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=fff0e3e6-f861-221f-5da4-b30db566d071 error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.460Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=294f74a8-d626-23aa-d7df-0123d027628c error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.460Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=294f74a8-d626-23aa-d7df-0123d027628c error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.462Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.462Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.464Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.464Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.465Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=fff0e3e6-f861-221f-5da4-b30db566d071 error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.465Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to heartbeat to: peer=fff0e3e6-f861-221f-5da4-b30db566d071 error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.473Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact: server-id=server-fff0e3e6-f861-221f-5da4-b30db566d071 time=52.596144ms
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.473Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact: server-id=server-fff0e3e6-f861-221f-5da4-b30db566d071 time=52.596144ms
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact: server-id=server-294f74a8-d626-23aa-d7df-0123d027628c time=61.692072ms
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact: server-id=server-294f74a8-d626-23aa-d7df-0123d027628c time=61.692072ms
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact: server-id=server-fff0e3e6-f861-221f-5da4-b30db566d071 time=65.279417ms
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact: server-id=server-fff0e3e6-f861-221f-5da4-b30db566d071 time=65.279417ms
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact quorum of nodes, stepping down
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to contact quorum of nodes, stepping down
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering follower state: follower="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Follower]" leader=
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.486Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering follower state: follower="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Follower]" leader=
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.487Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.487Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to appendEntries to: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.522Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: heartbeat timeout reached, starting election: last-leader=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.522Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: heartbeat timeout reached, starting election: last-leader=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.522Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering candidate state: node="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Candidate]" term=3
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.522Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering candidate state: node="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Candidate]" term=3
testLoggerAdapter verbose: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.522Z [WARN]  server-294f74a8-d626-23aa-d7df-0123d027628c: rejecting vote request since we have a leader: from=fff0e3e6-f861-221f-5da4-b30db566d071 leader=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
    testing.go:149: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.522Z [WARN]  server-294f74a8-d626-23aa-d7df-0123d027628c: rejecting vote request since we have a leader: from=fff0e3e6-f861-221f-5da4-b30db566d071 leader=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.523Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to make requestVote RPC: target="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.523Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to make requestVote RPC: target="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.546Z [WARN]  server-294f74a8-d626-23aa-d7df-0123d027628c: heartbeat timeout reached, starting election: last-leader=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
    testing.go:149: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.546Z [WARN]  server-294f74a8-d626-23aa-d7df-0123d027628c: heartbeat timeout reached, starting election: last-leader=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
testLoggerAdapter verbose: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.546Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: entering candidate state: node="Node at 294f74a8-d626-23aa-d7df-0123d027628c [Candidate]" term=3
    testing.go:149: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.546Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: entering candidate state: node="Node at 294f74a8-d626-23aa-d7df-0123d027628c [Candidate]" term=3
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.546Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: duplicate requestVote for same term: term=3
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.546Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: duplicate requestVote for same term: term=3
testLoggerAdapter verbose: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.547Z [ERROR] server-294f74a8-d626-23aa-d7df-0123d027628c: failed to make requestVote RPC: target="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.547Z [ERROR] server-294f74a8-d626-23aa-d7df-0123d027628c: failed to make requestVote RPC: target="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering candidate state: node="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Candidate]" term=3
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering candidate state: node="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Candidate]" term=3
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.582Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.601Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: Election timeout reached, restarting election
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.601Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: Election timeout reached, restarting election
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.602Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering candidate state: node="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Candidate]" term=4
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.602Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering candidate state: node="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Candidate]" term=4
testLoggerAdapter verbose: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.602Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: entering follower state: follower="Node at 294f74a8-d626-23aa-d7df-0123d027628c [Follower]" leader=
    testing.go:149: server-294f74a8-d626-23aa-d7df-0123d027628c: 2024-01-08T10:37:57.602Z [INFO]  server-294f74a8-d626-23aa-d7df-0123d027628c: entering follower state: follower="Node at 294f74a8-d626-23aa-d7df-0123d027628c [Follower]" leader=
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.602Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: election won: tally=2
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.602Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: election won: tally=2
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering leader state: leader="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Leader]"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: entering leader state: leader="Node at fff0e3e6-f861-221f-5da4-b30db566d071 [Leader]"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: added peer, starting replication: peer=server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: added peer, starting replication: peer=server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: added peer, starting replication: peer=server-294f74a8-d626-23aa-d7df-0123d027628c
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: added peer, starting replication: peer=server-294f74a8-d626-23aa-d7df-0123d027628c
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: pipelining replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.603Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: pipelining replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.604Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to make requestVote RPC: target="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.604Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to make requestVote RPC: target="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.604Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.604Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.611Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.611Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.638Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.638Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.638Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.638Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.653Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to contact: server-id=server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 time=50.436127ms
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.653Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to contact: server-id=server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 time=50.436127ms
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.657Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.657Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.662Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.662Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.677Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: Election timeout reached, restarting election
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.677Z [WARN]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: Election timeout reached, restarting election
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.678Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering candidate state: node="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Candidate]" term=4
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.678Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering candidate state: node="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Candidate]" term=4
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.678Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.678Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-fff0e3e6-f861-221f-5da4-b30db566d071 fff0e3e6-f861-221f-5da4-b30db566d071}" error="failed to connect to peer: fff0e3e6-f861-221f-5da4-b30db566d071"
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.679Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.679Z [ERROR] server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: failed to make requestVote RPC: target="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}" error="failed to connect to peer: 294f74a8-d626-23aa-d7df-0123d027628c"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.690Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.690Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to heartbeat to: peer=ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.698Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.698Z [ERROR] server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to appendEntries to: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}" error="failed to connect to peer: ccc441d5-5ed3-b0ff-1334-1c1f12e83de1"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.701Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to contact: server-id=server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 time=98.457441ms
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.701Z [WARN]  server-fff0e3e6-f861-221f-5da4-b30db566d071: failed to contact: server-id=server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 time=98.457441ms
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.710Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:57.604638139 +0000 UTC m=+5.756432287 (1 nodes), 171.255747ms from start of poll, 483.296044ms from cluster start. Timeout at 2024-01-08 10:37:57.709923094 +0000 UTC m=+5.861717231, 105.284944ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:57.710Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:57.604638139 +0000 UTC m=+5.756432287 (1 nodes), 171.255747ms from start of poll, 483.296044ms from cluster start. Timeout at 2024-01-08 10:37:57.709923094 +0000 UTC m=+5.861717231, 105.284944ms after stability
    raft_test.go:371: [INFO] Reconnecting Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Candidate]
testLoggerAdapter verbose: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.746Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering follower state: follower="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Follower]" leader=fff0e3e6-f861-221f-5da4-b30db566d071
    testing.go:149: server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: 2024-01-08T10:37:57.746Z [INFO]  server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1: entering follower state: follower="Node at ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 [Follower]" leader=fff0e3e6-f861-221f-5da4-b30db566d071
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.751Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: pipelining replication: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.751Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: pipelining replication: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.754Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: aborting pipeline replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.754Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: aborting pipeline replication: peer="{Voter server-294f74a8-d626-23aa-d7df-0123d027628c 294f74a8-d626-23aa-d7df-0123d027628c}"
testLoggerAdapter verbose: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.754Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: aborting pipeline replication: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}"
    testing.go:149: server-fff0e3e6-f861-221f-5da4-b30db566d071: 2024-01-08T10:37:57.754Z [INFO]  server-fff0e3e6-f861-221f-5da4-b30db566d071: aborting pipeline replication: peer="{Voter server-ccc441d5-5ed3-b0ff-1334-1c1f12e83de1 ccc441d5-5ed3-b0ff-1334-1c1f12e83de1}"
--- PASS: TestRaft_LeaderFail (0.65s)
=== RUN   TestRaft_BehindFollower
testLoggerAdapter verbose: server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: 2024-01-08T10:37:57.769Z [INFO]  server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 Address:e35886ff-b3c4-2beb-6f96-2bf8187ac724} {Suffrage:Voter ID:server-ffbe7595-de26-4eab-9c10-64ed72fc0acf Address:ffbe7595-de26-4eab-9c10-64ed72fc0acf} {Suffrage:Voter ID:server-554072fc-a8a1-bf41-8ac6-8596225fc000 Address:554072fc-a8a1-bf41-8ac6-8596225fc000}]"
    testing.go:149: server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: 2024-01-08T10:37:57.769Z [INFO]  server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 Address:e35886ff-b3c4-2beb-6f96-2bf8187ac724} {Suffrage:Voter ID:server-ffbe7595-de26-4eab-9c10-64ed72fc0acf Address:ffbe7595-de26-4eab-9c10-64ed72fc0acf} {Suffrage:Voter ID:server-554072fc-a8a1-bf41-8ac6-8596225fc000 Address:554072fc-a8a1-bf41-8ac6-8596225fc000}]"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.770Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 Address:e35886ff-b3c4-2beb-6f96-2bf8187ac724} {Suffrage:Voter ID:server-ffbe7595-de26-4eab-9c10-64ed72fc0acf Address:ffbe7595-de26-4eab-9c10-64ed72fc0acf} {Suffrage:Voter ID:server-554072fc-a8a1-bf41-8ac6-8596225fc000 Address:554072fc-a8a1-bf41-8ac6-8596225fc000}]"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.770Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 Address:e35886ff-b3c4-2beb-6f96-2bf8187ac724} {Suffrage:Voter ID:server-ffbe7595-de26-4eab-9c10-64ed72fc0acf Address:ffbe7595-de26-4eab-9c10-64ed72fc0acf} {Suffrage:Voter ID:server-554072fc-a8a1-bf41-8ac6-8596225fc000 Address:554072fc-a8a1-bf41-8ac6-8596225fc000}]"
testLoggerAdapter verbose: server-554072fc-a8a1-bf41-8ac6-8596225fc000: 2024-01-08T10:37:57.771Z [INFO]  server-554072fc-a8a1-bf41-8ac6-8596225fc000: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 Address:e35886ff-b3c4-2beb-6f96-2bf8187ac724} {Suffrage:Voter ID:server-ffbe7595-de26-4eab-9c10-64ed72fc0acf Address:ffbe7595-de26-4eab-9c10-64ed72fc0acf} {Suffrage:Voter ID:server-554072fc-a8a1-bf41-8ac6-8596225fc000 Address:554072fc-a8a1-bf41-8ac6-8596225fc000}]"
    testing.go:149: server-554072fc-a8a1-bf41-8ac6-8596225fc000: 2024-01-08T10:37:57.771Z [INFO]  server-554072fc-a8a1-bf41-8ac6-8596225fc000: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 Address:e35886ff-b3c4-2beb-6f96-2bf8187ac724} {Suffrage:Voter ID:server-ffbe7595-de26-4eab-9c10-64ed72fc0acf Address:ffbe7595-de26-4eab-9c10-64ed72fc0acf} {Suffrage:Voter ID:server-554072fc-a8a1-bf41-8ac6-8596225fc000 Address:554072fc-a8a1-bf41-8ac6-8596225fc000}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.771Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:57.771Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: 2024-01-08T10:37:57.770Z [INFO]  server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: entering follower state: follower="Node at e35886ff-b3c4-2beb-6f96-2bf8187ac724 [Follower]" leader=
    testing.go:149: server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: 2024-01-08T10:37:57.770Z [INFO]  server-e35886ff-b3c4-2beb-6f96-2bf8187ac724: entering follower state: follower="Node at e35886ff-b3c4-2beb-6f96-2bf8187ac724 [Follower]" leader=
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.771Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: entering follower state: follower="Node at ffbe7595-de26-4eab-9c10-64ed72fc0acf [Follower]" leader=
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.771Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: entering follower state: follower="Node at ffbe7595-de26-4eab-9c10-64ed72fc0acf [Follower]" leader=
testLoggerAdapter verbose: server-554072fc-a8a1-bf41-8ac6-8596225fc000: 2024-01-08T10:37:57.772Z [INFO]  server-554072fc-a8a1-bf41-8ac6-8596225fc000: entering follower state: follower="Node at 554072fc-a8a1-bf41-8ac6-8596225fc000 [Follower]" leader=
    testing.go:149: server-554072fc-a8a1-bf41-8ac6-8596225fc000: 2024-01-08T10:37:57.772Z [INFO]  server-554072fc-a8a1-bf41-8ac6-8596225fc000: entering follower state: follower="Node at 554072fc-a8a1-bf41-8ac6-8596225fc000 [Follower]" leader=
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.826Z [WARN]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.826Z [WARN]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.827Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: entering candidate state: node="Node at ffbe7595-de26-4eab-9c10-64ed72fc0acf [Candidate]" term=2
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.827Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: entering candidate state: node="Node at ffbe7595-de26-4eab-9c10-64ed72fc0acf [Candidate]" term=2
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.827Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: election won: tally=2
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.827Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: election won: tally=2
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.828Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: entering leader state: leader="Node at ffbe7595-de26-4eab-9c10-64ed72fc0acf [Leader]"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.828Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: entering leader state: leader="Node at ffbe7595-de26-4eab-9c10-64ed72fc0acf [Leader]"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.828Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: added peer, starting replication: peer=server-e35886ff-b3c4-2beb-6f96-2bf8187ac724
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.828Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: added peer, starting replication: peer=server-e35886ff-b3c4-2beb-6f96-2bf8187ac724
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.828Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: added peer, starting replication: peer=server-554072fc-a8a1-bf41-8ac6-8596225fc000
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.828Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: added peer, starting replication: peer=server-554072fc-a8a1-bf41-8ac6-8596225fc000
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.829Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: pipelining replication: peer="{Voter server-554072fc-a8a1-bf41-8ac6-8596225fc000 554072fc-a8a1-bf41-8ac6-8596225fc000}"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.829Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: pipelining replication: peer="{Voter server-554072fc-a8a1-bf41-8ac6-8596225fc000 554072fc-a8a1-bf41-8ac6-8596225fc000}"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.830Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: pipelining replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:57.830Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: pipelining replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.935Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:57.830316701 +0000 UTC m=+5.982110871 (1 nodes), 58.530192ms from start of poll, 72.617841ms from cluster start. Timeout at 2024-01-08 10:37:57.93583383 +0000 UTC m=+6.087627954, 105.517083ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:57.935Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:57.830316701 +0000 UTC m=+5.982110871 (1 nodes), 58.530192ms from start of poll, 72.617841ms from cluster start. Timeout at 2024-01-08 10:37:57.93583383 +0000 UTC m=+6.087627954, 105.517083ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:57.936Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:37:57.936Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.042Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:57.936311447 +0000 UTC m=+6.088105575 (2 nodes), 4.612µs from start of poll, 178.612545ms from cluster start. Timeout at 2024-01-08 10:37:58.0415297 +0000 UTC m=+6.193323817, 105.218242ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:58.042Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:57.936311447 +0000 UTC m=+6.088105575 (2 nodes), 4.612µs from start of poll, 178.612545ms from cluster start. Timeout at 2024-01-08 10:37:58.0415297 +0000 UTC m=+6.193323817, 105.218242ms after stability
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.042Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to pipeline appendEntries: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}" error="append pipeline closed"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.042Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to pipeline appendEntries: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}" error="append pipeline closed"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.043Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: aborting pipeline replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.043Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: aborting pipeline replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.043Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to appendEntries to: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}" error="failed to connect to peer: e35886ff-b3c4-2beb-6f96-2bf8187ac724"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.043Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to appendEntries to: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}" error="failed to connect to peer: e35886ff-b3c4-2beb-6f96-2bf8187ac724"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.046Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to heartbeat to: peer=e35886ff-b3c4-2beb-6f96-2bf8187ac724 error="failed to connect to peer: e35886ff-b3c4-2beb-6f96-2bf8187ac724"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.046Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to heartbeat to: peer=e35886ff-b3c4-2beb-6f96-2bf8187ac724 error="failed to connect to peer: e35886ff-b3c4-2beb-6f96-2bf8187ac724"
    raft_test.go:420: [INFO] Finished apply without behind follower
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.062Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: pipelining replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.062Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: pipelining replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.081Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:58.081Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.190Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:58.082272431 +0000 UTC m=+6.234066555 (1 nodes), 3.155µs from start of poll, 324.573525ms from cluster start. Timeout at 2024-01-08 10:37:58.189872873 +0000 UTC m=+6.341666987, 107.600432ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:58.190Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:58.082272431 +0000 UTC m=+6.234066555 (1 nodes), 3.155µs from start of poll, 324.573525ms from cluster start. Timeout at 2024-01-08 10:37:58.189872873 +0000 UTC m=+6.341666987, 107.600432ms after stability
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.190Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: aborting pipeline replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.190Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: aborting pipeline replication: peer="{Voter server-e35886ff-b3c4-2beb-6f96-2bf8187ac724 e35886ff-b3c4-2beb-6f96-2bf8187ac724}"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.190Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: aborting pipeline replication: peer="{Voter server-554072fc-a8a1-bf41-8ac6-8596225fc000 554072fc-a8a1-bf41-8ac6-8596225fc000}"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.190Z [INFO]  server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: aborting pipeline replication: peer="{Voter server-554072fc-a8a1-bf41-8ac6-8596225fc000 554072fc-a8a1-bf41-8ac6-8596225fc000}"
testLoggerAdapter verbose: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.241Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to heartbeat to: peer=e35886ff-b3c4-2beb-6f96-2bf8187ac724 error="command timed out"
    testing.go:149: server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: 2024-01-08T10:37:58.241Z [ERROR] server-ffbe7595-de26-4eab-9c10-64ed72fc0acf: failed to heartbeat to: peer=e35886ff-b3c4-2beb-6f96-2bf8187ac724 error="command timed out"
--- PASS: TestRaft_BehindFollower (0.49s)
=== RUN   TestRaft_ApplyNonLeader
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.250Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c43658-63e3-7733-1bec-90adffb6b5e4 Address:d7c43658-63e3-7733-1bec-90adffb6b5e4} {Suffrage:Voter ID:server-9069da43-9162-0980-805f-53736fbd3fb2 Address:9069da43-9162-0980-805f-53736fbd3fb2} {Suffrage:Voter ID:server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f Address:ba3b92fc-f488-8ff5-350c-9c6185a43d1f}]"
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.250Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c43658-63e3-7733-1bec-90adffb6b5e4 Address:d7c43658-63e3-7733-1bec-90adffb6b5e4} {Suffrage:Voter ID:server-9069da43-9162-0980-805f-53736fbd3fb2 Address:9069da43-9162-0980-805f-53736fbd3fb2} {Suffrage:Voter ID:server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f Address:ba3b92fc-f488-8ff5-350c-9c6185a43d1f}]"
testLoggerAdapter verbose: server-9069da43-9162-0980-805f-53736fbd3fb2: 2024-01-08T10:37:58.251Z [INFO]  server-9069da43-9162-0980-805f-53736fbd3fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c43658-63e3-7733-1bec-90adffb6b5e4 Address:d7c43658-63e3-7733-1bec-90adffb6b5e4} {Suffrage:Voter ID:server-9069da43-9162-0980-805f-53736fbd3fb2 Address:9069da43-9162-0980-805f-53736fbd3fb2} {Suffrage:Voter ID:server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f Address:ba3b92fc-f488-8ff5-350c-9c6185a43d1f}]"
    testing.go:149: server-9069da43-9162-0980-805f-53736fbd3fb2: 2024-01-08T10:37:58.251Z [INFO]  server-9069da43-9162-0980-805f-53736fbd3fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c43658-63e3-7733-1bec-90adffb6b5e4 Address:d7c43658-63e3-7733-1bec-90adffb6b5e4} {Suffrage:Voter ID:server-9069da43-9162-0980-805f-53736fbd3fb2 Address:9069da43-9162-0980-805f-53736fbd3fb2} {Suffrage:Voter ID:server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f Address:ba3b92fc-f488-8ff5-350c-9c6185a43d1f}]"
testLoggerAdapter verbose: server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: 2024-01-08T10:37:58.252Z [INFO]  server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c43658-63e3-7733-1bec-90adffb6b5e4 Address:d7c43658-63e3-7733-1bec-90adffb6b5e4} {Suffrage:Voter ID:server-9069da43-9162-0980-805f-53736fbd3fb2 Address:9069da43-9162-0980-805f-53736fbd3fb2} {Suffrage:Voter ID:server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f Address:ba3b92fc-f488-8ff5-350c-9c6185a43d1f}]"
    testing.go:149: server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: 2024-01-08T10:37:58.252Z [INFO]  server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c43658-63e3-7733-1bec-90adffb6b5e4 Address:d7c43658-63e3-7733-1bec-90adffb6b5e4} {Suffrage:Voter ID:server-9069da43-9162-0980-805f-53736fbd3fb2 Address:9069da43-9162-0980-805f-53736fbd3fb2} {Suffrage:Voter ID:server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f Address:ba3b92fc-f488-8ff5-350c-9c6185a43d1f}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.252Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:58.252Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.252Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: entering follower state: follower="Node at d7c43658-63e3-7733-1bec-90adffb6b5e4 [Follower]" leader=
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.252Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: entering follower state: follower="Node at d7c43658-63e3-7733-1bec-90adffb6b5e4 [Follower]" leader=
testLoggerAdapter verbose: server-9069da43-9162-0980-805f-53736fbd3fb2: 2024-01-08T10:37:58.252Z [INFO]  server-9069da43-9162-0980-805f-53736fbd3fb2: entering follower state: follower="Node at 9069da43-9162-0980-805f-53736fbd3fb2 [Follower]" leader=
    testing.go:149: server-9069da43-9162-0980-805f-53736fbd3fb2: 2024-01-08T10:37:58.252Z [INFO]  server-9069da43-9162-0980-805f-53736fbd3fb2: entering follower state: follower="Node at 9069da43-9162-0980-805f-53736fbd3fb2 [Follower]" leader=
testLoggerAdapter verbose: server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: 2024-01-08T10:37:58.253Z [INFO]  server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: entering follower state: follower="Node at ba3b92fc-f488-8ff5-350c-9c6185a43d1f [Follower]" leader=
    testing.go:149: server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: 2024-01-08T10:37:58.253Z [INFO]  server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f: entering follower state: follower="Node at ba3b92fc-f488-8ff5-350c-9c6185a43d1f [Follower]" leader=
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.318Z [WARN]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.318Z [WARN]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.318Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: entering candidate state: node="Node at d7c43658-63e3-7733-1bec-90adffb6b5e4 [Candidate]" term=2
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.318Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: entering candidate state: node="Node at d7c43658-63e3-7733-1bec-90adffb6b5e4 [Candidate]" term=2
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.318Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: election won: tally=2
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.318Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: election won: tally=2
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: entering leader state: leader="Node at d7c43658-63e3-7733-1bec-90adffb6b5e4 [Leader]"
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: entering leader state: leader="Node at d7c43658-63e3-7733-1bec-90adffb6b5e4 [Leader]"
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: added peer, starting replication: peer=server-9069da43-9162-0980-805f-53736fbd3fb2
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: added peer, starting replication: peer=server-9069da43-9162-0980-805f-53736fbd3fb2
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: added peer, starting replication: peer=server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: added peer, starting replication: peer=server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: pipelining replication: peer="{Voter server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f ba3b92fc-f488-8ff5-350c-9c6185a43d1f}"
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.319Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: pipelining replication: peer="{Voter server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f ba3b92fc-f488-8ff5-350c-9c6185a43d1f}"
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.320Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: pipelining replication: peer="{Voter server-9069da43-9162-0980-805f-53736fbd3fb2 9069da43-9162-0980-805f-53736fbd3fb2}"
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.320Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: pipelining replication: peer="{Voter server-9069da43-9162-0980-805f-53736fbd3fb2 9069da43-9162-0980-805f-53736fbd3fb2}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.426Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:58.320746129 +0000 UTC m=+6.472540287 (1 nodes), 68.259292ms from start of poll, 75.050338ms from cluster start. Timeout at 2024-01-08 10:37:58.426115722 +0000 UTC m=+6.577909846, 105.369559ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:58.426Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:58.320746129 +0000 UTC m=+6.472540287 (1 nodes), 68.259292ms from start of poll, 75.050338ms from cluster start. Timeout at 2024-01-08 10:37:58.426115722 +0000 UTC m=+6.577909846, 105.369559ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.426Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:37:58.426Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.542Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:58.426609313 +0000 UTC m=+6.578403436 (2 nodes), 4.519µs from start of poll, 180.913487ms from cluster start. Timeout at 2024-01-08 10:37:58.541898863 +0000 UTC m=+6.693692982, 115.289546ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:58.542Z [INFO]  stable state for Follower reached at 2024-01-08 10:37:58.426609313 +0000 UTC m=+6.578403436 (2 nodes), 4.519µs from start of poll, 180.913487ms from cluster start. Timeout at 2024-01-08 10:37:58.541898863 +0000 UTC m=+6.693692982, 115.289546ms after stability
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.542Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: aborting pipeline replication: peer="{Voter server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f ba3b92fc-f488-8ff5-350c-9c6185a43d1f}"
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.542Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: aborting pipeline replication: peer="{Voter server-ba3b92fc-f488-8ff5-350c-9c6185a43d1f ba3b92fc-f488-8ff5-350c-9c6185a43d1f}"
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.542Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: aborting pipeline replication: peer="{Voter server-9069da43-9162-0980-805f-53736fbd3fb2 9069da43-9162-0980-805f-53736fbd3fb2}"
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.542Z [INFO]  server-d7c43658-63e3-7733-1bec-90adffb6b5e4: aborting pipeline replication: peer="{Voter server-9069da43-9162-0980-805f-53736fbd3fb2 9069da43-9162-0980-805f-53736fbd3fb2}"
testLoggerAdapter verbose: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.593Z [ERROR] server-d7c43658-63e3-7733-1bec-90adffb6b5e4: failed to heartbeat to: peer=9069da43-9162-0980-805f-53736fbd3fb2 error="command timed out"
    testing.go:149: server-d7c43658-63e3-7733-1bec-90adffb6b5e4: 2024-01-08T10:37:58.593Z [ERROR] server-d7c43658-63e3-7733-1bec-90adffb6b5e4: failed to heartbeat to: peer=9069da43-9162-0980-805f-53736fbd3fb2 error="command timed out"
--- PASS: TestRaft_ApplyNonLeader (0.35s)
=== RUN   TestRaft_ApplyConcurrent
testLoggerAdapter verbose: server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: 2024-01-08T10:37:58.594Z [INFO]  server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 Address:166a5aa7-ac90-ae61-3cd7-834bf1b9af29} {Suffrage:Voter ID:server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf Address:6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf} {Suffrage:Voter ID:server-63826fad-7260-472a-c1f5-832dec34bd74 Address:63826fad-7260-472a-c1f5-832dec34bd74}]"
    testing.go:149: server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: 2024-01-08T10:37:58.594Z [INFO]  server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 Address:166a5aa7-ac90-ae61-3cd7-834bf1b9af29} {Suffrage:Voter ID:server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf Address:6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf} {Suffrage:Voter ID:server-63826fad-7260-472a-c1f5-832dec34bd74 Address:63826fad-7260-472a-c1f5-832dec34bd74}]"
testLoggerAdapter verbose: server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: 2024-01-08T10:37:58.595Z [INFO]  server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 Address:166a5aa7-ac90-ae61-3cd7-834bf1b9af29} {Suffrage:Voter ID:server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf Address:6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf} {Suffrage:Voter ID:server-63826fad-7260-472a-c1f5-832dec34bd74 Address:63826fad-7260-472a-c1f5-832dec34bd74}]"
    testing.go:149: server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: 2024-01-08T10:37:58.595Z [INFO]  server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 Address:166a5aa7-ac90-ae61-3cd7-834bf1b9af29} {Suffrage:Voter ID:server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf Address:6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf} {Suffrage:Voter ID:server-63826fad-7260-472a-c1f5-832dec34bd74 Address:63826fad-7260-472a-c1f5-832dec34bd74}]"
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.595Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 Address:166a5aa7-ac90-ae61-3cd7-834bf1b9af29} {Suffrage:Voter ID:server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf Address:6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf} {Suffrage:Voter ID:server-63826fad-7260-472a-c1f5-832dec34bd74 Address:63826fad-7260-472a-c1f5-832dec34bd74}]"
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.595Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 Address:166a5aa7-ac90-ae61-3cd7-834bf1b9af29} {Suffrage:Voter ID:server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf Address:6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf} {Suffrage:Voter ID:server-63826fad-7260-472a-c1f5-832dec34bd74 Address:63826fad-7260-472a-c1f5-832dec34bd74}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.595Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:58.595Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: 2024-01-08T10:37:58.595Z [INFO]  server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: entering follower state: follower="Node at 166a5aa7-ac90-ae61-3cd7-834bf1b9af29 [Follower]" leader=
    testing.go:149: server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: 2024-01-08T10:37:58.595Z [INFO]  server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29: entering follower state: follower="Node at 166a5aa7-ac90-ae61-3cd7-834bf1b9af29 [Follower]" leader=
testLoggerAdapter verbose: server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: 2024-01-08T10:37:58.595Z [INFO]  server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: entering follower state: follower="Node at 6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf [Follower]" leader=
    testing.go:149: server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: 2024-01-08T10:37:58.595Z [INFO]  server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf: entering follower state: follower="Node at 6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf [Follower]" leader=
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.595Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: entering follower state: follower="Node at 63826fad-7260-472a-c1f5-832dec34bd74 [Follower]" leader=
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.595Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: entering follower state: follower="Node at 63826fad-7260-472a-c1f5-832dec34bd74 [Follower]" leader=
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [WARN]  server-63826fad-7260-472a-c1f5-832dec34bd74: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [WARN]  server-63826fad-7260-472a-c1f5-832dec34bd74: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: entering candidate state: node="Node at 63826fad-7260-472a-c1f5-832dec34bd74 [Candidate]" term=2
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: entering candidate state: node="Node at 63826fad-7260-472a-c1f5-832dec34bd74 [Candidate]" term=2
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: election won: tally=2
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: election won: tally=2
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: entering leader state: leader="Node at 63826fad-7260-472a-c1f5-832dec34bd74 [Leader]"
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: entering leader state: leader="Node at 63826fad-7260-472a-c1f5-832dec34bd74 [Leader]"
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: added peer, starting replication: peer=server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: added peer, starting replication: peer=server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: added peer, starting replication: peer=server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: added peer, starting replication: peer=server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: pipelining replication: peer="{Voter server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf 6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf}"
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.705Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: pipelining replication: peer="{Voter server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf 6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf}"
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.706Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: pipelining replication: peer="{Voter server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 166a5aa7-ac90-ae61-3cd7-834bf1b9af29}"
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.706Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: pipelining replication: peer="{Voter server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 166a5aa7-ac90-ae61-3cd7-834bf1b9af29}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.914Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:58.706240295 +0000 UTC m=+6.858034449 (1 nodes), 110.99489ms from start of poll, 111.524995ms from cluster start. Timeout at 2024-01-08 10:37:58.913885085 +0000 UTC m=+7.065679209, 207.64476ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:58.914Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:58.706240295 +0000 UTC m=+6.858034449 (1 nodes), 110.99489ms from start of poll, 111.524995ms from cluster start. Timeout at 2024-01-08 10:37:58.913885085 +0000 UTC m=+7.065679209, 207.64476ms after stability
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.986Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: aborting pipeline replication: peer="{Voter server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 166a5aa7-ac90-ae61-3cd7-834bf1b9af29}"
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.986Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: aborting pipeline replication: peer="{Voter server-166a5aa7-ac90-ae61-3cd7-834bf1b9af29 166a5aa7-ac90-ae61-3cd7-834bf1b9af29}"
testLoggerAdapter verbose: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.986Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: aborting pipeline replication: peer="{Voter server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf 6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf}"
    testing.go:149: server-63826fad-7260-472a-c1f5-832dec34bd74: 2024-01-08T10:37:58.986Z [INFO]  server-63826fad-7260-472a-c1f5-832dec34bd74: aborting pipeline replication: peer="{Voter server-6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf 6254d8e8-e4ca-df41-ac3c-5f4e4e4b00cf}"
--- PASS: TestRaft_ApplyConcurrent (0.39s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
testLoggerAdapter verbose: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:58.990Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63ce1f14-adac-1038-0292-03e7961d66e8 Address:63ce1f14-adac-1038-0292-03e7961d66e8}]"
    testing.go:149: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:58.990Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63ce1f14-adac-1038-0292-03e7961d66e8 Address:63ce1f14-adac-1038-0292-03e7961d66e8}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:58.990Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:58.990Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:58.990Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: entering follower state: follower="Node at 63ce1f14-adac-1038-0292-03e7961d66e8 [Follower]" leader=
    testing.go:149: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:58.990Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: entering follower state: follower="Node at 63ce1f14-adac-1038-0292-03e7961d66e8 [Follower]" leader=
testLoggerAdapter verbose: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [WARN]  server-63ce1f14-adac-1038-0292-03e7961d66e8: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [WARN]  server-63ce1f14-adac-1038-0292-03e7961d66e8: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: entering candidate state: node="Node at 63ce1f14-adac-1038-0292-03e7961d66e8 [Candidate]" term=2
    testing.go:149: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: entering candidate state: node="Node at 63ce1f14-adac-1038-0292-03e7961d66e8 [Candidate]" term=2
testLoggerAdapter verbose: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: election won: tally=1
    testing.go:149: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: election won: tally=1
testLoggerAdapter verbose: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: entering leader state: leader="Node at 63ce1f14-adac-1038-0292-03e7961d66e8 [Leader]"
    testing.go:149: server-63ce1f14-adac-1038-0292-03e7961d66e8: 2024-01-08T10:37:59.122Z [INFO]  server-63ce1f14-adac-1038-0292-03e7961d66e8: entering leader state: leader="Node at 63ce1f14-adac-1038-0292-03e7961d66e8 [Leader]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.326Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.123103814 +0000 UTC m=+7.274897952 (1 nodes), 132.677798ms from start of poll, 133.88072ms from cluster start. Timeout at 2024-01-08 10:37:59.325924073 +0000 UTC m=+7.477718211, 202.820259ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:59.326Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.123103814 +0000 UTC m=+7.274897952 (1 nodes), 132.677798ms from start of poll, 133.88072ms from cluster start. Timeout at 2024-01-08 10:37:59.325924073 +0000 UTC m=+7.477718211, 202.820259ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.34s)
=== RUN   TestRaft_JoinNode
testLoggerAdapter verbose: server-feb85222-5296-bfa0-0383-89ba856dda79: 2024-01-08T10:37:59.330Z [INFO]  server-feb85222-5296-bfa0-0383-89ba856dda79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feb85222-5296-bfa0-0383-89ba856dda79 Address:feb85222-5296-bfa0-0383-89ba856dda79} {Suffrage:Voter ID:server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 Address:3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70}]"
    testing.go:149: server-feb85222-5296-bfa0-0383-89ba856dda79: 2024-01-08T10:37:59.330Z [INFO]  server-feb85222-5296-bfa0-0383-89ba856dda79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feb85222-5296-bfa0-0383-89ba856dda79 Address:feb85222-5296-bfa0-0383-89ba856dda79} {Suffrage:Voter ID:server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 Address:3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70}]"
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.330Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feb85222-5296-bfa0-0383-89ba856dda79 Address:feb85222-5296-bfa0-0383-89ba856dda79} {Suffrage:Voter ID:server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 Address:3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70}]"
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.330Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feb85222-5296-bfa0-0383-89ba856dda79 Address:feb85222-5296-bfa0-0383-89ba856dda79} {Suffrage:Voter ID:server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 Address:3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70}]"
testLoggerAdapter verbose: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.331Z [INFO]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: initial configuration: index=0 servers=[]
    testing.go:149: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.331Z [INFO]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.331Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:59.331Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-feb85222-5296-bfa0-0383-89ba856dda79: 2024-01-08T10:37:59.331Z [INFO]  server-feb85222-5296-bfa0-0383-89ba856dda79: entering follower state: follower="Node at feb85222-5296-bfa0-0383-89ba856dda79 [Follower]" leader=
    testing.go:149: server-feb85222-5296-bfa0-0383-89ba856dda79: 2024-01-08T10:37:59.331Z [INFO]  server-feb85222-5296-bfa0-0383-89ba856dda79: entering follower state: follower="Node at feb85222-5296-bfa0-0383-89ba856dda79 [Follower]" leader=
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.331Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: entering follower state: follower="Node at 3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 [Follower]" leader=
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.331Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: entering follower state: follower="Node at 3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 [Follower]" leader=
testLoggerAdapter verbose: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.331Z [INFO]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: entering follower state: follower="Node at a4b0e509-d79a-99fe-abbc-798b2ec1a97a [Follower]" leader=
    testing.go:149: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.331Z [INFO]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: entering follower state: follower="Node at a4b0e509-d79a-99fe-abbc-798b2ec1a97a [Follower]" leader=
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.404Z [WARN]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.404Z [WARN]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.405Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: entering candidate state: node="Node at 3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 [Candidate]" term=2
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.405Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: entering candidate state: node="Node at 3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 [Candidate]" term=2
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.405Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: election won: tally=2
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.405Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: election won: tally=2
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.406Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: entering leader state: leader="Node at 3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 [Leader]"
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.406Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: entering leader state: leader="Node at 3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 [Leader]"
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.406Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: added peer, starting replication: peer=server-feb85222-5296-bfa0-0383-89ba856dda79
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.406Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: added peer, starting replication: peer=server-feb85222-5296-bfa0-0383-89ba856dda79
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.406Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: pipelining replication: peer="{Voter server-feb85222-5296-bfa0-0383-89ba856dda79 feb85222-5296-bfa0-0383-89ba856dda79}"
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.406Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: pipelining replication: peer="{Voter server-feb85222-5296-bfa0-0383-89ba856dda79 feb85222-5296-bfa0-0383-89ba856dda79}"
testLoggerAdapter verbose: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.421Z [WARN]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: no known peers, aborting election
    testing.go:149: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.421Z [WARN]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: no known peers, aborting election
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.522Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.40707159 +0000 UTC m=+7.558865732 (1 nodes), 75.847827ms from start of poll, 76.508703ms from cluster start. Timeout at 2024-01-08 10:37:59.521934511 +0000 UTC m=+7.673728630, 114.862898ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:59.522Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.40707159 +0000 UTC m=+7.558865732 (1 nodes), 75.847827ms from start of poll, 76.508703ms from cluster start. Timeout at 2024-01-08 10:37:59.521934511 +0000 UTC m=+7.673728630, 114.862898ms after stability
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.522Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: updating configuration: command=AddStaging server-id=server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a server-addr=a4b0e509-d79a-99fe-abbc-798b2ec1a97a servers="[{Suffrage:Voter ID:server-feb85222-5296-bfa0-0383-89ba856dda79 Address:feb85222-5296-bfa0-0383-89ba856dda79} {Suffrage:Voter ID:server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 Address:3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70} {Suffrage:Voter ID:server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a Address:a4b0e509-d79a-99fe-abbc-798b2ec1a97a}]"
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.522Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: updating configuration: command=AddStaging server-id=server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a server-addr=a4b0e509-d79a-99fe-abbc-798b2ec1a97a servers="[{Suffrage:Voter ID:server-feb85222-5296-bfa0-0383-89ba856dda79 Address:feb85222-5296-bfa0-0383-89ba856dda79} {Suffrage:Voter ID:server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70 Address:3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70} {Suffrage:Voter ID:server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a Address:a4b0e509-d79a-99fe-abbc-798b2ec1a97a}]"
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.523Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: added peer, starting replication: peer=server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.523Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: added peer, starting replication: peer=server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a
testLoggerAdapter verbose: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.523Z [WARN]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: failed to get previous log: previous-index=3 last-index=0 error="log not found"
    testing.go:149: server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: 2024-01-08T10:37:59.523Z [WARN]  server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a: failed to get previous log: previous-index=3 last-index=0 error="log not found"
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.524Z [WARN]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: appendEntries rejected, sending older logs: peer="{Voter server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a a4b0e509-d79a-99fe-abbc-798b2ec1a97a}" next=1
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.524Z [WARN]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: appendEntries rejected, sending older logs: peer="{Voter server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a a4b0e509-d79a-99fe-abbc-798b2ec1a97a}" next=1
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.524Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: pipelining replication: peer="{Voter server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a a4b0e509-d79a-99fe-abbc-798b2ec1a97a}"
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.524Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: pipelining replication: peer="{Voter server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a a4b0e509-d79a-99fe-abbc-798b2ec1a97a}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.524Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:59.524Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.634Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.525201917 +0000 UTC m=+7.676996043 (1 nodes), 2.45µs from start of poll, 194.639014ms from cluster start. Timeout at 2024-01-08 10:37:59.633850906 +0000 UTC m=+7.785645044, 108.649001ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:59.634Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.525201917 +0000 UTC m=+7.676996043 (1 nodes), 2.45µs from start of poll, 194.639014ms from cluster start. Timeout at 2024-01-08 10:37:59.633850906 +0000 UTC m=+7.785645044, 108.649001ms after stability
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.634Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: aborting pipeline replication: peer="{Voter server-feb85222-5296-bfa0-0383-89ba856dda79 feb85222-5296-bfa0-0383-89ba856dda79}"
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.634Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: aborting pipeline replication: peer="{Voter server-feb85222-5296-bfa0-0383-89ba856dda79 feb85222-5296-bfa0-0383-89ba856dda79}"
testLoggerAdapter verbose: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.634Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: aborting pipeline replication: peer="{Voter server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a a4b0e509-d79a-99fe-abbc-798b2ec1a97a}"
    testing.go:149: server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: 2024-01-08T10:37:59.634Z [INFO]  server-3cc7c253-7f41-4b47-7f5e-9ae3f81d0f70: aborting pipeline replication: peer="{Voter server-a4b0e509-d79a-99fe-abbc-798b2ec1a97a a4b0e509-d79a-99fe-abbc-798b2ec1a97a}"
--- PASS: TestRaft_JoinNode (0.31s)
=== RUN   TestRaft_JoinNode_ConfigStore
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.638Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8c2b609-a396-d65e-45f9-3a954f8530bd Address:f8c2b609-a396-d65e-45f9-3a954f8530bd}]"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.638Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8c2b609-a396-d65e-45f9-3a954f8530bd Address:f8c2b609-a396-d65e-45f9-3a954f8530bd}]"
testLoggerAdapter verbose: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.639Z [INFO]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: initial configuration: index=0 servers=[]
    testing.go:149: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.639Z [INFO]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.640Z [INFO]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: initial configuration: index=0 servers=[]
    testing.go:149: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.640Z [INFO]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.640Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:59.640Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.640Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: entering follower state: follower="Node at f8c2b609-a396-d65e-45f9-3a954f8530bd [Follower]" leader=
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.640Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: entering follower state: follower="Node at f8c2b609-a396-d65e-45f9-3a954f8530bd [Follower]" leader=
testLoggerAdapter verbose: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.640Z [INFO]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: entering follower state: follower="Node at 65330ca5-3335-8d48-175f-81f7a5d07b8e [Follower]" leader=
    testing.go:149: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.640Z [INFO]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: entering follower state: follower="Node at 65330ca5-3335-8d48-175f-81f7a5d07b8e [Follower]" leader=
testLoggerAdapter verbose: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.641Z [INFO]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: entering follower state: follower="Node at 3b55b2ef-97da-0f62-e94f-bc736faf72be [Follower]" leader=
    testing.go:149: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.641Z [INFO]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: entering follower state: follower="Node at 3b55b2ef-97da-0f62-e94f-bc736faf72be [Follower]" leader=
testLoggerAdapter verbose: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.714Z [WARN]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: no known peers, aborting election
    testing.go:149: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.714Z [WARN]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: no known peers, aborting election
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.721Z [WARN]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.721Z [WARN]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.722Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: entering candidate state: node="Node at f8c2b609-a396-d65e-45f9-3a954f8530bd [Candidate]" term=2
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.722Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: entering candidate state: node="Node at f8c2b609-a396-d65e-45f9-3a954f8530bd [Candidate]" term=2
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.722Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: election won: tally=1
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.722Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: election won: tally=1
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.722Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: entering leader state: leader="Node at f8c2b609-a396-d65e-45f9-3a954f8530bd [Leader]"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.722Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: entering leader state: leader="Node at f8c2b609-a396-d65e-45f9-3a954f8530bd [Leader]"
testLoggerAdapter verbose: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.722Z [WARN]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: no known peers, aborting election
    testing.go:149: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.722Z [WARN]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: no known peers, aborting election
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.830Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.723143436 +0000 UTC m=+7.874937574 (1 nodes), 82.665467ms from start of poll, 85.565863ms from cluster start. Timeout at 2024-01-08 10:37:59.82988355 +0000 UTC m=+7.981677689, 106.740115ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:59.830Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.723143436 +0000 UTC m=+7.874937574 (1 nodes), 82.665467ms from start of poll, 85.565863ms from cluster start. Timeout at 2024-01-08 10:37:59.82988355 +0000 UTC m=+7.981677689, 106.740115ms after stability
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.830Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: updating configuration: command=AddStaging server-id=server-65330ca5-3335-8d48-175f-81f7a5d07b8e server-addr=65330ca5-3335-8d48-175f-81f7a5d07b8e servers="[{Suffrage:Voter ID:server-f8c2b609-a396-d65e-45f9-3a954f8530bd Address:f8c2b609-a396-d65e-45f9-3a954f8530bd} {Suffrage:Voter ID:server-65330ca5-3335-8d48-175f-81f7a5d07b8e Address:65330ca5-3335-8d48-175f-81f7a5d07b8e}]"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.830Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: updating configuration: command=AddStaging server-id=server-65330ca5-3335-8d48-175f-81f7a5d07b8e server-addr=65330ca5-3335-8d48-175f-81f7a5d07b8e servers="[{Suffrage:Voter ID:server-f8c2b609-a396-d65e-45f9-3a954f8530bd Address:f8c2b609-a396-d65e-45f9-3a954f8530bd} {Suffrage:Voter ID:server-65330ca5-3335-8d48-175f-81f7a5d07b8e Address:65330ca5-3335-8d48-175f-81f7a5d07b8e}]"
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.830Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: added peer, starting replication: peer=server-65330ca5-3335-8d48-175f-81f7a5d07b8e
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.830Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: added peer, starting replication: peer=server-65330ca5-3335-8d48-175f-81f7a5d07b8e
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.831Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:59.831Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.831Z [WARN]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: failed to get previous log: previous-index=3 last-index=0 error="log not found"
    testing.go:149: server-65330ca5-3335-8d48-175f-81f7a5d07b8e: 2024-01-08T10:37:59.831Z [WARN]  server-65330ca5-3335-8d48-175f-81f7a5d07b8e: failed to get previous log: previous-index=3 last-index=0 error="log not found"
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.831Z [WARN]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: appendEntries rejected, sending older logs: peer="{Voter server-65330ca5-3335-8d48-175f-81f7a5d07b8e 65330ca5-3335-8d48-175f-81f7a5d07b8e}" next=1
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.831Z [WARN]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: appendEntries rejected, sending older logs: peer="{Voter server-65330ca5-3335-8d48-175f-81f7a5d07b8e 65330ca5-3335-8d48-175f-81f7a5d07b8e}" next=1
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.831Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: pipelining replication: peer="{Voter server-65330ca5-3335-8d48-175f-81f7a5d07b8e 65330ca5-3335-8d48-175f-81f7a5d07b8e}"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.831Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: pipelining replication: peer="{Voter server-65330ca5-3335-8d48-175f-81f7a5d07b8e 65330ca5-3335-8d48-175f-81f7a5d07b8e}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.938Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.831216899 +0000 UTC m=+7.983011028 (1 nodes), 2.499µs from start of poll, 193.639317ms from cluster start. Timeout at 2024-01-08 10:37:59.938120845 +0000 UTC m=+8.089914962, 106.903934ms after stability
    testing.go:149: cluster: 2024-01-08T10:37:59.938Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.831216899 +0000 UTC m=+7.983011028 (1 nodes), 2.499µs from start of poll, 193.639317ms from cluster start. Timeout at 2024-01-08 10:37:59.938120845 +0000 UTC m=+8.089914962, 106.903934ms after stability
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.938Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: updating configuration: command=AddStaging server-id=server-3b55b2ef-97da-0f62-e94f-bc736faf72be server-addr=3b55b2ef-97da-0f62-e94f-bc736faf72be servers="[{Suffrage:Voter ID:server-f8c2b609-a396-d65e-45f9-3a954f8530bd Address:f8c2b609-a396-d65e-45f9-3a954f8530bd} {Suffrage:Voter ID:server-65330ca5-3335-8d48-175f-81f7a5d07b8e Address:65330ca5-3335-8d48-175f-81f7a5d07b8e} {Suffrage:Voter ID:server-3b55b2ef-97da-0f62-e94f-bc736faf72be Address:3b55b2ef-97da-0f62-e94f-bc736faf72be}]"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.938Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: updating configuration: command=AddStaging server-id=server-3b55b2ef-97da-0f62-e94f-bc736faf72be server-addr=3b55b2ef-97da-0f62-e94f-bc736faf72be servers="[{Suffrage:Voter ID:server-f8c2b609-a396-d65e-45f9-3a954f8530bd Address:f8c2b609-a396-d65e-45f9-3a954f8530bd} {Suffrage:Voter ID:server-65330ca5-3335-8d48-175f-81f7a5d07b8e Address:65330ca5-3335-8d48-175f-81f7a5d07b8e} {Suffrage:Voter ID:server-3b55b2ef-97da-0f62-e94f-bc736faf72be Address:3b55b2ef-97da-0f62-e94f-bc736faf72be}]"
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.938Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: added peer, starting replication: peer=server-3b55b2ef-97da-0f62-e94f-bc736faf72be
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.938Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: added peer, starting replication: peer=server-3b55b2ef-97da-0f62-e94f-bc736faf72be
testLoggerAdapter verbose: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.939Z [WARN]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: failed to get previous log: previous-index=4 last-index=0 error="log not found"
    testing.go:149: server-3b55b2ef-97da-0f62-e94f-bc736faf72be: 2024-01-08T10:37:59.939Z [WARN]  server-3b55b2ef-97da-0f62-e94f-bc736faf72be: failed to get previous log: previous-index=4 last-index=0 error="log not found"
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.939Z [WARN]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: appendEntries rejected, sending older logs: peer="{Voter server-3b55b2ef-97da-0f62-e94f-bc736faf72be 3b55b2ef-97da-0f62-e94f-bc736faf72be}" next=1
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.939Z [WARN]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: appendEntries rejected, sending older logs: peer="{Voter server-3b55b2ef-97da-0f62-e94f-bc736faf72be 3b55b2ef-97da-0f62-e94f-bc736faf72be}" next=1
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.939Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: pipelining replication: peer="{Voter server-3b55b2ef-97da-0f62-e94f-bc736faf72be 3b55b2ef-97da-0f62-e94f-bc736faf72be}"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:37:59.939Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: pipelining replication: peer="{Voter server-3b55b2ef-97da-0f62-e94f-bc736faf72be 3b55b2ef-97da-0f62-e94f-bc736faf72be}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:37:59.940Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:37:59.940Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.046Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.94105412 +0000 UTC m=+8.092848248 (1 nodes), 2.618µs from start of poll, 303.476537ms from cluster start. Timeout at 2024-01-08 10:38:00.046064514 +0000 UTC m=+8.197858656, 105.010408ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:00.046Z [INFO]  stable state for Leader reached at 2024-01-08 10:37:59.94105412 +0000 UTC m=+8.092848248 (1 nodes), 2.618µs from start of poll, 303.476537ms from cluster start. Timeout at 2024-01-08 10:38:00.046064514 +0000 UTC m=+8.197858656, 105.010408ms after stability
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:38:00.046Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: aborting pipeline replication: peer="{Voter server-3b55b2ef-97da-0f62-e94f-bc736faf72be 3b55b2ef-97da-0f62-e94f-bc736faf72be}"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:38:00.046Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: aborting pipeline replication: peer="{Voter server-3b55b2ef-97da-0f62-e94f-bc736faf72be 3b55b2ef-97da-0f62-e94f-bc736faf72be}"
testLoggerAdapter verbose: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:38:00.046Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: aborting pipeline replication: peer="{Voter server-65330ca5-3335-8d48-175f-81f7a5d07b8e 65330ca5-3335-8d48-175f-81f7a5d07b8e}"
    testing.go:149: server-f8c2b609-a396-d65e-45f9-3a954f8530bd: 2024-01-08T10:38:00.046Z [INFO]  server-f8c2b609-a396-d65e-45f9-3a954f8530bd: aborting pipeline replication: peer="{Voter server-65330ca5-3335-8d48-175f-81f7a5d07b8e 65330ca5-3335-8d48-175f-81f7a5d07b8e}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.41s)
=== RUN   TestRaft_RemoveFollower
testLoggerAdapter verbose: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.050Z [INFO]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 Address:84b70fe6-c76d-64db-55d2-6d8ba56b05a0} {Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
    testing.go:149: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.050Z [INFO]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 Address:84b70fe6-c76d-64db-55d2-6d8ba56b05a0} {Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
testLoggerAdapter verbose: server-22547879-cf85-d863-3037-0bc987d590a5: 2024-01-08T10:38:00.051Z [INFO]  server-22547879-cf85-d863-3037-0bc987d590a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 Address:84b70fe6-c76d-64db-55d2-6d8ba56b05a0} {Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
    testing.go:149: server-22547879-cf85-d863-3037-0bc987d590a5: 2024-01-08T10:38:00.051Z [INFO]  server-22547879-cf85-d863-3037-0bc987d590a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 Address:84b70fe6-c76d-64db-55d2-6d8ba56b05a0} {Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.052Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 Address:84b70fe6-c76d-64db-55d2-6d8ba56b05a0} {Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.052Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 Address:84b70fe6-c76d-64db-55d2-6d8ba56b05a0} {Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.052Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:00.052Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.052Z [INFO]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: entering follower state: follower="Node at 84b70fe6-c76d-64db-55d2-6d8ba56b05a0 [Follower]" leader=
    testing.go:149: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.052Z [INFO]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: entering follower state: follower="Node at 84b70fe6-c76d-64db-55d2-6d8ba56b05a0 [Follower]" leader=
testLoggerAdapter verbose: server-22547879-cf85-d863-3037-0bc987d590a5: 2024-01-08T10:38:00.052Z [INFO]  server-22547879-cf85-d863-3037-0bc987d590a5: entering follower state: follower="Node at 22547879-cf85-d863-3037-0bc987d590a5 [Follower]" leader=
    testing.go:149: server-22547879-cf85-d863-3037-0bc987d590a5: 2024-01-08T10:38:00.052Z [INFO]  server-22547879-cf85-d863-3037-0bc987d590a5: entering follower state: follower="Node at 22547879-cf85-d863-3037-0bc987d590a5 [Follower]" leader=
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.053Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: entering follower state: follower="Node at f9d19753-8b02-b0b4-8691-a94389c53005 [Follower]" leader=
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.053Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: entering follower state: follower="Node at f9d19753-8b02-b0b4-8691-a94389c53005 [Follower]" leader=
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.118Z [WARN]  server-f9d19753-8b02-b0b4-8691-a94389c53005: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.118Z [WARN]  server-f9d19753-8b02-b0b4-8691-a94389c53005: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: entering candidate state: node="Node at f9d19753-8b02-b0b4-8691-a94389c53005 [Candidate]" term=2
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: entering candidate state: node="Node at f9d19753-8b02-b0b4-8691-a94389c53005 [Candidate]" term=2
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: election won: tally=2
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: election won: tally=2
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: entering leader state: leader="Node at f9d19753-8b02-b0b4-8691-a94389c53005 [Leader]"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: entering leader state: leader="Node at f9d19753-8b02-b0b4-8691-a94389c53005 [Leader]"
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: added peer, starting replication: peer=server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.119Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: added peer, starting replication: peer=server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.120Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: added peer, starting replication: peer=server-22547879-cf85-d863-3037-0bc987d590a5
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.120Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: added peer, starting replication: peer=server-22547879-cf85-d863-3037-0bc987d590a5
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.120Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: pipelining replication: peer="{Voter server-22547879-cf85-d863-3037-0bc987d590a5 22547879-cf85-d863-3037-0bc987d590a5}"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.120Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: pipelining replication: peer="{Voter server-22547879-cf85-d863-3037-0bc987d590a5 22547879-cf85-d863-3037-0bc987d590a5}"
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.121Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: pipelining replication: peer="{Voter server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 84b70fe6-c76d-64db-55d2-6d8ba56b05a0}"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.121Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: pipelining replication: peer="{Voter server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 84b70fe6-c76d-64db-55d2-6d8ba56b05a0}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.229Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:00.12153303 +0000 UTC m=+8.273327200 (1 nodes), 68.87672ms from start of poll, 71.134863ms from cluster start. Timeout at 2024-01-08 10:38:00.229050753 +0000 UTC m=+8.380844879, 107.517679ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:00.229Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:00.12153303 +0000 UTC m=+8.273327200 (1 nodes), 68.87672ms from start of poll, 71.134863ms from cluster start. Timeout at 2024-01-08 10:38:00.229050753 +0000 UTC m=+8.380844879, 107.517679ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.234Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:00.234Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.342Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:00.234951916 +0000 UTC m=+8.386746030 (2 nodes), 4.984µs from start of poll, 184.553693ms from cluster start. Timeout at 2024-01-08 10:38:00.342022669 +0000 UTC m=+8.493816803, 107.070773ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:00.342Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:00.234951916 +0000 UTC m=+8.386746030 (2 nodes), 4.984µs from start of poll, 184.553693ms from cluster start. Timeout at 2024-01-08 10:38:00.342022669 +0000 UTC m=+8.493816803, 107.070773ms after stability
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.342Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: updating configuration: command=RemoveServer server-id=server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 server-addr= servers="[{Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.342Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: updating configuration: command=RemoveServer server-id=server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 server-addr= servers="[{Suffrage:Voter ID:server-22547879-cf85-d863-3037-0bc987d590a5 Address:22547879-cf85-d863-3037-0bc987d590a5} {Suffrage:Voter ID:server-f9d19753-8b02-b0b4-8691-a94389c53005 Address:f9d19753-8b02-b0b4-8691-a94389c53005}]"
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.342Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: removed peer, stopping replication: peer=server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 last-index=3
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.342Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: removed peer, stopping replication: peer=server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 last-index=3
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.343Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: aborting pipeline replication: peer="{Voter server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 84b70fe6-c76d-64db-55d2-6d8ba56b05a0}"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.343Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: aborting pipeline replication: peer="{Voter server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0 84b70fe6-c76d-64db-55d2-6d8ba56b05a0}"
testLoggerAdapter verbose: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.438Z [WARN]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: heartbeat timeout reached, starting election: last-leader=f9d19753-8b02-b0b4-8691-a94389c53005
    testing.go:149: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.438Z [WARN]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: heartbeat timeout reached, starting election: last-leader=f9d19753-8b02-b0b4-8691-a94389c53005
testLoggerAdapter verbose: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.438Z [INFO]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: entering candidate state: node="Node at 84b70fe6-c76d-64db-55d2-6d8ba56b05a0 [Candidate]" term=3
    testing.go:149: server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: 2024-01-08T10:38:00.438Z [INFO]  server-84b70fe6-c76d-64db-55d2-6d8ba56b05a0: entering candidate state: node="Node at 84b70fe6-c76d-64db-55d2-6d8ba56b05a0 [Candidate]" term=3
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.438Z [WARN]  server-f9d19753-8b02-b0b4-8691-a94389c53005: rejecting vote request since we have a leader: from=84b70fe6-c76d-64db-55d2-6d8ba56b05a0 leader=f9d19753-8b02-b0b4-8691-a94389c53005
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.438Z [WARN]  server-f9d19753-8b02-b0b4-8691-a94389c53005: rejecting vote request since we have a leader: from=84b70fe6-c76d-64db-55d2-6d8ba56b05a0 leader=f9d19753-8b02-b0b4-8691-a94389c53005
testLoggerAdapter verbose: server-22547879-cf85-d863-3037-0bc987d590a5: 2024-01-08T10:38:00.439Z [WARN]  server-22547879-cf85-d863-3037-0bc987d590a5: rejecting vote request since we have a leader: from=84b70fe6-c76d-64db-55d2-6d8ba56b05a0 leader=f9d19753-8b02-b0b4-8691-a94389c53005
    testing.go:149: server-22547879-cf85-d863-3037-0bc987d590a5: 2024-01-08T10:38:00.439Z [WARN]  server-22547879-cf85-d863-3037-0bc987d590a5: rejecting vote request since we have a leader: from=84b70fe6-c76d-64db-55d2-6d8ba56b05a0 leader=f9d19753-8b02-b0b4-8691-a94389c53005
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.450Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: aborting pipeline replication: peer="{Voter server-22547879-cf85-d863-3037-0bc987d590a5 22547879-cf85-d863-3037-0bc987d590a5}"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.450Z [INFO]  server-f9d19753-8b02-b0b4-8691-a94389c53005: aborting pipeline replication: peer="{Voter server-22547879-cf85-d863-3037-0bc987d590a5 22547879-cf85-d863-3037-0bc987d590a5}"
testLoggerAdapter verbose: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.505Z [ERROR] server-f9d19753-8b02-b0b4-8691-a94389c53005: failed to heartbeat to: peer=22547879-cf85-d863-3037-0bc987d590a5 error="command timed out"
    testing.go:149: server-f9d19753-8b02-b0b4-8691-a94389c53005: 2024-01-08T10:38:00.505Z [ERROR] server-f9d19753-8b02-b0b4-8691-a94389c53005: failed to heartbeat to: peer=22547879-cf85-d863-3037-0bc987d590a5 error="command timed out"
--- PASS: TestRaft_RemoveFollower (0.46s)
=== RUN   TestRaft_RemoveLeader
testLoggerAdapter verbose: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.510Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-5975690d-1868-8a83-6ce3-4ae1c6ed04de Address:5975690d-1868-8a83-6ce3-4ae1c6ed04de} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
    testing.go:149: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.510Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-5975690d-1868-8a83-6ce3-4ae1c6ed04de Address:5975690d-1868-8a83-6ce3-4ae1c6ed04de} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.511Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-5975690d-1868-8a83-6ce3-4ae1c6ed04de Address:5975690d-1868-8a83-6ce3-4ae1c6ed04de} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.511Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-5975690d-1868-8a83-6ce3-4ae1c6ed04de Address:5975690d-1868-8a83-6ce3-4ae1c6ed04de} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.512Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-5975690d-1868-8a83-6ce3-4ae1c6ed04de Address:5975690d-1868-8a83-6ce3-4ae1c6ed04de} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.512Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-5975690d-1868-8a83-6ce3-4ae1c6ed04de Address:5975690d-1868-8a83-6ce3-4ae1c6ed04de} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.512Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:00.512Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.512Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: entering follower state: follower="Node at d83d07be-447f-0bc8-4019-bcb5a108c89b [Follower]" leader=
    testing.go:149: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.512Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: entering follower state: follower="Node at d83d07be-447f-0bc8-4019-bcb5a108c89b [Follower]" leader=
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.512Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: entering follower state: follower="Node at 5975690d-1868-8a83-6ce3-4ae1c6ed04de [Follower]" leader=
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.512Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: entering follower state: follower="Node at 5975690d-1868-8a83-6ce3-4ae1c6ed04de [Follower]" leader=
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.513Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering follower state: follower="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Follower]" leader=
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.513Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering follower state: follower="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Follower]" leader=
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.582Z [WARN]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.582Z [WARN]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.582Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: entering candidate state: node="Node at 5975690d-1868-8a83-6ce3-4ae1c6ed04de [Candidate]" term=2
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.582Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: entering candidate state: node="Node at 5975690d-1868-8a83-6ce3-4ae1c6ed04de [Candidate]" term=2
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.582Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: election won: tally=2
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.582Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: election won: tally=2
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: entering leader state: leader="Node at 5975690d-1868-8a83-6ce3-4ae1c6ed04de [Leader]"
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: entering leader state: leader="Node at 5975690d-1868-8a83-6ce3-4ae1c6ed04de [Leader]"
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: added peer, starting replication: peer=server-d83d07be-447f-0bc8-4019-bcb5a108c89b
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: added peer, starting replication: peer=server-d83d07be-447f-0bc8-4019-bcb5a108c89b
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: added peer, starting replication: peer=server-7a048273-947a-e9e8-e289-44a30dd899ff
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: added peer, starting replication: peer=server-7a048273-947a-e9e8-e289-44a30dd899ff
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: pipelining replication: peer="{Voter server-7a048273-947a-e9e8-e289-44a30dd899ff 7a048273-947a-e9e8-e289-44a30dd899ff}"
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.583Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: pipelining replication: peer="{Voter server-7a048273-947a-e9e8-e289-44a30dd899ff 7a048273-947a-e9e8-e289-44a30dd899ff}"
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.584Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: pipelining replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.584Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: pipelining replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.694Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:00.585196963 +0000 UTC m=+8.736991136 (1 nodes), 72.617444ms from start of poll, 75.967886ms from cluster start. Timeout at 2024-01-08 10:38:00.694087325 +0000 UTC m=+8.845881486, 108.89035ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:00.694Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:00.585196963 +0000 UTC m=+8.736991136 (1 nodes), 72.617444ms from start of poll, 75.967886ms from cluster start. Timeout at 2024-01-08 10:38:00.694087325 +0000 UTC m=+8.845881486, 108.89035ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.701Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:00.701Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:00.810Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:00.70229509 +0000 UTC m=+8.854089216 (2 nodes), 5.042µs from start of poll, 193.065966ms from cluster start. Timeout at 2024-01-08 10:38:00.809962127 +0000 UTC m=+8.961756365, 107.667149ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:00.810Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:00.70229509 +0000 UTC m=+8.854089216 (2 nodes), 5.042µs from start of poll, 193.065966ms from cluster start. Timeout at 2024-01-08 10:38:00.809962127 +0000 UTC m=+8.961756365, 107.667149ms after stability
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.810Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: updating configuration: command=RemoveServer server-id=server-5975690d-1868-8a83-6ce3-4ae1c6ed04de server-addr= servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.810Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: updating configuration: command=RemoveServer server-id=server-5975690d-1868-8a83-6ce3-4ae1c6ed04de server-addr= servers="[{Suffrage:Voter ID:server-d83d07be-447f-0bc8-4019-bcb5a108c89b Address:d83d07be-447f-0bc8-4019-bcb5a108c89b} {Suffrage:Voter ID:server-7a048273-947a-e9e8-e289-44a30dd899ff Address:7a048273-947a-e9e8-e289-44a30dd899ff}]"
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.811Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: removed ourself, shutting down
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.811Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: removed ourself, shutting down
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.811Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: aborting pipeline replication: peer="{Voter server-7a048273-947a-e9e8-e289-44a30dd899ff 7a048273-947a-e9e8-e289-44a30dd899ff}"
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.811Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: aborting pipeline replication: peer="{Voter server-7a048273-947a-e9e8-e289-44a30dd899ff 7a048273-947a-e9e8-e289-44a30dd899ff}"
testLoggerAdapter verbose: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.811Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: aborting pipeline replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
    testing.go:149: server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: 2024-01-08T10:38:00.811Z [INFO]  server-5975690d-1868-8a83-6ce3-4ae1c6ed04de: aborting pipeline replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.914Z [WARN]  server-7a048273-947a-e9e8-e289-44a30dd899ff: heartbeat timeout reached, starting election: last-leader=5975690d-1868-8a83-6ce3-4ae1c6ed04de
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.914Z [WARN]  server-7a048273-947a-e9e8-e289-44a30dd899ff: heartbeat timeout reached, starting election: last-leader=5975690d-1868-8a83-6ce3-4ae1c6ed04de
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.914Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering candidate state: node="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Candidate]" term=3
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.914Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering candidate state: node="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Candidate]" term=3
testLoggerAdapter verbose: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.914Z [WARN]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: rejecting vote request since we have a leader: from=7a048273-947a-e9e8-e289-44a30dd899ff leader=5975690d-1868-8a83-6ce3-4ae1c6ed04de
    testing.go:149: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.914Z [WARN]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: rejecting vote request since we have a leader: from=7a048273-947a-e9e8-e289-44a30dd899ff leader=5975690d-1868-8a83-6ce3-4ae1c6ed04de
testLoggerAdapter verbose: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.934Z [WARN]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: heartbeat timeout reached, starting election: last-leader=5975690d-1868-8a83-6ce3-4ae1c6ed04de
    testing.go:149: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.934Z [WARN]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: heartbeat timeout reached, starting election: last-leader=5975690d-1868-8a83-6ce3-4ae1c6ed04de
testLoggerAdapter verbose: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.934Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: entering candidate state: node="Node at d83d07be-447f-0bc8-4019-bcb5a108c89b [Candidate]" term=3
    testing.go:149: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:00.934Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: entering candidate state: node="Node at d83d07be-447f-0bc8-4019-bcb5a108c89b [Candidate]" term=3
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.934Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: duplicate requestVote for same term: term=3
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:00.934Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: duplicate requestVote for same term: term=3
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.006Z [WARN]  server-7a048273-947a-e9e8-e289-44a30dd899ff: Election timeout reached, restarting election
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.006Z [WARN]  server-7a048273-947a-e9e8-e289-44a30dd899ff: Election timeout reached, restarting election
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.006Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering candidate state: node="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Candidate]" term=4
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.006Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering candidate state: node="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Candidate]" term=4
testLoggerAdapter verbose: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:01.006Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: entering follower state: follower="Node at d83d07be-447f-0bc8-4019-bcb5a108c89b [Follower]" leader=
    testing.go:149: server-d83d07be-447f-0bc8-4019-bcb5a108c89b: 2024-01-08T10:38:01.006Z [INFO]  server-d83d07be-447f-0bc8-4019-bcb5a108c89b: entering follower state: follower="Node at d83d07be-447f-0bc8-4019-bcb5a108c89b [Follower]" leader=
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.007Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: election won: tally=2
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.007Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: election won: tally=2
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.007Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering leader state: leader="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Leader]"
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.007Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: entering leader state: leader="Node at 7a048273-947a-e9e8-e289-44a30dd899ff [Leader]"
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.007Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: added peer, starting replication: peer=server-d83d07be-447f-0bc8-4019-bcb5a108c89b
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.007Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: added peer, starting replication: peer=server-d83d07be-447f-0bc8-4019-bcb5a108c89b
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.008Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: pipelining replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.008Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: pipelining replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.025Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:01.025Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.138Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.025947315 +0000 UTC m=+9.177741167 (1 nodes), 43.009µs from start of poll, 516.717917ms from cluster start. Timeout at 2024-01-08 10:38:01.138269052 +0000 UTC m=+9.290063193, 112.322026ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:01.138Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.025947315 +0000 UTC m=+9.177741167 (1 nodes), 43.009µs from start of poll, 516.717917ms from cluster start. Timeout at 2024-01-08 10:38:01.138269052 +0000 UTC m=+9.290063193, 112.322026ms after stability
testLoggerAdapter verbose: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.138Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: aborting pipeline replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
    testing.go:149: server-7a048273-947a-e9e8-e289-44a30dd899ff: 2024-01-08T10:38:01.138Z [INFO]  server-7a048273-947a-e9e8-e289-44a30dd899ff: aborting pipeline replication: peer="{Voter server-d83d07be-447f-0bc8-4019-bcb5a108c89b d83d07be-447f-0bc8-4019-bcb5a108c89b}"
--- PASS: TestRaft_RemoveLeader (0.63s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.143Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-230b57ef-cd8a-8661-7c26-df088e513909 Address:230b57ef-cd8a-8661-7c26-df088e513909} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.143Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-230b57ef-cd8a-8661-7c26-df088e513909 Address:230b57ef-cd8a-8661-7c26-df088e513909} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.144Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-230b57ef-cd8a-8661-7c26-df088e513909 Address:230b57ef-cd8a-8661-7c26-df088e513909} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.144Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-230b57ef-cd8a-8661-7c26-df088e513909 Address:230b57ef-cd8a-8661-7c26-df088e513909} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.145Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-230b57ef-cd8a-8661-7c26-df088e513909 Address:230b57ef-cd8a-8661-7c26-df088e513909} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.145Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering follower state: follower="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Follower]" leader=
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.146Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering follower state: follower="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Follower]" leader=
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.145Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-230b57ef-cd8a-8661-7c26-df088e513909 Address:230b57ef-cd8a-8661-7c26-df088e513909} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.148Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:01.148Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.145Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering follower state: follower="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Follower]" leader=
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.146Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering follower state: follower="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Follower]" leader=
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.149Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering follower state: follower="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Follower]" leader=
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.149Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering follower state: follower="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Follower]" leader=
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.209Z [WARN]  server-230b57ef-cd8a-8661-7c26-df088e513909: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.209Z [WARN]  server-230b57ef-cd8a-8661-7c26-df088e513909: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering candidate state: node="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Candidate]" term=2
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering candidate state: node="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Candidate]" term=2
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: election won: tally=2
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: election won: tally=2
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering leader state: leader="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Leader]"
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering leader state: leader="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Leader]"
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: added peer, starting replication: peer=server-ebf30ee7-202e-5f99-3085-d984e07eb50e
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.210Z [WARN]  server-a374c972-55fc-f15c-3680-3b7df70966f9: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: added peer, starting replication: peer=server-ebf30ee7-202e-5f99-3085-d984e07eb50e
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: added peer, starting replication: peer=server-a374c972-55fc-f15c-3680-3b7df70966f9
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.210Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: added peer, starting replication: peer=server-a374c972-55fc-f15c-3680-3b7df70966f9
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.210Z [WARN]  server-a374c972-55fc-f15c-3680-3b7df70966f9: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.211Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering candidate state: node="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Candidate]" term=2
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.211Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering candidate state: node="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Candidate]" term=2
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.211Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: duplicate requestVote for same term: term=2
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.211Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: duplicate requestVote for same term: term=2
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.211Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering follower state: follower="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Follower]" leader=230b57ef-cd8a-8661-7c26-df088e513909
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.211Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering follower state: follower="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Follower]" leader=230b57ef-cd8a-8661-7c26-df088e513909
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.211Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: pipelining replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.211Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: pipelining replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.212Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: pipelining replication: peer="{Voter server-ebf30ee7-202e-5f99-3085-d984e07eb50e ebf30ee7-202e-5f99-3085-d984e07eb50e}"
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.212Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: pipelining replication: peer="{Voter server-ebf30ee7-202e-5f99-3085-d984e07eb50e ebf30ee7-202e-5f99-3085-d984e07eb50e}"
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.212Z [WARN]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: rejecting vote request since we have a leader: from=a374c972-55fc-f15c-3680-3b7df70966f9 leader=230b57ef-cd8a-8661-7c26-df088e513909
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.212Z [WARN]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: rejecting vote request since we have a leader: from=a374c972-55fc-f15c-3680-3b7df70966f9 leader=230b57ef-cd8a-8661-7c26-df088e513909
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.213Z [WARN]  server-230b57ef-cd8a-8661-7c26-df088e513909: rejecting vote request since we have a leader: from=a374c972-55fc-f15c-3680-3b7df70966f9 leader=230b57ef-cd8a-8661-7c26-df088e513909
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.213Z [WARN]  server-230b57ef-cd8a-8661-7c26-df088e513909: rejecting vote request since we have a leader: from=a374c972-55fc-f15c-3680-3b7df70966f9 leader=230b57ef-cd8a-8661-7c26-df088e513909
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.318Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:01.213557868 +0000 UTC m=+9.365352050 (2 nodes), 64.415558ms from start of poll, 70.419254ms from cluster start. Timeout at 2024-01-08 10:38:01.318673161 +0000 UTC m=+9.470467013, 105.114963ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:01.318Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:01.213557868 +0000 UTC m=+9.365352050 (2 nodes), 64.415558ms from start of poll, 70.419254ms from cluster start. Timeout at 2024-01-08 10:38:01.318673161 +0000 UTC m=+9.470467013, 105.114963ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.318Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:01.318Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.425Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.318907449 +0000 UTC m=+9.470701299 (1 nodes), 1.406µs from start of poll, 175.768503ms from cluster start. Timeout at 2024-01-08 10:38:01.425795872 +0000 UTC m=+9.577589724, 106.888425ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:01.425Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.318907449 +0000 UTC m=+9.470701299 (1 nodes), 1.406µs from start of poll, 175.768503ms from cluster start. Timeout at 2024-01-08 10:38:01.425795872 +0000 UTC m=+9.577589724, 106.888425ms after stability
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.440Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: updating configuration: command=RemoveServer server-id=server-230b57ef-cd8a-8661-7c26-df088e513909 server-addr= servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.440Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: updating configuration: command=RemoveServer server-id=server-230b57ef-cd8a-8661-7c26-df088e513909 server-addr= servers="[{Suffrage:Voter ID:server-ebf30ee7-202e-5f99-3085-d984e07eb50e Address:ebf30ee7-202e-5f99-3085-d984e07eb50e} {Suffrage:Voter ID:server-a374c972-55fc-f15c-3680-3b7df70966f9 Address:a374c972-55fc-f15c-3680-3b7df70966f9}]"
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: removed ourself, transitioning to follower
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: removed ourself, transitioning to follower
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering follower state: follower="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Follower]" leader=
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: entering follower state: follower="Node at 230b57ef-cd8a-8661-7c26-df088e513909 [Follower]" leader=
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: aborting pipeline replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: aborting pipeline replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: aborting pipeline replication: peer="{Voter server-ebf30ee7-202e-5f99-3085-d984e07eb50e ebf30ee7-202e-5f99-3085-d984e07eb50e}"
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.443Z [INFO]  server-230b57ef-cd8a-8661-7c26-df088e513909: aborting pipeline replication: peer="{Voter server-ebf30ee7-202e-5f99-3085-d984e07eb50e ebf30ee7-202e-5f99-3085-d984e07eb50e}"
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.519Z [WARN]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: heartbeat timeout reached, starting election: last-leader=230b57ef-cd8a-8661-7c26-df088e513909
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.519Z [WARN]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: heartbeat timeout reached, starting election: last-leader=230b57ef-cd8a-8661-7c26-df088e513909
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.519Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering candidate state: node="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Candidate]" term=3
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.519Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering candidate state: node="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Candidate]" term=3
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.519Z [WARN]  server-a374c972-55fc-f15c-3680-3b7df70966f9: rejecting vote request since we have a leader: from=ebf30ee7-202e-5f99-3085-d984e07eb50e leader=230b57ef-cd8a-8661-7c26-df088e513909
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.519Z [WARN]  server-a374c972-55fc-f15c-3680-3b7df70966f9: rejecting vote request since we have a leader: from=ebf30ee7-202e-5f99-3085-d984e07eb50e leader=230b57ef-cd8a-8661-7c26-df088e513909
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.526Z [WARN]  server-a374c972-55fc-f15c-3680-3b7df70966f9: heartbeat timeout reached, starting election: last-leader=230b57ef-cd8a-8661-7c26-df088e513909
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.526Z [WARN]  server-a374c972-55fc-f15c-3680-3b7df70966f9: heartbeat timeout reached, starting election: last-leader=230b57ef-cd8a-8661-7c26-df088e513909
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.527Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering candidate state: node="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Candidate]" term=3
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.527Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering candidate state: node="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Candidate]" term=3
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.527Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: duplicate requestVote for same term: term=3
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.527Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: duplicate requestVote for same term: term=3
testLoggerAdapter verbose: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.544Z [WARN]  server-230b57ef-cd8a-8661-7c26-df088e513909: not part of stable configuration, aborting election
    testing.go:149: server-230b57ef-cd8a-8661-7c26-df088e513909: 2024-01-08T10:38:01.544Z [WARN]  server-230b57ef-cd8a-8661-7c26-df088e513909: not part of stable configuration, aborting election
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.551Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:01.551Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.574Z [WARN]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: Election timeout reached, restarting election
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.574Z [WARN]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: Election timeout reached, restarting election
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.574Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering candidate state: node="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Candidate]" term=4
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.574Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering candidate state: node="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Candidate]" term=4
testLoggerAdapter verbose: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.574Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering follower state: follower="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Follower]" leader=
    testing.go:149: server-a374c972-55fc-f15c-3680-3b7df70966f9: 2024-01-08T10:38:01.574Z [INFO]  server-a374c972-55fc-f15c-3680-3b7df70966f9: entering follower state: follower="Node at a374c972-55fc-f15c-3680-3b7df70966f9 [Follower]" leader=
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: election won: tally=2
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: election won: tally=2
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering leader state: leader="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Leader]"
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: entering leader state: leader="Node at ebf30ee7-202e-5f99-3085-d984e07eb50e [Leader]"
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: added peer, starting replication: peer=server-a374c972-55fc-f15c-3680-3b7df70966f9
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: added peer, starting replication: peer=server-a374c972-55fc-f15c-3680-3b7df70966f9
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: pipelining replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.575Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: pipelining replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.682Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.576079745 +0000 UTC m=+9.727873882 (1 nodes), 24.669794ms from start of poll, 432.941086ms from cluster start. Timeout at 2024-01-08 10:38:01.681882083 +0000 UTC m=+9.833676222, 105.80234ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:01.682Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.576079745 +0000 UTC m=+9.727873882 (1 nodes), 24.669794ms from start of poll, 432.941086ms from cluster start. Timeout at 2024-01-08 10:38:01.681882083 +0000 UTC m=+9.833676222, 105.80234ms after stability
testLoggerAdapter verbose: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.790Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: aborting pipeline replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
    testing.go:149: server-ebf30ee7-202e-5f99-3085-d984e07eb50e: 2024-01-08T10:38:01.790Z [INFO]  server-ebf30ee7-202e-5f99-3085-d984e07eb50e: aborting pipeline replication: peer="{Voter server-a374c972-55fc-f15c-3680-3b7df70966f9 a374c972-55fc-f15c-3680-3b7df70966f9}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.794Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.794Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
testLoggerAdapter verbose: server-2690d475-d2db-301d-8d3e-5ec651a9bf47: 2024-01-08T10:38:01.795Z [INFO]  server-2690d475-d2db-301d-8d3e-5ec651a9bf47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
    testing.go:149: server-2690d475-d2db-301d-8d3e-5ec651a9bf47: 2024-01-08T10:38:01.795Z [INFO]  server-2690d475-d2db-301d-8d3e-5ec651a9bf47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
testLoggerAdapter verbose: server-21169459-7460-dcbb-5b04-fb17e5a32d36: 2024-01-08T10:38:01.796Z [INFO]  server-21169459-7460-dcbb-5b04-fb17e5a32d36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
    testing.go:149: server-21169459-7460-dcbb-5b04-fb17e5a32d36: 2024-01-08T10:38:01.796Z [INFO]  server-21169459-7460-dcbb-5b04-fb17e5a32d36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
testLoggerAdapter verbose: server-a4489edd-597d-3127-6e12-5f29081acf2f: 2024-01-08T10:38:01.797Z [INFO]  server-a4489edd-597d-3127-6e12-5f29081acf2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
    testing.go:149: server-a4489edd-597d-3127-6e12-5f29081acf2f: 2024-01-08T10:38:01.797Z [INFO]  server-a4489edd-597d-3127-6e12-5f29081acf2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36} {Suffrage:Voter ID:server-a4489edd-597d-3127-6e12-5f29081acf2f Address:a4489edd-597d-3127-6e12-5f29081acf2f}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.797Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:01.797Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-21169459-7460-dcbb-5b04-fb17e5a32d36: 2024-01-08T10:38:01.798Z [INFO]  server-21169459-7460-dcbb-5b04-fb17e5a32d36: entering follower state: follower="Node at 21169459-7460-dcbb-5b04-fb17e5a32d36 [Follower]" leader=
    testing.go:149: server-21169459-7460-dcbb-5b04-fb17e5a32d36: 2024-01-08T10:38:01.798Z [INFO]  server-21169459-7460-dcbb-5b04-fb17e5a32d36: entering follower state: follower="Node at 21169459-7460-dcbb-5b04-fb17e5a32d36 [Follower]" leader=
testLoggerAdapter verbose: server-a4489edd-597d-3127-6e12-5f29081acf2f: 2024-01-08T10:38:01.798Z [INFO]  server-a4489edd-597d-3127-6e12-5f29081acf2f: entering follower state: follower="Node at a4489edd-597d-3127-6e12-5f29081acf2f [Follower]" leader=
testLoggerAdapter verbose: server-2690d475-d2db-301d-8d3e-5ec651a9bf47: 2024-01-08T10:38:01.798Z [INFO]  server-2690d475-d2db-301d-8d3e-5ec651a9bf47: entering follower state: follower="Node at 2690d475-d2db-301d-8d3e-5ec651a9bf47 [Follower]" leader=
    testing.go:149: server-a4489edd-597d-3127-6e12-5f29081acf2f: 2024-01-08T10:38:01.798Z [INFO]  server-a4489edd-597d-3127-6e12-5f29081acf2f: entering follower state: follower="Node at a4489edd-597d-3127-6e12-5f29081acf2f [Follower]" leader=
    testing.go:149: server-2690d475-d2db-301d-8d3e-5ec651a9bf47: 2024-01-08T10:38:01.798Z [INFO]  server-2690d475-d2db-301d-8d3e-5ec651a9bf47: entering follower state: follower="Node at 2690d475-d2db-301d-8d3e-5ec651a9bf47 [Follower]" leader=
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.805Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering follower state: follower="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Follower]" leader=
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.805Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering follower state: follower="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Follower]" leader=
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.862Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.862Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.862Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering candidate state: node="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Candidate]" term=2
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.862Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering candidate state: node="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Candidate]" term=2
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: election won: tally=3
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: election won: tally=3
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering leader state: leader="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Leader]"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering leader state: leader="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Leader]"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: added peer, starting replication: peer=server-2690d475-d2db-301d-8d3e-5ec651a9bf47
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: added peer, starting replication: peer=server-2690d475-d2db-301d-8d3e-5ec651a9bf47
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: added peer, starting replication: peer=server-21169459-7460-dcbb-5b04-fb17e5a32d36
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: added peer, starting replication: peer=server-21169459-7460-dcbb-5b04-fb17e5a32d36
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: added peer, starting replication: peer=server-a4489edd-597d-3127-6e12-5f29081acf2f
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: added peer, starting replication: peer=server-a4489edd-597d-3127-6e12-5f29081acf2f
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: pipelining replication: peer="{Voter server-a4489edd-597d-3127-6e12-5f29081acf2f a4489edd-597d-3127-6e12-5f29081acf2f}"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.863Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: pipelining replication: peer="{Voter server-a4489edd-597d-3127-6e12-5f29081acf2f a4489edd-597d-3127-6e12-5f29081acf2f}"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.864Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: pipelining replication: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.864Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: pipelining replication: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.864Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: pipelining replication: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:01.864Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: pipelining replication: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:01.970Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.864363077 +0000 UTC m=+10.016157230 (1 nodes), 66.689184ms from start of poll, 70.279584ms from cluster start. Timeout at 2024-01-08 10:38:01.969948491 +0000 UTC m=+10.121742611, 105.585381ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:01.970Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:01.864363077 +0000 UTC m=+10.016157230 (1 nodes), 66.689184ms from start of poll, 70.279584ms from cluster start. Timeout at 2024-01-08 10:38:01.969948491 +0000 UTC m=+10.121742611, 105.585381ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.077Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:02.077Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.190Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:02.077973131 +0000 UTC m=+10.229766981 (3 nodes), 1.938µs from start of poll, 283.889335ms from cluster start. Timeout at 2024-01-08 10:38:02.189965125 +0000 UTC m=+10.341759306, 111.992325ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:02.190Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:02.077973131 +0000 UTC m=+10.229766981 (3 nodes), 1.938µs from start of poll, 283.889335ms from cluster start. Timeout at 2024-01-08 10:38:02.189965125 +0000 UTC m=+10.341759306, 111.992325ms after stability
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.190Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: updating configuration: command=RemoveServer server-id=server-a4489edd-597d-3127-6e12-5f29081acf2f server-addr= servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36}]"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.190Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: updating configuration: command=RemoveServer server-id=server-a4489edd-597d-3127-6e12-5f29081acf2f server-addr= servers="[{Suffrage:Voter ID:server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f Address:7f43a2d6-3266-2da2-1833-8f96c5e76f9f} {Suffrage:Voter ID:server-2690d475-d2db-301d-8d3e-5ec651a9bf47 Address:2690d475-d2db-301d-8d3e-5ec651a9bf47} {Suffrage:Voter ID:server-21169459-7460-dcbb-5b04-fb17e5a32d36 Address:21169459-7460-dcbb-5b04-fb17e5a32d36}]"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: removed peer, stopping replication: peer=server-a4489edd-597d-3127-6e12-5f29081acf2f last-index=3
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: removed peer, stopping replication: peer=server-a4489edd-597d-3127-6e12-5f29081acf2f last-index=3
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to pipeline appendEntries: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="append pipeline closed"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to pipeline appendEntries: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="append pipeline closed"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: aborting pipeline replication: peer="{Voter server-a4489edd-597d-3127-6e12-5f29081acf2f a4489edd-597d-3127-6e12-5f29081acf2f}"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: aborting pipeline replication: peer="{Voter server-a4489edd-597d-3127-6e12-5f29081acf2f a4489edd-597d-3127-6e12-5f29081acf2f}"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: aborting pipeline replication: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: aborting pipeline replication: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to pipeline appendEntries: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="append pipeline closed"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to pipeline appendEntries: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="append pipeline closed"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: aborting pipeline replication: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: aborting pipeline replication: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=2690d475-d2db-301d-8d3e-5ec651a9bf47 error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.191Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=2690d475-d2db-301d-8d3e-5ec651a9bf47 error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.197Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.197Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.198Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=21169459-7460-dcbb-5b04-fb17e5a32d36 error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.198Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=21169459-7460-dcbb-5b04-fb17e5a32d36 error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.201Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.201Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.214Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=2690d475-d2db-301d-8d3e-5ec651a9bf47 error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.214Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=2690d475-d2db-301d-8d3e-5ec651a9bf47 error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.217Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.217Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.221Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.221Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.221Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=21169459-7460-dcbb-5b04-fb17e5a32d36 error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.221Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=21169459-7460-dcbb-5b04-fb17e5a32d36 error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.234Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=2690d475-d2db-301d-8d3e-5ec651a9bf47 error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.234Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to heartbeat to: peer=2690d475-d2db-301d-8d3e-5ec651a9bf47 error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.237Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.237Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-2690d475-d2db-301d-8d3e-5ec651a9bf47 2690d475-d2db-301d-8d3e-5ec651a9bf47}" error="failed to connect to peer: 2690d475-d2db-301d-8d3e-5ec651a9bf47"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.241Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.241Z [ERROR] server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to appendEntries to: peer="{Voter server-21169459-7460-dcbb-5b04-fb17e5a32d36 21169459-7460-dcbb-5b04-fb17e5a32d36}" error="failed to connect to peer: 21169459-7460-dcbb-5b04-fb17e5a32d36"
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to contact: server-id=server-2690d475-d2db-301d-8d3e-5ec651a9bf47 time=52.196544ms
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to contact: server-id=server-2690d475-d2db-301d-8d3e-5ec651a9bf47 time=52.196544ms
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to contact: server-id=server-21169459-7460-dcbb-5b04-fb17e5a32d36 time=52.212357ms
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to contact: server-id=server-21169459-7460-dcbb-5b04-fb17e5a32d36 time=52.212357ms
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to contact quorum of nodes, stepping down
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [WARN]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: failed to contact quorum of nodes, stepping down
testLoggerAdapter verbose: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering follower state: follower="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Follower]" leader=
    testing.go:149: server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: 2024-01-08T10:38:02.242Z [INFO]  server-7f43a2d6-3266-2da2-1833-8f96c5e76f9f: entering follower state: follower="Node at 7f43a2d6-3266-2da2-1833-8f96c5e76f9f [Follower]" leader=
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s)
=== RUN   TestRaft_AddKnownPeer
testLoggerAdapter verbose: server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: 2024-01-08T10:38:02.248Z [INFO]  server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
    testing.go:149: server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: 2024-01-08T10:38:02.248Z [INFO]  server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
testLoggerAdapter verbose: server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: 2024-01-08T10:38:02.249Z [INFO]  server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
    testing.go:149: server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: 2024-01-08T10:38:02.249Z [INFO]  server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
testLoggerAdapter verbose: server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: 2024-01-08T10:38:02.249Z [INFO]  server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: entering follower state: follower="Node at e32fa2a5-850a-2cc6-1983-26fdde8b73f9 [Follower]" leader=
    testing.go:149: server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: 2024-01-08T10:38:02.249Z [INFO]  server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9: entering follower state: follower="Node at e32fa2a5-850a-2cc6-1983-26fdde8b73f9 [Follower]" leader=
testLoggerAdapter verbose: server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: 2024-01-08T10:38:02.250Z [INFO]  server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: entering follower state: follower="Node at 80e15918-92f5-b6dd-cfcc-6851d04b47e9 [Follower]" leader=
    testing.go:149: server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: 2024-01-08T10:38:02.250Z [INFO]  server-80e15918-92f5-b6dd-cfcc-6851d04b47e9: entering follower state: follower="Node at 80e15918-92f5-b6dd-cfcc-6851d04b47e9 [Follower]" leader=
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.258Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.258Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.259Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:02.259Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.259Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: entering follower state: follower="Node at 4422c506-4a59-7a56-b46d-ac54f91a712f [Follower]" leader=
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.259Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: entering follower state: follower="Node at 4422c506-4a59-7a56-b46d-ac54f91a712f [Follower]" leader=
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.310Z [WARN]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.310Z [WARN]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.310Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: entering candidate state: node="Node at 4422c506-4a59-7a56-b46d-ac54f91a712f [Candidate]" term=2
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.310Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: entering candidate state: node="Node at 4422c506-4a59-7a56-b46d-ac54f91a712f [Candidate]" term=2
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.311Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: election won: tally=2
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.311Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: election won: tally=2
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.311Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: entering leader state: leader="Node at 4422c506-4a59-7a56-b46d-ac54f91a712f [Leader]"
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.311Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: entering leader state: leader="Node at 4422c506-4a59-7a56-b46d-ac54f91a712f [Leader]"
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.311Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: added peer, starting replication: peer=server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.311Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: added peer, starting replication: peer=server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.312Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: added peer, starting replication: peer=server-80e15918-92f5-b6dd-cfcc-6851d04b47e9
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.312Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: added peer, starting replication: peer=server-80e15918-92f5-b6dd-cfcc-6851d04b47e9
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.312Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: pipelining replication: peer="{Voter server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 80e15918-92f5-b6dd-cfcc-6851d04b47e9}"
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.312Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: pipelining replication: peer="{Voter server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 80e15918-92f5-b6dd-cfcc-6851d04b47e9}"
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.313Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: pipelining replication: peer="{Voter server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 e32fa2a5-850a-2cc6-1983-26fdde8b73f9}"
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.313Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: pipelining replication: peer="{Voter server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 e32fa2a5-850a-2cc6-1983-26fdde8b73f9}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.422Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:02.313430057 +0000 UTC m=+10.465224214 (1 nodes), 54.188788ms from start of poll, 65.550585ms from cluster start. Timeout at 2024-01-08 10:38:02.421926804 +0000 UTC m=+10.573720950, 108.496736ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:02.422Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:02.313430057 +0000 UTC m=+10.465224214 (1 nodes), 54.188788ms from start of poll, 65.550585ms from cluster start. Timeout at 2024-01-08 10:38:02.421926804 +0000 UTC m=+10.573720950, 108.496736ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.422Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:02.422Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.530Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:02.42294832 +0000 UTC m=+10.574742444 (2 nodes), 4.904µs from start of poll, 175.068815ms from cluster start. Timeout at 2024-01-08 10:38:02.530213593 +0000 UTC m=+10.682007720, 107.265276ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:02.530Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:02.42294832 +0000 UTC m=+10.574742444 (2 nodes), 4.904µs from start of poll, 175.068815ms from cluster start. Timeout at 2024-01-08 10:38:02.530213593 +0000 UTC m=+10.682007720, 107.265276ms after stability
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.530Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: updating configuration: command=AddStaging server-id=server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 server-addr=e32fa2a5-850a-2cc6-1983-26fdde8b73f9 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.530Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: updating configuration: command=AddStaging server-id=server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 server-addr=e32fa2a5-850a-2cc6-1983-26fdde8b73f9 servers="[{Suffrage:Voter ID:server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 Address:e32fa2a5-850a-2cc6-1983-26fdde8b73f9} {Suffrage:Voter ID:server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 Address:80e15918-92f5-b6dd-cfcc-6851d04b47e9} {Suffrage:Voter ID:server-4422c506-4a59-7a56-b46d-ac54f91a712f Address:4422c506-4a59-7a56-b46d-ac54f91a712f}]"
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.532Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: aborting pipeline replication: peer="{Voter server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 80e15918-92f5-b6dd-cfcc-6851d04b47e9}"
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.532Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: aborting pipeline replication: peer="{Voter server-80e15918-92f5-b6dd-cfcc-6851d04b47e9 80e15918-92f5-b6dd-cfcc-6851d04b47e9}"
testLoggerAdapter verbose: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.532Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: aborting pipeline replication: peer="{Voter server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 e32fa2a5-850a-2cc6-1983-26fdde8b73f9}"
    testing.go:149: server-4422c506-4a59-7a56-b46d-ac54f91a712f: 2024-01-08T10:38:02.532Z [INFO]  server-4422c506-4a59-7a56-b46d-ac54f91a712f: aborting pipeline replication: peer="{Voter server-e32fa2a5-850a-2cc6-1983-26fdde8b73f9 e32fa2a5-850a-2cc6-1983-26fdde8b73f9}"
--- PASS: TestRaft_AddKnownPeer (0.29s)
=== RUN   TestRaft_RemoveUnknownPeer
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.537Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.537Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.538Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: entering follower state: follower="Node at 5788c91d-83af-7a39-eac1-3822769cb799 [Follower]" leader=
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.538Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: entering follower state: follower="Node at 5788c91d-83af-7a39-eac1-3822769cb799 [Follower]" leader=
testLoggerAdapter verbose: server-9bb34778-c256-a797-dbc1-ced20d69e690: 2024-01-08T10:38:02.538Z [INFO]  server-9bb34778-c256-a797-dbc1-ced20d69e690: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
    testing.go:149: server-9bb34778-c256-a797-dbc1-ced20d69e690: 2024-01-08T10:38:02.538Z [INFO]  server-9bb34778-c256-a797-dbc1-ced20d69e690: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
testLoggerAdapter verbose: server-40ce7698-4390-7e03-4047-621fbe0b0290: 2024-01-08T10:38:02.539Z [INFO]  server-40ce7698-4390-7e03-4047-621fbe0b0290: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
    testing.go:149: server-40ce7698-4390-7e03-4047-621fbe0b0290: 2024-01-08T10:38:02.539Z [INFO]  server-40ce7698-4390-7e03-4047-621fbe0b0290: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.539Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:02.539Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-9bb34778-c256-a797-dbc1-ced20d69e690: 2024-01-08T10:38:02.539Z [INFO]  server-9bb34778-c256-a797-dbc1-ced20d69e690: entering follower state: follower="Node at 9bb34778-c256-a797-dbc1-ced20d69e690 [Follower]" leader=
    testing.go:149: server-9bb34778-c256-a797-dbc1-ced20d69e690: 2024-01-08T10:38:02.539Z [INFO]  server-9bb34778-c256-a797-dbc1-ced20d69e690: entering follower state: follower="Node at 9bb34778-c256-a797-dbc1-ced20d69e690 [Follower]" leader=
testLoggerAdapter verbose: server-40ce7698-4390-7e03-4047-621fbe0b0290: 2024-01-08T10:38:02.540Z [INFO]  server-40ce7698-4390-7e03-4047-621fbe0b0290: entering follower state: follower="Node at 40ce7698-4390-7e03-4047-621fbe0b0290 [Follower]" leader=
    testing.go:149: server-40ce7698-4390-7e03-4047-621fbe0b0290: 2024-01-08T10:38:02.540Z [INFO]  server-40ce7698-4390-7e03-4047-621fbe0b0290: entering follower state: follower="Node at 40ce7698-4390-7e03-4047-621fbe0b0290 [Follower]" leader=
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.591Z [WARN]  server-5788c91d-83af-7a39-eac1-3822769cb799: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.591Z [WARN]  server-5788c91d-83af-7a39-eac1-3822769cb799: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: entering candidate state: node="Node at 5788c91d-83af-7a39-eac1-3822769cb799 [Candidate]" term=2
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: entering candidate state: node="Node at 5788c91d-83af-7a39-eac1-3822769cb799 [Candidate]" term=2
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: election won: tally=2
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: election won: tally=2
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: entering leader state: leader="Node at 5788c91d-83af-7a39-eac1-3822769cb799 [Leader]"
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: entering leader state: leader="Node at 5788c91d-83af-7a39-eac1-3822769cb799 [Leader]"
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: added peer, starting replication: peer=server-9bb34778-c256-a797-dbc1-ced20d69e690
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.592Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: added peer, starting replication: peer=server-9bb34778-c256-a797-dbc1-ced20d69e690
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.593Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: added peer, starting replication: peer=server-40ce7698-4390-7e03-4047-621fbe0b0290
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.593Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: added peer, starting replication: peer=server-40ce7698-4390-7e03-4047-621fbe0b0290
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.593Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: pipelining replication: peer="{Voter server-40ce7698-4390-7e03-4047-621fbe0b0290 40ce7698-4390-7e03-4047-621fbe0b0290}"
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.593Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: pipelining replication: peer="{Voter server-40ce7698-4390-7e03-4047-621fbe0b0290 40ce7698-4390-7e03-4047-621fbe0b0290}"
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.594Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: pipelining replication: peer="{Voter server-9bb34778-c256-a797-dbc1-ced20d69e690 9bb34778-c256-a797-dbc1-ced20d69e690}"
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.594Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: pipelining replication: peer="{Voter server-9bb34778-c256-a797-dbc1-ced20d69e690 9bb34778-c256-a797-dbc1-ced20d69e690}"
testLoggerAdapter verbose: server-9bb34778-c256-a797-dbc1-ced20d69e690: 2024-01-08T10:38:02.594Z [WARN]  server-9bb34778-c256-a797-dbc1-ced20d69e690: rejecting vote request since our last term is greater: candidate=5788c91d-83af-7a39-eac1-3822769cb799 last-term=2 last-candidate-term=1
    testing.go:149: server-9bb34778-c256-a797-dbc1-ced20d69e690: 2024-01-08T10:38:02.594Z [WARN]  server-9bb34778-c256-a797-dbc1-ced20d69e690: rejecting vote request since our last term is greater: candidate=5788c91d-83af-7a39-eac1-3822769cb799 last-term=2 last-candidate-term=1
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.702Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:02.594352253 +0000 UTC m=+10.746146104 (1 nodes), 54.647871ms from start of poll, 57.546462ms from cluster start. Timeout at 2024-01-08 10:38:02.702537152 +0000 UTC m=+10.854331270, 108.185166ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:02.702Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:02.594352253 +0000 UTC m=+10.746146104 (1 nodes), 54.647871ms from start of poll, 57.546462ms from cluster start. Timeout at 2024-01-08 10:38:02.702537152 +0000 UTC m=+10.854331270, 108.185166ms after stability
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.703Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: updating configuration: command=RemoveServer server-id=87ec7aef-9de0-91c3-ed6d-282494dd1528 server-addr= servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.703Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: updating configuration: command=RemoveServer server-id=87ec7aef-9de0-91c3-ed6d-282494dd1528 server-addr= servers="[{Suffrage:Voter ID:server-5788c91d-83af-7a39-eac1-3822769cb799 Address:5788c91d-83af-7a39-eac1-3822769cb799} {Suffrage:Voter ID:server-9bb34778-c256-a797-dbc1-ced20d69e690 Address:9bb34778-c256-a797-dbc1-ced20d69e690} {Suffrage:Voter ID:server-40ce7698-4390-7e03-4047-621fbe0b0290 Address:40ce7698-4390-7e03-4047-621fbe0b0290}]"
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.721Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: aborting pipeline replication: peer="{Voter server-40ce7698-4390-7e03-4047-621fbe0b0290 40ce7698-4390-7e03-4047-621fbe0b0290}"
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.721Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: aborting pipeline replication: peer="{Voter server-40ce7698-4390-7e03-4047-621fbe0b0290 40ce7698-4390-7e03-4047-621fbe0b0290}"
testLoggerAdapter verbose: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.721Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: aborting pipeline replication: peer="{Voter server-9bb34778-c256-a797-dbc1-ced20d69e690 9bb34778-c256-a797-dbc1-ced20d69e690}"
    testing.go:149: server-5788c91d-83af-7a39-eac1-3822769cb799: 2024-01-08T10:38:02.721Z [INFO]  server-5788c91d-83af-7a39-eac1-3822769cb799: aborting pipeline replication: peer="{Voter server-9bb34778-c256-a797-dbc1-ced20d69e690 9bb34778-c256-a797-dbc1-ced20d69e690}"
--- PASS: TestRaft_RemoveUnknownPeer (0.19s)
=== RUN   TestRaft_SnapshotRestore
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.734Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ba3a5a6-8dde-359e-af29-debeaec619c0 Address:6ba3a5a6-8dde-359e-af29-debeaec619c0}]"
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.734Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ba3a5a6-8dde-359e-af29-debeaec619c0 Address:6ba3a5a6-8dde-359e-af29-debeaec619c0}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.734Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:02.734Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.734Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: entering follower state: follower="Node at 6ba3a5a6-8dde-359e-af29-debeaec619c0 [Follower]" leader=
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.734Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: entering follower state: follower="Node at 6ba3a5a6-8dde-359e-af29-debeaec619c0 [Follower]" leader=
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.810Z [WARN]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.810Z [WARN]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.810Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: entering candidate state: node="Node at 6ba3a5a6-8dde-359e-af29-debeaec619c0 [Candidate]" term=2
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.810Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: entering candidate state: node="Node at 6ba3a5a6-8dde-359e-af29-debeaec619c0 [Candidate]" term=2
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.811Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: election won: tally=1
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.811Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: election won: tally=1
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.811Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: entering leader state: leader="Node at 6ba3a5a6-8dde-359e-af29-debeaec619c0 [Leader]"
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.811Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: entering leader state: leader="Node at 6ba3a5a6-8dde-359e-af29-debeaec619c0 [Leader]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.918Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:02.81167285 +0000 UTC m=+10.963466985 (1 nodes), 77.103265ms from start of poll, 86.457391ms from cluster start. Timeout at 2024-01-08 10:38:02.917742145 +0000 UTC m=+11.069536291, 106.069306ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:02.918Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:02.81167285 +0000 UTC m=+10.963466985 (1 nodes), 77.103265ms from start of poll, 86.457391ms from cluster start. Timeout at 2024-01-08 10:38:02.917742145 +0000 UTC m=+11.069536291, 106.069306ms after stability
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.928Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: starting snapshot up to: index=102
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.928Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: starting snapshot up to: index=102
    testing.go:153: 2024-01-08T10:38:02.929Z [INFO]  creating new snapshot: path=/tmp/raft1501555841/snapshots/2-102-1704710282929.tmp
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.935Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: compacting logs: from=1 to=92
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.935Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: compacting logs: from=1 to=92
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.935Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: snapshot complete up to: index=102
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.935Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: snapshot complete up to: index=102
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.937Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: restored from snapshot: id=2-102-1704710282929
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.937Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: restored from snapshot: id=2-102-1704710282929
testLoggerAdapter verbose: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.937Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ba3a5a6-8dde-359e-af29-debeaec619c0 Address:6ba3a5a6-8dde-359e-af29-debeaec619c0}]"
    testing.go:149: server-6ba3a5a6-8dde-359e-af29-debeaec619c0: 2024-01-08T10:38:02.937Z [INFO]  server-6ba3a5a6-8dde-359e-af29-debeaec619c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ba3a5a6-8dde-359e-af29-debeaec619c0 Address:6ba3a5a6-8dde-359e-af29-debeaec619c0}]"
--- PASS: TestRaft_SnapshotRestore (0.23s)
=== RUN   TestRaft_NoRestoreOnStart
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:02.953Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3510b723-9adc-80e1-6b6a-c301169d94db Address:3510b723-9adc-80e1-6b6a-c301169d94db}]"
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:02.953Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3510b723-9adc-80e1-6b6a-c301169d94db Address:3510b723-9adc-80e1-6b6a-c301169d94db}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:02.953Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:02.953Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:02.953Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering follower state: follower="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Follower]" leader=
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:02.953Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering follower state: follower="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Follower]" leader=
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [WARN]  server-3510b723-9adc-80e1-6b6a-c301169d94db: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [WARN]  server-3510b723-9adc-80e1-6b6a-c301169d94db: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering candidate state: node="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Candidate]" term=2
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering candidate state: node="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Candidate]" term=2
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: election won: tally=1
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: election won: tally=1
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering leader state: leader="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Leader]"
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.030Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering leader state: leader="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Leader]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.138Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.031065279 +0000 UTC m=+11.182859417 (1 nodes), 77.562918ms from start of poll, 78.327548ms from cluster start. Timeout at 2024-01-08 10:38:03.137944111 +0000 UTC m=+11.289738279, 106.878862ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:03.138Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.031065279 +0000 UTC m=+11.182859417 (1 nodes), 77.562918ms from start of poll, 78.327548ms from cluster start. Timeout at 2024-01-08 10:38:03.137944111 +0000 UTC m=+11.289738279, 106.878862ms after stability
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.144Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: starting snapshot up to: index=102
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.144Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: starting snapshot up to: index=102
    testing.go:153: 2024-01-08T10:38:03.144Z [INFO]  creating new snapshot: path=/tmp/raft3618615501/snapshots/2-102-1704710283144.tmp
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.148Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: compacting logs: from=1 to=92
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.148Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: compacting logs: from=1 to=92
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.148Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: snapshot complete up to: index=102
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.148Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: snapshot complete up to: index=102
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.149Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3510b723-9adc-80e1-6b6a-c301169d94db Address:3510b723-9adc-80e1-6b6a-c301169d94db}]"
    testing.go:149: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.149Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3510b723-9adc-80e1-6b6a-c301169d94db Address:3510b723-9adc-80e1-6b6a-c301169d94db}]"
--- PASS: TestRaft_NoRestoreOnStart (0.20s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1026: Disabled unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
    raft_test.go:1132: Disabled unreliable test
--- SKIP: TestRaft_AutoSnapshot (0.00s)
=== RUN   TestRaft_UserSnapshot
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.152Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-176652f0-1cee-e0cf-daf0-9375065c0d28 Address:176652f0-1cee-e0cf-daf0-9375065c0d28}]"
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.152Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-176652f0-1cee-e0cf-daf0-9375065c0d28 Address:176652f0-1cee-e0cf-daf0-9375065c0d28}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.152Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:03.152Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.152Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering follower state: follower="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Follower]" leader=
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.153Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: entering follower state: follower="Node at 176652f0-1cee-e0cf-daf0-9375065c0d28 [Follower]" leader=
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.153Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: entering follower state: follower="Node at 176652f0-1cee-e0cf-daf0-9375065c0d28 [Follower]" leader=
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [WARN]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [WARN]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: entering candidate state: node="Node at 176652f0-1cee-e0cf-daf0-9375065c0d28 [Candidate]" term=2
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: entering candidate state: node="Node at 176652f0-1cee-e0cf-daf0-9375065c0d28 [Candidate]" term=2
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: election won: tally=1
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: election won: tally=1
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: entering leader state: leader="Node at 176652f0-1cee-e0cf-daf0-9375065c0d28 [Leader]"
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.234Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: entering leader state: leader="Node at 176652f0-1cee-e0cf-daf0-9375065c0d28 [Leader]"
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.253Z [WARN]  server-3510b723-9adc-80e1-6b6a-c301169d94db: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.254Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering candidate state: node="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Candidate]" term=3
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.254Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: election won: tally=1
testLoggerAdapter verbose: server-3510b723-9adc-80e1-6b6a-c301169d94db: 2024-01-08T10:38:03.254Z [INFO]  server-3510b723-9adc-80e1-6b6a-c301169d94db: entering leader state: leader="Node at 3510b723-9adc-80e1-6b6a-c301169d94db [Leader]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.342Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.234934266 +0000 UTC m=+11.386728432 (1 nodes), 82.111989ms from start of poll, 83.177219ms from cluster start. Timeout at 2024-01-08 10:38:03.341900654 +0000 UTC m=+11.493694786, 106.966354ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:03.342Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.234934266 +0000 UTC m=+11.386728432 (1 nodes), 82.111989ms from start of poll, 83.177219ms from cluster start. Timeout at 2024-01-08 10:38:03.341900654 +0000 UTC m=+11.493694786, 106.966354ms after stability
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.342Z [ERROR] server-176652f0-1cee-e0cf-daf0-9375065c0d28: failed to take snapshot: error="nothing new to snapshot"
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.342Z [ERROR] server-176652f0-1cee-e0cf-daf0-9375065c0d28: failed to take snapshot: error="nothing new to snapshot"
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.343Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: starting snapshot up to: index=12
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.343Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: starting snapshot up to: index=12
    testing.go:153: 2024-01-08T10:38:03.344Z [INFO]  creating new snapshot: path=/tmp/raft2020843945/snapshots/2-12-1704710283344.tmp
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.346Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: compacting logs: from=1 to=2
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.346Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: compacting logs: from=1 to=2
testLoggerAdapter verbose: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.347Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: snapshot complete up to: index=12
    testing.go:149: server-176652f0-1cee-e0cf-daf0-9375065c0d28: 2024-01-08T10:38:03.347Z [INFO]  server-176652f0-1cee-e0cf-daf0-9375065c0d28: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.20s)
=== RUN   TestRaft_UserRestore
testLoggerAdapter verbose: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.366Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 Address:e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09} {Suffrage:Voter ID:server-9785057a-3a69-99de-5328-33b85c34a47b Address:9785057a-3a69-99de-5328-33b85c34a47b} {Suffrage:Voter ID:server-a041f63f-a19b-e910-929e-0eb39f5307ce Address:a041f63f-a19b-e910-929e-0eb39f5307ce}]"
    testing.go:149: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.366Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 Address:e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09} {Suffrage:Voter ID:server-9785057a-3a69-99de-5328-33b85c34a47b Address:9785057a-3a69-99de-5328-33b85c34a47b} {Suffrage:Voter ID:server-a041f63f-a19b-e910-929e-0eb39f5307ce Address:a041f63f-a19b-e910-929e-0eb39f5307ce}]"
testLoggerAdapter verbose: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.367Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 Address:e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09} {Suffrage:Voter ID:server-9785057a-3a69-99de-5328-33b85c34a47b Address:9785057a-3a69-99de-5328-33b85c34a47b} {Suffrage:Voter ID:server-a041f63f-a19b-e910-929e-0eb39f5307ce Address:a041f63f-a19b-e910-929e-0eb39f5307ce}]"
    testing.go:149: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.367Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 Address:e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09} {Suffrage:Voter ID:server-9785057a-3a69-99de-5328-33b85c34a47b Address:9785057a-3a69-99de-5328-33b85c34a47b} {Suffrage:Voter ID:server-a041f63f-a19b-e910-929e-0eb39f5307ce Address:a041f63f-a19b-e910-929e-0eb39f5307ce}]"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.368Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 Address:e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09} {Suffrage:Voter ID:server-9785057a-3a69-99de-5328-33b85c34a47b Address:9785057a-3a69-99de-5328-33b85c34a47b} {Suffrage:Voter ID:server-a041f63f-a19b-e910-929e-0eb39f5307ce Address:a041f63f-a19b-e910-929e-0eb39f5307ce}]"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.368Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 Address:e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09} {Suffrage:Voter ID:server-9785057a-3a69-99de-5328-33b85c34a47b Address:9785057a-3a69-99de-5328-33b85c34a47b} {Suffrage:Voter ID:server-a041f63f-a19b-e910-929e-0eb39f5307ce Address:a041f63f-a19b-e910-929e-0eb39f5307ce}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.368Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:03.368Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.368Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: entering follower state: follower="Node at e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 [Follower]" leader=
    testing.go:149: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.368Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: entering follower state: follower="Node at e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 [Follower]" leader=
testLoggerAdapter verbose: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.368Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: entering follower state: follower="Node at 9785057a-3a69-99de-5328-33b85c34a47b [Follower]" leader=
    testing.go:149: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.368Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: entering follower state: follower="Node at 9785057a-3a69-99de-5328-33b85c34a47b [Follower]" leader=
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.369Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: entering follower state: follower="Node at a041f63f-a19b-e910-929e-0eb39f5307ce [Follower]" leader=
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.369Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: entering follower state: follower="Node at a041f63f-a19b-e910-929e-0eb39f5307ce [Follower]" leader=
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.457Z [WARN]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.457Z [WARN]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.458Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: entering candidate state: node="Node at a041f63f-a19b-e910-929e-0eb39f5307ce [Candidate]" term=2
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.458Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: entering candidate state: node="Node at a041f63f-a19b-e910-929e-0eb39f5307ce [Candidate]" term=2
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.458Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: election won: tally=2
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.458Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: election won: tally=2
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.458Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: entering leader state: leader="Node at a041f63f-a19b-e910-929e-0eb39f5307ce [Leader]"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.458Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: entering leader state: leader="Node at a041f63f-a19b-e910-929e-0eb39f5307ce [Leader]"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.459Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: added peer, starting replication: peer=server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.459Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: added peer, starting replication: peer=server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.459Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: added peer, starting replication: peer=server-9785057a-3a69-99de-5328-33b85c34a47b
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.459Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: added peer, starting replication: peer=server-9785057a-3a69-99de-5328-33b85c34a47b
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.459Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.459Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.460Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.460Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.566Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.460786511 +0000 UTC m=+11.612580641 (1 nodes), 92.261104ms from start of poll, 111.054932ms from cluster start. Timeout at 2024-01-08 10:38:03.566124384 +0000 UTC m=+11.717918508, 105.337867ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:03.566Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.460786511 +0000 UTC m=+11.612580641 (1 nodes), 92.261104ms from start of poll, 111.054932ms from cluster start. Timeout at 2024-01-08 10:38:03.566124384 +0000 UTC m=+11.717918508, 105.337867ms after stability
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.568Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: starting snapshot up to: index=12
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.568Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: starting snapshot up to: index=12
    testing.go:153: 2024-01-08T10:38:03.568Z [INFO]  creating new snapshot: path=/tmp/raft4190350935/snapshots/2-12-1704710283568.tmp
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.575Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: snapshot complete up to: index=12
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.575Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: snapshot complete up to: index=12
    testing.go:153: 2024-01-08T10:38:03.590Z [INFO]  creating new snapshot: path=/tmp/raft4190350935/snapshots/2-23-1704710283590.tmp
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.593Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: copied to local snapshot: bytes=71
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.593Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: restored user snapshot: index=1
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: restored user snapshot: index=1
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.599Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.609Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.609Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.610Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.610Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to get log: index=23 error="log not found"
    testing.go:153: 2024-01-08T10:38:03.612Z [INFO]  creating new snapshot: path=/tmp/raft289762415/snapshots/2-23-1704710283612.tmp
    testing.go:153: 2024-01-08T10:38:03.616Z [INFO]  creating new snapshot: path=/tmp/raft3548154031/snapshots/2-23-1704710283616.tmp
testLoggerAdapter verbose: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.622Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: copied to local snapshot: bytes=71
    testing.go:149: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.622Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.623Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: Installed remote snapshot
    testing.go:149: server-9785057a-3a69-99de-5328-33b85c34a47b: 2024-01-08T10:38:03.623Z [INFO]  server-9785057a-3a69-99de-5328-33b85c34a47b: Installed remote snapshot
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.624Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.624Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
testLoggerAdapter verbose: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.630Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: copied to local snapshot: bytes=71
    testing.go:149: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.630Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.632Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: Installed remote snapshot
    testing.go:149: server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: 2024-01-08T10:38:03.632Z [INFO]  server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09: Installed remote snapshot
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.632Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.632Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: pipelining replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.650Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.650Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-9785057a-3a69-99de-5328-33b85c34a47b 9785057a-3a69-99de-5328-33b85c34a47b}"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.650Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.650Z [INFO]  server-a041f63f-a19b-e910-929e-0eb39f5307ce: aborting pipeline replication: peer="{Voter server-e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09}"
testLoggerAdapter verbose: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.706Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to heartbeat to: peer=e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 error="command timed out"
    testing.go:149: server-a041f63f-a19b-e910-929e-0eb39f5307ce: 2024-01-08T10:38:03.706Z [ERROR] server-a041f63f-a19b-e910-929e-0eb39f5307ce: failed to heartbeat to: peer=e844dedc-9ce1-ad19-3f6d-f4c1e24b8d09 error="command timed out"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.715Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3766eda-dd80-744e-1a9d-714ac039ff1e Address:a3766eda-dd80-744e-1a9d-714ac039ff1e} {Suffrage:Voter ID:server-35f1b44b-1f16-6a77-f106-55f2efd188bd Address:35f1b44b-1f16-6a77-f106-55f2efd188bd} {Suffrage:Voter ID:server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d Address:5c438a9b-9e5b-be07-6446-2a2f9ac6762d}]"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.715Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3766eda-dd80-744e-1a9d-714ac039ff1e Address:a3766eda-dd80-744e-1a9d-714ac039ff1e} {Suffrage:Voter ID:server-35f1b44b-1f16-6a77-f106-55f2efd188bd Address:35f1b44b-1f16-6a77-f106-55f2efd188bd} {Suffrage:Voter ID:server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d Address:5c438a9b-9e5b-be07-6446-2a2f9ac6762d}]"
testLoggerAdapter verbose: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.716Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3766eda-dd80-744e-1a9d-714ac039ff1e Address:a3766eda-dd80-744e-1a9d-714ac039ff1e} {Suffrage:Voter ID:server-35f1b44b-1f16-6a77-f106-55f2efd188bd Address:35f1b44b-1f16-6a77-f106-55f2efd188bd} {Suffrage:Voter ID:server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d Address:5c438a9b-9e5b-be07-6446-2a2f9ac6762d}]"
    testing.go:149: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.716Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3766eda-dd80-744e-1a9d-714ac039ff1e Address:a3766eda-dd80-744e-1a9d-714ac039ff1e} {Suffrage:Voter ID:server-35f1b44b-1f16-6a77-f106-55f2efd188bd Address:35f1b44b-1f16-6a77-f106-55f2efd188bd} {Suffrage:Voter ID:server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d Address:5c438a9b-9e5b-be07-6446-2a2f9ac6762d}]"
testLoggerAdapter verbose: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.716Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3766eda-dd80-744e-1a9d-714ac039ff1e Address:a3766eda-dd80-744e-1a9d-714ac039ff1e} {Suffrage:Voter ID:server-35f1b44b-1f16-6a77-f106-55f2efd188bd Address:35f1b44b-1f16-6a77-f106-55f2efd188bd} {Suffrage:Voter ID:server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d Address:5c438a9b-9e5b-be07-6446-2a2f9ac6762d}]"
    testing.go:149: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.716Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3766eda-dd80-744e-1a9d-714ac039ff1e Address:a3766eda-dd80-744e-1a9d-714ac039ff1e} {Suffrage:Voter ID:server-35f1b44b-1f16-6a77-f106-55f2efd188bd Address:35f1b44b-1f16-6a77-f106-55f2efd188bd} {Suffrage:Voter ID:server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d Address:5c438a9b-9e5b-be07-6446-2a2f9ac6762d}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.717Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:03.717Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.717Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: entering follower state: follower="Node at a3766eda-dd80-744e-1a9d-714ac039ff1e [Follower]" leader=
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.717Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: entering follower state: follower="Node at a3766eda-dd80-744e-1a9d-714ac039ff1e [Follower]" leader=
testLoggerAdapter verbose: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.718Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: entering follower state: follower="Node at 5c438a9b-9e5b-be07-6446-2a2f9ac6762d [Follower]" leader=
    testing.go:149: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.718Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: entering follower state: follower="Node at 5c438a9b-9e5b-be07-6446-2a2f9ac6762d [Follower]" leader=
testLoggerAdapter verbose: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.717Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: entering follower state: follower="Node at 35f1b44b-1f16-6a77-f106-55f2efd188bd [Follower]" leader=
    testing.go:149: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.717Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: entering follower state: follower="Node at 35f1b44b-1f16-6a77-f106-55f2efd188bd [Follower]" leader=
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.774Z [WARN]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.774Z [WARN]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.774Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: entering candidate state: node="Node at a3766eda-dd80-744e-1a9d-714ac039ff1e [Candidate]" term=2
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.774Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: entering candidate state: node="Node at a3766eda-dd80-744e-1a9d-714ac039ff1e [Candidate]" term=2
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.774Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: election won: tally=2
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.774Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: election won: tally=2
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.783Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: entering leader state: leader="Node at a3766eda-dd80-744e-1a9d-714ac039ff1e [Leader]"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.783Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: entering leader state: leader="Node at a3766eda-dd80-744e-1a9d-714ac039ff1e [Leader]"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.783Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: added peer, starting replication: peer=server-35f1b44b-1f16-6a77-f106-55f2efd188bd
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.783Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: added peer, starting replication: peer=server-35f1b44b-1f16-6a77-f106-55f2efd188bd
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.783Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: added peer, starting replication: peer=server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.783Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: added peer, starting replication: peer=server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.792Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.792Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.793Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.793Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.899Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.793374913 +0000 UTC m=+11.945169064 (1 nodes), 76.118884ms from start of poll, 78.507867ms from cluster start. Timeout at 2024-01-08 10:38:03.899073842 +0000 UTC m=+12.050868010, 105.698946ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:03.899Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:03.793374913 +0000 UTC m=+11.945169064 (1 nodes), 76.118884ms from start of poll, 78.507867ms from cluster start. Timeout at 2024-01-08 10:38:03.899073842 +0000 UTC m=+12.050868010, 105.698946ms after stability
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.902Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: starting snapshot up to: index=12
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.902Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: starting snapshot up to: index=12
    testing.go:153: 2024-01-08T10:38:03.902Z [INFO]  creating new snapshot: path=/tmp/raft347889942/snapshots/2-12-1704710283902.tmp
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.905Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: snapshot complete up to: index=12
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.905Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: snapshot complete up to: index=12
    testing.go:153: 2024-01-08T10:38:03.907Z [INFO]  creating new snapshot: path=/tmp/raft347889942/snapshots/2-23-1704710283907.tmp
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.910Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: copied to local snapshot: bytes=71
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.910Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.911Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: restored user snapshot: index=1
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.911Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: restored user snapshot: index=1
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.911Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.911Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.911Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.911Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.912Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.912Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.912Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.912Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.918Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.918Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.920Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.920Z [ERROR] server-a3766eda-dd80-744e-1a9d-714ac039ff1e: failed to get log: index=23 error="log not found"
    testing.go:153: 2024-01-08T10:38:03.922Z [INFO]  creating new snapshot: path=/tmp/raft3698323689/snapshots/2-23-1704710283922.tmp
testLoggerAdapter verbose: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.924Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: copied to local snapshot: bytes=71
    testing.go:149: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.924Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.925Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: Installed remote snapshot
    testing.go:149: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.925Z [INFO]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: Installed remote snapshot
testLoggerAdapter verbose: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.925Z [WARN]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: skipping application of old log: index=22
    testing.go:149: server-35f1b44b-1f16-6a77-f106-55f2efd188bd: 2024-01-08T10:38:03.925Z [WARN]  server-35f1b44b-1f16-6a77-f106-55f2efd188bd: skipping application of old log: index=22
    testing.go:153: 2024-01-08T10:38:03.921Z [INFO]  creating new snapshot: path=/tmp/raft33263998/snapshots/2-23-1704710283921.tmp
testLoggerAdapter verbose: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.927Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: copied to local snapshot: bytes=71
    testing.go:149: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.927Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.928Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: Installed remote snapshot
    testing.go:149: server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: 2024-01-08T10:38:03.928Z [INFO]  server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d: Installed remote snapshot
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.933Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.933Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.934Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.934Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: pipelining replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.966Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.966Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-5c438a9b-9e5b-be07-6446-2a2f9ac6762d 5c438a9b-9e5b-be07-6446-2a2f9ac6762d}"
testLoggerAdapter verbose: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.966Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
    testing.go:149: server-a3766eda-dd80-744e-1a9d-714ac039ff1e: 2024-01-08T10:38:03.966Z [INFO]  server-a3766eda-dd80-744e-1a9d-714ac039ff1e: aborting pipeline replication: peer="{Voter server-35f1b44b-1f16-6a77-f106-55f2efd188bd 35f1b44b-1f16-6a77-f106-55f2efd188bd}"
testLoggerAdapter verbose: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:03.975Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f45de3da-83af-17e7-245b-749c49dd5d03 Address:f45de3da-83af-17e7-245b-749c49dd5d03} {Suffrage:Voter ID:server-94420a5d-3657-3aa5-48df-ad3c6306faf8 Address:94420a5d-3657-3aa5-48df-ad3c6306faf8} {Suffrage:Voter ID:server-098dcf7a-87ca-f538-a019-1aad945c1a01 Address:098dcf7a-87ca-f538-a019-1aad945c1a01}]"
    testing.go:149: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:03.975Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f45de3da-83af-17e7-245b-749c49dd5d03 Address:f45de3da-83af-17e7-245b-749c49dd5d03} {Suffrage:Voter ID:server-94420a5d-3657-3aa5-48df-ad3c6306faf8 Address:94420a5d-3657-3aa5-48df-ad3c6306faf8} {Suffrage:Voter ID:server-098dcf7a-87ca-f538-a019-1aad945c1a01 Address:098dcf7a-87ca-f538-a019-1aad945c1a01}]"
testLoggerAdapter verbose: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:03.976Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: entering follower state: follower="Node at f45de3da-83af-17e7-245b-749c49dd5d03 [Follower]" leader=
    testing.go:149: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:03.976Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: entering follower state: follower="Node at f45de3da-83af-17e7-245b-749c49dd5d03 [Follower]" leader=
testLoggerAdapter verbose: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:03.976Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f45de3da-83af-17e7-245b-749c49dd5d03 Address:f45de3da-83af-17e7-245b-749c49dd5d03} {Suffrage:Voter ID:server-94420a5d-3657-3aa5-48df-ad3c6306faf8 Address:94420a5d-3657-3aa5-48df-ad3c6306faf8} {Suffrage:Voter ID:server-098dcf7a-87ca-f538-a019-1aad945c1a01 Address:098dcf7a-87ca-f538-a019-1aad945c1a01}]"
    testing.go:149: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:03.976Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f45de3da-83af-17e7-245b-749c49dd5d03 Address:f45de3da-83af-17e7-245b-749c49dd5d03} {Suffrage:Voter ID:server-94420a5d-3657-3aa5-48df-ad3c6306faf8 Address:94420a5d-3657-3aa5-48df-ad3c6306faf8} {Suffrage:Voter ID:server-098dcf7a-87ca-f538-a019-1aad945c1a01 Address:098dcf7a-87ca-f538-a019-1aad945c1a01}]"
testLoggerAdapter verbose: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:03.976Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: entering follower state: follower="Node at 94420a5d-3657-3aa5-48df-ad3c6306faf8 [Follower]" leader=
    testing.go:149: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:03.976Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: entering follower state: follower="Node at 94420a5d-3657-3aa5-48df-ad3c6306faf8 [Follower]" leader=
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:03.976Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f45de3da-83af-17e7-245b-749c49dd5d03 Address:f45de3da-83af-17e7-245b-749c49dd5d03} {Suffrage:Voter ID:server-94420a5d-3657-3aa5-48df-ad3c6306faf8 Address:94420a5d-3657-3aa5-48df-ad3c6306faf8} {Suffrage:Voter ID:server-098dcf7a-87ca-f538-a019-1aad945c1a01 Address:098dcf7a-87ca-f538-a019-1aad945c1a01}]"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:03.976Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f45de3da-83af-17e7-245b-749c49dd5d03 Address:f45de3da-83af-17e7-245b-749c49dd5d03} {Suffrage:Voter ID:server-94420a5d-3657-3aa5-48df-ad3c6306faf8 Address:94420a5d-3657-3aa5-48df-ad3c6306faf8} {Suffrage:Voter ID:server-098dcf7a-87ca-f538-a019-1aad945c1a01 Address:098dcf7a-87ca-f538-a019-1aad945c1a01}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:03.977Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:03.977Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:03.977Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: entering follower state: follower="Node at 098dcf7a-87ca-f538-a019-1aad945c1a01 [Follower]" leader=
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:03.977Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: entering follower state: follower="Node at 098dcf7a-87ca-f538-a019-1aad945c1a01 [Follower]" leader=
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.054Z [WARN]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.054Z [WARN]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.054Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: entering candidate state: node="Node at 098dcf7a-87ca-f538-a019-1aad945c1a01 [Candidate]" term=2
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.054Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: entering candidate state: node="Node at 098dcf7a-87ca-f538-a019-1aad945c1a01 [Candidate]" term=2
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.054Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: election won: tally=2
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.054Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: election won: tally=2
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: entering leader state: leader="Node at 098dcf7a-87ca-f538-a019-1aad945c1a01 [Leader]"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: entering leader state: leader="Node at 098dcf7a-87ca-f538-a019-1aad945c1a01 [Leader]"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: added peer, starting replication: peer=server-f45de3da-83af-17e7-245b-749c49dd5d03
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: added peer, starting replication: peer=server-f45de3da-83af-17e7-245b-749c49dd5d03
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: added peer, starting replication: peer=server-94420a5d-3657-3aa5-48df-ad3c6306faf8
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: added peer, starting replication: peer=server-94420a5d-3657-3aa5-48df-ad3c6306faf8
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.055Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.056Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.056Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:04.162Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.056887976 +0000 UTC m=+12.208682123 (1 nodes), 79.618253ms from start of poll, 82.003825ms from cluster start. Timeout at 2024-01-08 10:38:04.162249684 +0000 UTC m=+12.314043808, 105.361685ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:04.162Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.056887976 +0000 UTC m=+12.208682123 (1 nodes), 79.618253ms from start of poll, 82.003825ms from cluster start. Timeout at 2024-01-08 10:38:04.162249684 +0000 UTC m=+12.314043808, 105.361685ms after stability
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.164Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: starting snapshot up to: index=12
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.164Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: starting snapshot up to: index=12
    testing.go:153: 2024-01-08T10:38:04.164Z [INFO]  creating new snapshot: path=/tmp/raft714474000/snapshots/2-12-1704710284164.tmp
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.166Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: snapshot complete up to: index=12
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.166Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: snapshot complete up to: index=12
    testing.go:153: 2024-01-08T10:38:04.169Z [INFO]  creating new snapshot: path=/tmp/raft714474000/snapshots/2-23-1704710284169.tmp
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.171Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: copied to local snapshot: bytes=71
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.171Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.178Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: restored user snapshot: index=1
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.178Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: restored user snapshot: index=1
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.178Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.178Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.178Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.178Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.179Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.179Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.180Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.180Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.181Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.181Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.185Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.185Z [ERROR] server-098dcf7a-87ca-f538-a019-1aad945c1a01: failed to get log: index=23 error="log not found"
    testing.go:153: 2024-01-08T10:38:04.193Z [INFO]  creating new snapshot: path=/tmp/raft1904461371/snapshots/2-23-1704710284190.tmp
    testing.go:153: 2024-01-08T10:38:04.195Z [INFO]  creating new snapshot: path=/tmp/raft804999286/snapshots/2-23-1704710284195.tmp
testLoggerAdapter verbose: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:04.198Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: copied to local snapshot: bytes=71
    testing.go:149: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:04.198Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:04.198Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: Installed remote snapshot
    testing.go:149: server-94420a5d-3657-3aa5-48df-ad3c6306faf8: 2024-01-08T10:38:04.198Z [INFO]  server-94420a5d-3657-3aa5-48df-ad3c6306faf8: Installed remote snapshot
testLoggerAdapter verbose: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:04.203Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: copied to local snapshot: bytes=71
    testing.go:149: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:04.203Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:04.204Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: Installed remote snapshot
    testing.go:149: server-f45de3da-83af-17e7-245b-749c49dd5d03: 2024-01-08T10:38:04.204Z [INFO]  server-f45de3da-83af-17e7-245b-749c49dd5d03: Installed remote snapshot
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.205Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.205Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.206Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.206Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: pipelining replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.234Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.234Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-94420a5d-3657-3aa5-48df-ad3c6306faf8 94420a5d-3657-3aa5-48df-ad3c6306faf8}"
testLoggerAdapter verbose: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.234Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
    testing.go:149: server-098dcf7a-87ca-f538-a019-1aad945c1a01: 2024-01-08T10:38:04.234Z [INFO]  server-098dcf7a-87ca-f538-a019-1aad945c1a01: aborting pipeline replication: peer="{Voter server-f45de3da-83af-17e7-245b-749c49dd5d03 f45de3da-83af-17e7-245b-749c49dd5d03}"
testLoggerAdapter verbose: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.239Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4986a131-0e29-d8db-84f0-b13d86ecffb0 Address:4986a131-0e29-d8db-84f0-b13d86ecffb0} {Suffrage:Voter ID:server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 Address:fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7} {Suffrage:Voter ID:server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 Address:d00bdb0e-fdd7-ae44-7942-9e3b40b208d6}]"
    testing.go:149: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.239Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4986a131-0e29-d8db-84f0-b13d86ecffb0 Address:4986a131-0e29-d8db-84f0-b13d86ecffb0} {Suffrage:Voter ID:server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 Address:fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7} {Suffrage:Voter ID:server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 Address:d00bdb0e-fdd7-ae44-7942-9e3b40b208d6}]"
testLoggerAdapter verbose: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.240Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4986a131-0e29-d8db-84f0-b13d86ecffb0 Address:4986a131-0e29-d8db-84f0-b13d86ecffb0} {Suffrage:Voter ID:server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 Address:fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7} {Suffrage:Voter ID:server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 Address:d00bdb0e-fdd7-ae44-7942-9e3b40b208d6}]"
    testing.go:149: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.240Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4986a131-0e29-d8db-84f0-b13d86ecffb0 Address:4986a131-0e29-d8db-84f0-b13d86ecffb0} {Suffrage:Voter ID:server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 Address:fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7} {Suffrage:Voter ID:server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 Address:d00bdb0e-fdd7-ae44-7942-9e3b40b208d6}]"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.240Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4986a131-0e29-d8db-84f0-b13d86ecffb0 Address:4986a131-0e29-d8db-84f0-b13d86ecffb0} {Suffrage:Voter ID:server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 Address:fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7} {Suffrage:Voter ID:server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 Address:d00bdb0e-fdd7-ae44-7942-9e3b40b208d6}]"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.240Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4986a131-0e29-d8db-84f0-b13d86ecffb0 Address:4986a131-0e29-d8db-84f0-b13d86ecffb0} {Suffrage:Voter ID:server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 Address:fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7} {Suffrage:Voter ID:server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 Address:d00bdb0e-fdd7-ae44-7942-9e3b40b208d6}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:04.241Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:04.241Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.241Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: entering follower state: follower="Node at 4986a131-0e29-d8db-84f0-b13d86ecffb0 [Follower]" leader=
    testing.go:149: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.241Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: entering follower state: follower="Node at 4986a131-0e29-d8db-84f0-b13d86ecffb0 [Follower]" leader=
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.242Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: entering follower state: follower="Node at d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 [Follower]" leader=
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.242Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: entering follower state: follower="Node at d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 [Follower]" leader=
testLoggerAdapter verbose: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.254Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: entering follower state: follower="Node at fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 [Follower]" leader=
    testing.go:149: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.254Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: entering follower state: follower="Node at fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 [Follower]" leader=
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.306Z [WARN]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.306Z [WARN]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.306Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: entering candidate state: node="Node at d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 [Candidate]" term=2
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.306Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: entering candidate state: node="Node at d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 [Candidate]" term=2
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: election won: tally=2
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: election won: tally=2
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: entering leader state: leader="Node at d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 [Leader]"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: entering leader state: leader="Node at d00bdb0e-fdd7-ae44-7942-9e3b40b208d6 [Leader]"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: added peer, starting replication: peer=server-4986a131-0e29-d8db-84f0-b13d86ecffb0
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: added peer, starting replication: peer=server-4986a131-0e29-d8db-84f0-b13d86ecffb0
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: added peer, starting replication: peer=server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.307Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: added peer, starting replication: peer=server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.308Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.308Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.309Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.309Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:04.414Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.309642855 +0000 UTC m=+12.461436995 (1 nodes), 68.324811ms from start of poll, 70.65164ms from cluster start. Timeout at 2024-01-08 10:38:04.414695634 +0000 UTC m=+12.566489771, 105.052776ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:04.414Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.309642855 +0000 UTC m=+12.461436995 (1 nodes), 68.324811ms from start of poll, 70.65164ms from cluster start. Timeout at 2024-01-08 10:38:04.414695634 +0000 UTC m=+12.566489771, 105.052776ms after stability
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.417Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: starting snapshot up to: index=12
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.417Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: starting snapshot up to: index=12
    testing.go:153: 2024-01-08T10:38:04.417Z [INFO]  creating new snapshot: path=/tmp/raft1977575614/snapshots/2-12-1704710284417.tmp
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.420Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: snapshot complete up to: index=12
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.420Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: snapshot complete up to: index=12
    testing.go:153: 2024-01-08T10:38:04.423Z [INFO]  creating new snapshot: path=/tmp/raft1977575614/snapshots/2-113-1704710284423.tmp
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.425Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: copied to local snapshot: bytes=71
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.425Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.432Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: restored user snapshot: index=1
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.432Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: restored user snapshot: index=1
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.432Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.432Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.432Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.432Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.433Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.433Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.433Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.433Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.442Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.442Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
    testing.go:153: 2024-01-08T10:38:04.444Z [INFO]  creating new snapshot: path=/tmp/raft1064515515/snapshots/2-113-1704710284444.tmp
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.445Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.445Z [ERROR] server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: failed to get log: index=23 error="log not found"
    testing.go:153: 2024-01-08T10:38:04.446Z [INFO]  creating new snapshot: path=/tmp/raft1617187950/snapshots/2-113-1704710284446.tmp
testLoggerAdapter verbose: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.446Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: copied to local snapshot: bytes=71
    testing.go:149: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.446Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.447Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: Installed remote snapshot
    testing.go:149: server-4986a131-0e29-d8db-84f0-b13d86ecffb0: 2024-01-08T10:38:04.447Z [INFO]  server-4986a131-0e29-d8db-84f0-b13d86ecffb0: Installed remote snapshot
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.447Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.447Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
testLoggerAdapter verbose: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.455Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: copied to local snapshot: bytes=71
    testing.go:149: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.455Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.456Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: Installed remote snapshot
    testing.go:149: server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: 2024-01-08T10:38:04.456Z [INFO]  server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7: Installed remote snapshot
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.457Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.457Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: pipelining replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.479Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.479Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-4986a131-0e29-d8db-84f0-b13d86ecffb0 4986a131-0e29-d8db-84f0-b13d86ecffb0}"
testLoggerAdapter verbose: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.479Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
    testing.go:149: server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: 2024-01-08T10:38:04.479Z [INFO]  server-d00bdb0e-fdd7-ae44-7942-9e3b40b208d6: aborting pipeline replication: peer="{Voter server-fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7 fd738e3b-e743-1f65-9a98-2eb7aa1dd6b7}"
testLoggerAdapter verbose: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.486Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feadc6ef-66f3-ec21-4196-056bfb476f4e Address:feadc6ef-66f3-ec21-4196-056bfb476f4e} {Suffrage:Voter ID:server-775f9ab0-931c-56fd-fc34-06b6882983b0 Address:775f9ab0-931c-56fd-fc34-06b6882983b0} {Suffrage:Voter ID:server-0e4f2742-a822-31ba-1a6c-d32ef487226b Address:0e4f2742-a822-31ba-1a6c-d32ef487226b}]"
    testing.go:149: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.486Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feadc6ef-66f3-ec21-4196-056bfb476f4e Address:feadc6ef-66f3-ec21-4196-056bfb476f4e} {Suffrage:Voter ID:server-775f9ab0-931c-56fd-fc34-06b6882983b0 Address:775f9ab0-931c-56fd-fc34-06b6882983b0} {Suffrage:Voter ID:server-0e4f2742-a822-31ba-1a6c-d32ef487226b Address:0e4f2742-a822-31ba-1a6c-d32ef487226b}]"
testLoggerAdapter verbose: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.487Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feadc6ef-66f3-ec21-4196-056bfb476f4e Address:feadc6ef-66f3-ec21-4196-056bfb476f4e} {Suffrage:Voter ID:server-775f9ab0-931c-56fd-fc34-06b6882983b0 Address:775f9ab0-931c-56fd-fc34-06b6882983b0} {Suffrage:Voter ID:server-0e4f2742-a822-31ba-1a6c-d32ef487226b Address:0e4f2742-a822-31ba-1a6c-d32ef487226b}]"
    testing.go:149: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.487Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feadc6ef-66f3-ec21-4196-056bfb476f4e Address:feadc6ef-66f3-ec21-4196-056bfb476f4e} {Suffrage:Voter ID:server-775f9ab0-931c-56fd-fc34-06b6882983b0 Address:775f9ab0-931c-56fd-fc34-06b6882983b0} {Suffrage:Voter ID:server-0e4f2742-a822-31ba-1a6c-d32ef487226b Address:0e4f2742-a822-31ba-1a6c-d32ef487226b}]"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.487Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feadc6ef-66f3-ec21-4196-056bfb476f4e Address:feadc6ef-66f3-ec21-4196-056bfb476f4e} {Suffrage:Voter ID:server-775f9ab0-931c-56fd-fc34-06b6882983b0 Address:775f9ab0-931c-56fd-fc34-06b6882983b0} {Suffrage:Voter ID:server-0e4f2742-a822-31ba-1a6c-d32ef487226b Address:0e4f2742-a822-31ba-1a6c-d32ef487226b}]"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.487Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-feadc6ef-66f3-ec21-4196-056bfb476f4e Address:feadc6ef-66f3-ec21-4196-056bfb476f4e} {Suffrage:Voter ID:server-775f9ab0-931c-56fd-fc34-06b6882983b0 Address:775f9ab0-931c-56fd-fc34-06b6882983b0} {Suffrage:Voter ID:server-0e4f2742-a822-31ba-1a6c-d32ef487226b Address:0e4f2742-a822-31ba-1a6c-d32ef487226b}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:04.488Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:04.488Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.488Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: entering follower state: follower="Node at feadc6ef-66f3-ec21-4196-056bfb476f4e [Follower]" leader=
    testing.go:149: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.488Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: entering follower state: follower="Node at feadc6ef-66f3-ec21-4196-056bfb476f4e [Follower]" leader=
testLoggerAdapter verbose: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.488Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: entering follower state: follower="Node at 775f9ab0-931c-56fd-fc34-06b6882983b0 [Follower]" leader=
    testing.go:149: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.488Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: entering follower state: follower="Node at 775f9ab0-931c-56fd-fc34-06b6882983b0 [Follower]" leader=
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.488Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: entering follower state: follower="Node at 0e4f2742-a822-31ba-1a6c-d32ef487226b [Follower]" leader=
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.488Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: entering follower state: follower="Node at 0e4f2742-a822-31ba-1a6c-d32ef487226b [Follower]" leader=
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.553Z [WARN]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.553Z [WARN]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.553Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: entering candidate state: node="Node at 0e4f2742-a822-31ba-1a6c-d32ef487226b [Candidate]" term=2
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.553Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: entering candidate state: node="Node at 0e4f2742-a822-31ba-1a6c-d32ef487226b [Candidate]" term=2
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: election won: tally=2
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: election won: tally=2
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: entering leader state: leader="Node at 0e4f2742-a822-31ba-1a6c-d32ef487226b [Leader]"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: entering leader state: leader="Node at 0e4f2742-a822-31ba-1a6c-d32ef487226b [Leader]"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: added peer, starting replication: peer=server-feadc6ef-66f3-ec21-4196-056bfb476f4e
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: added peer, starting replication: peer=server-feadc6ef-66f3-ec21-4196-056bfb476f4e
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: added peer, starting replication: peer=server-775f9ab0-931c-56fd-fc34-06b6882983b0
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: added peer, starting replication: peer=server-775f9ab0-931c-56fd-fc34-06b6882983b0
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.554Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:04.662Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.554441032 +0000 UTC m=+12.706234883 (1 nodes), 66.177681ms from start of poll, 70.387416ms from cluster start. Timeout at 2024-01-08 10:38:04.661993672 +0000 UTC m=+12.813787791, 107.552908ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:04.662Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.554441032 +0000 UTC m=+12.706234883 (1 nodes), 66.177681ms from start of poll, 70.387416ms from cluster start. Timeout at 2024-01-08 10:38:04.661993672 +0000 UTC m=+12.813787791, 107.552908ms after stability
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.665Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: starting snapshot up to: index=12
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.665Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: starting snapshot up to: index=12
    testing.go:153: 2024-01-08T10:38:04.665Z [INFO]  creating new snapshot: path=/tmp/raft4035640968/snapshots/2-12-1704710284665.tmp
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.675Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: snapshot complete up to: index=12
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.675Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: snapshot complete up to: index=12
    testing.go:153: 2024-01-08T10:38:04.678Z [INFO]  creating new snapshot: path=/tmp/raft4035640968/snapshots/2-1013-1704710284678.tmp
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.690Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: copied to local snapshot: bytes=71
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.690Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.695Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: restored user snapshot: index=1
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.695Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: restored user snapshot: index=1
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.696Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.696Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.697Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.697Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.697Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.697Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.697Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.697Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.712Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.712Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.714Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.714Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to get log: index=23 error="log not found"
    testing.go:153: 2024-01-08T10:38:04.714Z [INFO]  creating new snapshot: path=/tmp/raft3227314528/snapshots/2-1013-1704710284714.tmp
testLoggerAdapter verbose: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.716Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: copied to local snapshot: bytes=71
    testing.go:153: 2024-01-08T10:38:04.716Z [INFO]  creating new snapshot: path=/tmp/raft4173808371/snapshots/2-1013-1704710284716.tmp
    testing.go:149: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.716Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.718Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: Installed remote snapshot
    testing.go:149: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.718Z [INFO]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: Installed remote snapshot
testLoggerAdapter verbose: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.719Z [WARN]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: skipping application of old log: index=22
    testing.go:149: server-feadc6ef-66f3-ec21-4196-056bfb476f4e: 2024-01-08T10:38:04.719Z [WARN]  server-feadc6ef-66f3-ec21-4196-056bfb476f4e: skipping application of old log: index=22
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.719Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.719Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
testLoggerAdapter verbose: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.727Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: copied to local snapshot: bytes=71
    testing.go:149: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.727Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.728Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: Installed remote snapshot
    testing.go:149: server-775f9ab0-931c-56fd-fc34-06b6882983b0: 2024-01-08T10:38:04.728Z [INFO]  server-775f9ab0-931c-56fd-fc34-06b6882983b0: Installed remote snapshot
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.729Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.729Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: pipelining replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.758Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.758Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-775f9ab0-931c-56fd-fc34-06b6882983b0 775f9ab0-931c-56fd-fc34-06b6882983b0}"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.759Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.759Z [INFO]  server-0e4f2742-a822-31ba-1a6c-d32ef487226b: aborting pipeline replication: peer="{Voter server-feadc6ef-66f3-ec21-4196-056bfb476f4e feadc6ef-66f3-ec21-4196-056bfb476f4e}"
testLoggerAdapter verbose: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.810Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to heartbeat to: peer=775f9ab0-931c-56fd-fc34-06b6882983b0 error="command timed out"
    testing.go:149: server-0e4f2742-a822-31ba-1a6c-d32ef487226b: 2024-01-08T10:38:04.810Z [ERROR] server-0e4f2742-a822-31ba-1a6c-d32ef487226b: failed to heartbeat to: peer=775f9ab0-931c-56fd-fc34-06b6882983b0 error="command timed out"
testLoggerAdapter verbose: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:04.815Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db Address:ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db} {Suffrage:Voter ID:server-3e9c2b68-3865-be7f-e7f4-51c30731865e Address:3e9c2b68-3865-be7f-e7f4-51c30731865e} {Suffrage:Voter ID:server-8d254c8d-6475-2d18-bac1-5407b74eda87 Address:8d254c8d-6475-2d18-bac1-5407b74eda87}]"
    testing.go:149: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:04.815Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db Address:ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db} {Suffrage:Voter ID:server-3e9c2b68-3865-be7f-e7f4-51c30731865e Address:3e9c2b68-3865-be7f-e7f4-51c30731865e} {Suffrage:Voter ID:server-8d254c8d-6475-2d18-bac1-5407b74eda87 Address:8d254c8d-6475-2d18-bac1-5407b74eda87}]"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.816Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db Address:ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db} {Suffrage:Voter ID:server-3e9c2b68-3865-be7f-e7f4-51c30731865e Address:3e9c2b68-3865-be7f-e7f4-51c30731865e} {Suffrage:Voter ID:server-8d254c8d-6475-2d18-bac1-5407b74eda87 Address:8d254c8d-6475-2d18-bac1-5407b74eda87}]"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.816Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db Address:ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db} {Suffrage:Voter ID:server-3e9c2b68-3865-be7f-e7f4-51c30731865e Address:3e9c2b68-3865-be7f-e7f4-51c30731865e} {Suffrage:Voter ID:server-8d254c8d-6475-2d18-bac1-5407b74eda87 Address:8d254c8d-6475-2d18-bac1-5407b74eda87}]"
testLoggerAdapter verbose: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:04.816Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db Address:ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db} {Suffrage:Voter ID:server-3e9c2b68-3865-be7f-e7f4-51c30731865e Address:3e9c2b68-3865-be7f-e7f4-51c30731865e} {Suffrage:Voter ID:server-8d254c8d-6475-2d18-bac1-5407b74eda87 Address:8d254c8d-6475-2d18-bac1-5407b74eda87}]"
    testing.go:149: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:04.816Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db Address:ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db} {Suffrage:Voter ID:server-3e9c2b68-3865-be7f-e7f4-51c30731865e Address:3e9c2b68-3865-be7f-e7f4-51c30731865e} {Suffrage:Voter ID:server-8d254c8d-6475-2d18-bac1-5407b74eda87 Address:8d254c8d-6475-2d18-bac1-5407b74eda87}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:04.816Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:04.816Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.816Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: entering follower state: follower="Node at 3e9c2b68-3865-be7f-e7f4-51c30731865e [Follower]" leader=
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.816Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: entering follower state: follower="Node at 3e9c2b68-3865-be7f-e7f4-51c30731865e [Follower]" leader=
testLoggerAdapter verbose: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:04.817Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: entering follower state: follower="Node at 8d254c8d-6475-2d18-bac1-5407b74eda87 [Follower]" leader=
    testing.go:149: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:04.817Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: entering follower state: follower="Node at 8d254c8d-6475-2d18-bac1-5407b74eda87 [Follower]" leader=
testLoggerAdapter verbose: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:04.817Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: entering follower state: follower="Node at ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db [Follower]" leader=
    testing.go:149: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:04.817Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: entering follower state: follower="Node at ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db [Follower]" leader=
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.870Z [WARN]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.870Z [WARN]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.870Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: entering candidate state: node="Node at 3e9c2b68-3865-be7f-e7f4-51c30731865e [Candidate]" term=2
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.870Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: entering candidate state: node="Node at 3e9c2b68-3865-be7f-e7f4-51c30731865e [Candidate]" term=2
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: election won: tally=2
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: election won: tally=2
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: entering leader state: leader="Node at 3e9c2b68-3865-be7f-e7f4-51c30731865e [Leader]"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: entering leader state: leader="Node at 3e9c2b68-3865-be7f-e7f4-51c30731865e [Leader]"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: added peer, starting replication: peer=server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: added peer, starting replication: peer=server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: added peer, starting replication: peer=server-8d254c8d-6475-2d18-bac1-5407b74eda87
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: added peer, starting replication: peer=server-8d254c8d-6475-2d18-bac1-5407b74eda87
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.871Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.874Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.874Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
testLoggerAdapter verbose: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:04.877Z [WARN]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: rejecting vote request since our last term is greater: candidate=3e9c2b68-3865-be7f-e7f4-51c30731865e last-term=2 last-candidate-term=1
    testing.go:149: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:04.877Z [WARN]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: rejecting vote request since our last term is greater: candidate=3e9c2b68-3865-be7f-e7f4-51c30731865e last-term=2 last-candidate-term=1
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:04.985Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.877796802 +0000 UTC m=+13.029590655 (1 nodes), 60.913446ms from start of poll, 63.134868ms from cluster start. Timeout at 2024-01-08 10:38:04.985804701 +0000 UTC m=+13.137598553, 108.007898ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:04.985Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:04.877796802 +0000 UTC m=+13.029590655 (1 nodes), 60.913446ms from start of poll, 63.134868ms from cluster start. Timeout at 2024-01-08 10:38:04.985804701 +0000 UTC m=+13.137598553, 108.007898ms after stability
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.986Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: starting snapshot up to: index=12
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.986Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: starting snapshot up to: index=12
    testing.go:153: 2024-01-08T10:38:04.986Z [INFO]  creating new snapshot: path=/tmp/raft1807093782/snapshots/2-12-1704710284986.tmp
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.986Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: snapshot complete up to: index=12
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.986Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: snapshot complete up to: index=12
    testing.go:153: 2024-01-08T10:38:04.987Z [INFO]  creating new snapshot: path=/tmp/raft1807093782/snapshots/2-10013-1704710284987.tmp
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: copied to local snapshot: bytes=71
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: restored user snapshot: index=1
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: restored user snapshot: index=1
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.987Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.993Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.993Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.996Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:04.996Z [ERROR] server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to get log: index=23 error="log not found"
    testing.go:153: 2024-01-08T10:38:05.006Z [INFO]  creating new snapshot: path=/tmp/raft1386349252/snapshots/2-10013-1704710285006.tmp
testLoggerAdapter verbose: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:05.007Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: copied to local snapshot: bytes=71
    testing.go:149: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:05.007Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:05.007Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: Installed remote snapshot
    testing.go:153: 2024-01-08T10:38:05.007Z [INFO]  creating new snapshot: path=/tmp/raft934918510/snapshots/2-10013-1704710285007.tmp
    testing.go:149: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:05.007Z [INFO]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: Installed remote snapshot
testLoggerAdapter verbose: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:05.011Z [WARN]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: skipping application of old log: index=22
    testing.go:149: server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: 2024-01-08T10:38:05.011Z [WARN]  server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db: skipping application of old log: index=22
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.012Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.012Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.043Z [WARN]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to contact: server-id=server-8d254c8d-6475-2d18-bac1-5407b74eda87 time=54.571778ms
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.043Z [WARN]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: failed to contact: server-id=server-8d254c8d-6475-2d18-bac1-5407b74eda87 time=54.571778ms
testLoggerAdapter verbose: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:05.046Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: copied to local snapshot: bytes=71
    testing.go:149: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:05.046Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: copied to local snapshot: bytes=71
testLoggerAdapter verbose: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:05.047Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: Installed remote snapshot
    testing.go:149: server-8d254c8d-6475-2d18-bac1-5407b74eda87: 2024-01-08T10:38:05.047Z [INFO]  server-8d254c8d-6475-2d18-bac1-5407b74eda87: Installed remote snapshot
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.047Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.047Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: pipelining replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.098Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.098Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-8d254c8d-6475-2d18-bac1-5407b74eda87 8d254c8d-6475-2d18-bac1-5407b74eda87}"
testLoggerAdapter verbose: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.099Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
    testing.go:149: server-3e9c2b68-3865-be7f-e7f4-51c30731865e: 2024-01-08T10:38:05.099Z [INFO]  server-3e9c2b68-3865-be7f-e7f4-51c30731865e: aborting pipeline replication: peer="{Voter server-ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db ff9e13ce-2ef8-5dd4-fedb-e8f70f6a71db}"
--- PASS: TestRaft_UserRestore (1.75s)
=== RUN   TestRaft_SendSnapshotFollower
testLoggerAdapter verbose: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.104Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50818b78-21b2-f069-74c7-8e8c01d67b7b Address:50818b78-21b2-f069-74c7-8e8c01d67b7b} {Suffrage:Voter ID:server-c3c6dfcf-faaa-d01c-6992-647e765c609e Address:c3c6dfcf-faaa-d01c-6992-647e765c609e} {Suffrage:Voter ID:server-260b7ece-6832-1529-4e2f-25505b09c99e Address:260b7ece-6832-1529-4e2f-25505b09c99e}]"
    testing.go:149: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.104Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50818b78-21b2-f069-74c7-8e8c01d67b7b Address:50818b78-21b2-f069-74c7-8e8c01d67b7b} {Suffrage:Voter ID:server-c3c6dfcf-faaa-d01c-6992-647e765c609e Address:c3c6dfcf-faaa-d01c-6992-647e765c609e} {Suffrage:Voter ID:server-260b7ece-6832-1529-4e2f-25505b09c99e Address:260b7ece-6832-1529-4e2f-25505b09c99e}]"
testLoggerAdapter verbose: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.105Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50818b78-21b2-f069-74c7-8e8c01d67b7b Address:50818b78-21b2-f069-74c7-8e8c01d67b7b} {Suffrage:Voter ID:server-c3c6dfcf-faaa-d01c-6992-647e765c609e Address:c3c6dfcf-faaa-d01c-6992-647e765c609e} {Suffrage:Voter ID:server-260b7ece-6832-1529-4e2f-25505b09c99e Address:260b7ece-6832-1529-4e2f-25505b09c99e}]"
    testing.go:149: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.105Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50818b78-21b2-f069-74c7-8e8c01d67b7b Address:50818b78-21b2-f069-74c7-8e8c01d67b7b} {Suffrage:Voter ID:server-c3c6dfcf-faaa-d01c-6992-647e765c609e Address:c3c6dfcf-faaa-d01c-6992-647e765c609e} {Suffrage:Voter ID:server-260b7ece-6832-1529-4e2f-25505b09c99e Address:260b7ece-6832-1529-4e2f-25505b09c99e}]"
testLoggerAdapter verbose: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.105Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: entering follower state: follower="Node at 50818b78-21b2-f069-74c7-8e8c01d67b7b [Follower]" leader=
    testing.go:149: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.105Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: entering follower state: follower="Node at 50818b78-21b2-f069-74c7-8e8c01d67b7b [Follower]" leader=
testLoggerAdapter verbose: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.106Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: entering follower state: follower="Node at c3c6dfcf-faaa-d01c-6992-647e765c609e [Follower]" leader=
    testing.go:149: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.106Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: entering follower state: follower="Node at c3c6dfcf-faaa-d01c-6992-647e765c609e [Follower]" leader=
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.107Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50818b78-21b2-f069-74c7-8e8c01d67b7b Address:50818b78-21b2-f069-74c7-8e8c01d67b7b} {Suffrage:Voter ID:server-c3c6dfcf-faaa-d01c-6992-647e765c609e Address:c3c6dfcf-faaa-d01c-6992-647e765c609e} {Suffrage:Voter ID:server-260b7ece-6832-1529-4e2f-25505b09c99e Address:260b7ece-6832-1529-4e2f-25505b09c99e}]"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.107Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50818b78-21b2-f069-74c7-8e8c01d67b7b Address:50818b78-21b2-f069-74c7-8e8c01d67b7b} {Suffrage:Voter ID:server-c3c6dfcf-faaa-d01c-6992-647e765c609e Address:c3c6dfcf-faaa-d01c-6992-647e765c609e} {Suffrage:Voter ID:server-260b7ece-6832-1529-4e2f-25505b09c99e Address:260b7ece-6832-1529-4e2f-25505b09c99e}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.107Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:05.107Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.107Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: entering follower state: follower="Node at 260b7ece-6832-1529-4e2f-25505b09c99e [Follower]" leader=
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.107Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: entering follower state: follower="Node at 260b7ece-6832-1529-4e2f-25505b09c99e [Follower]" leader=
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.181Z [WARN]  server-260b7ece-6832-1529-4e2f-25505b09c99e: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.181Z [WARN]  server-260b7ece-6832-1529-4e2f-25505b09c99e: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.181Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: entering candidate state: node="Node at 260b7ece-6832-1529-4e2f-25505b09c99e [Candidate]" term=2
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.181Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: entering candidate state: node="Node at 260b7ece-6832-1529-4e2f-25505b09c99e [Candidate]" term=2
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: election won: tally=2
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: election won: tally=2
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: entering leader state: leader="Node at 260b7ece-6832-1529-4e2f-25505b09c99e [Leader]"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: entering leader state: leader="Node at 260b7ece-6832-1529-4e2f-25505b09c99e [Leader]"
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: added peer, starting replication: peer=server-50818b78-21b2-f069-74c7-8e8c01d67b7b
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: added peer, starting replication: peer=server-50818b78-21b2-f069-74c7-8e8c01d67b7b
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: added peer, starting replication: peer=server-c3c6dfcf-faaa-d01c-6992-647e765c609e
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: added peer, starting replication: peer=server-c3c6dfcf-faaa-d01c-6992-647e765c609e
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: pipelining replication: peer="{Voter server-c3c6dfcf-faaa-d01c-6992-647e765c609e c3c6dfcf-faaa-d01c-6992-647e765c609e}"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: pipelining replication: peer="{Voter server-c3c6dfcf-faaa-d01c-6992-647e765c609e c3c6dfcf-faaa-d01c-6992-647e765c609e}"
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: pipelining replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.182Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: pipelining replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.290Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:05.182522759 +0000 UTC m=+13.334316610 (2 nodes), 74.879522ms from start of poll, 78.763423ms from cluster start. Timeout at 2024-01-08 10:38:05.290116762 +0000 UTC m=+13.441910875, 107.594265ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:05.290Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:05.182522759 +0000 UTC m=+13.334316610 (2 nodes), 74.879522ms from start of poll, 78.763423ms from cluster start. Timeout at 2024-01-08 10:38:05.290116762 +0000 UTC m=+13.441910875, 107.594265ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.290Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:05.290Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.434Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:05.29090896 +0000 UTC m=+13.442703074 (1 nodes), 3.984µs from start of poll, 187.149887ms from cluster start. Timeout at 2024-01-08 10:38:05.433965027 +0000 UTC m=+13.585758877, 143.055803ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:05.434Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:05.29090896 +0000 UTC m=+13.442703074 (1 nodes), 3.984µs from start of poll, 187.149887ms from cluster start. Timeout at 2024-01-08 10:38:05.433965027 +0000 UTC m=+13.585758877, 143.055803ms after stability
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.434Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to pipeline appendEntries: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}" error="append pipeline closed"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.434Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to pipeline appendEntries: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}" error="append pipeline closed"
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.434Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: aborting pipeline replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.434Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: aborting pipeline replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.434Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to appendEntries to: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}" error="failed to connect to peer: 50818b78-21b2-f069-74c7-8e8c01d67b7b"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.434Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to appendEntries to: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}" error="failed to connect to peer: 50818b78-21b2-f069-74c7-8e8c01d67b7b"
    raft_test.go:1320: [INFO] Finished apply without behind follower
testLoggerAdapter verbose: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.436Z [ERROR] server-50818b78-21b2-f069-74c7-8e8c01d67b7b: failed to take snapshot: error="nothing new to snapshot"
    testing.go:149: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.436Z [ERROR] server-50818b78-21b2-f069-74c7-8e8c01d67b7b: failed to take snapshot: error="nothing new to snapshot"
testLoggerAdapter verbose: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.436Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: starting snapshot up to: index=100
    testing.go:149: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.436Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: starting snapshot up to: index=100
    testing.go:153: 2024-01-08T10:38:05.436Z [INFO]  creating new snapshot: path=/tmp/raft1515685710/snapshots/2-100-1704710285436.tmp
testLoggerAdapter verbose: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.436Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: compacting logs: from=1 to=92
    testing.go:149: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.436Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: compacting logs: from=1 to=92
testLoggerAdapter verbose: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.436Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: snapshot complete up to: index=100
    testing.go:149: server-c3c6dfcf-faaa-d01c-6992-647e765c609e: 2024-01-08T10:38:05.436Z [INFO]  server-c3c6dfcf-faaa-d01c-6992-647e765c609e: snapshot complete up to: index=100
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.436Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: starting snapshot up to: index=102
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.436Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: starting snapshot up to: index=102
    testing.go:153: 2024-01-08T10:38:05.436Z [INFO]  creating new snapshot: path=/tmp/raft2647540892/snapshots/2-102-1704710285436.tmp
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.437Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: compacting logs: from=1 to=92
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.437Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: compacting logs: from=1 to=92
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.437Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: snapshot complete up to: index=102
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.437Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: snapshot complete up to: index=102
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.445Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to get log: index=2 error="log not found"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.445Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to get log: index=2 error="log not found"
    testing.go:153: 2024-01-08T10:38:05.446Z [INFO]  creating new snapshot: path=/tmp/raft2889670843/snapshots/2-102-1704710285446.tmp
testLoggerAdapter verbose: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.446Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: copied to local snapshot: bytes=693
    testing.go:149: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.446Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: copied to local snapshot: bytes=693
testLoggerAdapter verbose: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.446Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: Installed remote snapshot
    testing.go:149: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.446Z [INFO]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: Installed remote snapshot
testLoggerAdapter verbose: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.447Z [WARN]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: skipping application of old log: index=102
    testing.go:149: server-50818b78-21b2-f069-74c7-8e8c01d67b7b: 2024-01-08T10:38:05.447Z [WARN]  server-50818b78-21b2-f069-74c7-8e8c01d67b7b: skipping application of old log: index=102
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.447Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: pipelining replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.447Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: pipelining replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.447Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: aborting pipeline replication: peer="{Voter server-c3c6dfcf-faaa-d01c-6992-647e765c609e c3c6dfcf-faaa-d01c-6992-647e765c609e}"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.447Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: aborting pipeline replication: peer="{Voter server-c3c6dfcf-faaa-d01c-6992-647e765c609e c3c6dfcf-faaa-d01c-6992-647e765c609e}"
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.447Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: aborting pipeline replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.447Z [INFO]  server-260b7ece-6832-1529-4e2f-25505b09c99e: aborting pipeline replication: peer="{Voter server-50818b78-21b2-f069-74c7-8e8c01d67b7b 50818b78-21b2-f069-74c7-8e8c01d67b7b}"
testLoggerAdapter verbose: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.497Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to heartbeat to: peer=50818b78-21b2-f069-74c7-8e8c01d67b7b error="command timed out"
    testing.go:149: server-260b7ece-6832-1529-4e2f-25505b09c99e: 2024-01-08T10:38:05.497Z [ERROR] server-260b7ece-6832-1529-4e2f-25505b09c99e: failed to heartbeat to: peer=50818b78-21b2-f069-74c7-8e8c01d67b7b error="command timed out"
--- PASS: TestRaft_SendSnapshotFollower (0.40s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.507Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd998d89-f625-c93a-5456-57496f6a21e7 Address:cd998d89-f625-c93a-5456-57496f6a21e7} {Suffrage:Voter ID:server-6b413bb1-5a87-f14d-40cb-58dabc00e143 Address:6b413bb1-5a87-f14d-40cb-58dabc00e143} {Suffrage:Voter ID:server-7cc1abf9-3541-33cc-2557-1e4612c59148 Address:7cc1abf9-3541-33cc-2557-1e4612c59148}]"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.507Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd998d89-f625-c93a-5456-57496f6a21e7 Address:cd998d89-f625-c93a-5456-57496f6a21e7} {Suffrage:Voter ID:server-6b413bb1-5a87-f14d-40cb-58dabc00e143 Address:6b413bb1-5a87-f14d-40cb-58dabc00e143} {Suffrage:Voter ID:server-7cc1abf9-3541-33cc-2557-1e4612c59148 Address:7cc1abf9-3541-33cc-2557-1e4612c59148}]"
testLoggerAdapter verbose: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.507Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd998d89-f625-c93a-5456-57496f6a21e7 Address:cd998d89-f625-c93a-5456-57496f6a21e7} {Suffrage:Voter ID:server-6b413bb1-5a87-f14d-40cb-58dabc00e143 Address:6b413bb1-5a87-f14d-40cb-58dabc00e143} {Suffrage:Voter ID:server-7cc1abf9-3541-33cc-2557-1e4612c59148 Address:7cc1abf9-3541-33cc-2557-1e4612c59148}]"
    testing.go:149: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.507Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd998d89-f625-c93a-5456-57496f6a21e7 Address:cd998d89-f625-c93a-5456-57496f6a21e7} {Suffrage:Voter ID:server-6b413bb1-5a87-f14d-40cb-58dabc00e143 Address:6b413bb1-5a87-f14d-40cb-58dabc00e143} {Suffrage:Voter ID:server-7cc1abf9-3541-33cc-2557-1e4612c59148 Address:7cc1abf9-3541-33cc-2557-1e4612c59148}]"
testLoggerAdapter verbose: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.508Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd998d89-f625-c93a-5456-57496f6a21e7 Address:cd998d89-f625-c93a-5456-57496f6a21e7} {Suffrage:Voter ID:server-6b413bb1-5a87-f14d-40cb-58dabc00e143 Address:6b413bb1-5a87-f14d-40cb-58dabc00e143} {Suffrage:Voter ID:server-7cc1abf9-3541-33cc-2557-1e4612c59148 Address:7cc1abf9-3541-33cc-2557-1e4612c59148}]"
    testing.go:149: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.508Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd998d89-f625-c93a-5456-57496f6a21e7 Address:cd998d89-f625-c93a-5456-57496f6a21e7} {Suffrage:Voter ID:server-6b413bb1-5a87-f14d-40cb-58dabc00e143 Address:6b413bb1-5a87-f14d-40cb-58dabc00e143} {Suffrage:Voter ID:server-7cc1abf9-3541-33cc-2557-1e4612c59148 Address:7cc1abf9-3541-33cc-2557-1e4612c59148}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.508Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:05.508Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.508Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: entering follower state: follower="Node at cd998d89-f625-c93a-5456-57496f6a21e7 [Follower]" leader=
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.508Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: entering follower state: follower="Node at cd998d89-f625-c93a-5456-57496f6a21e7 [Follower]" leader=
testLoggerAdapter verbose: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.509Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: entering follower state: follower="Node at 6b413bb1-5a87-f14d-40cb-58dabc00e143 [Follower]" leader=
    testing.go:149: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.509Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: entering follower state: follower="Node at 6b413bb1-5a87-f14d-40cb-58dabc00e143 [Follower]" leader=
testLoggerAdapter verbose: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.509Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: entering follower state: follower="Node at 7cc1abf9-3541-33cc-2557-1e4612c59148 [Follower]" leader=
    testing.go:149: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.509Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: entering follower state: follower="Node at 7cc1abf9-3541-33cc-2557-1e4612c59148 [Follower]" leader=
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.577Z [WARN]  server-cd998d89-f625-c93a-5456-57496f6a21e7: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.577Z [WARN]  server-cd998d89-f625-c93a-5456-57496f6a21e7: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.577Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: entering candidate state: node="Node at cd998d89-f625-c93a-5456-57496f6a21e7 [Candidate]" term=2
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.577Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: entering candidate state: node="Node at cd998d89-f625-c93a-5456-57496f6a21e7 [Candidate]" term=2
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: election won: tally=2
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: election won: tally=2
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: entering leader state: leader="Node at cd998d89-f625-c93a-5456-57496f6a21e7 [Leader]"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: entering leader state: leader="Node at cd998d89-f625-c93a-5456-57496f6a21e7 [Leader]"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: added peer, starting replication: peer=server-6b413bb1-5a87-f14d-40cb-58dabc00e143
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: added peer, starting replication: peer=server-6b413bb1-5a87-f14d-40cb-58dabc00e143
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: added peer, starting replication: peer=server-7cc1abf9-3541-33cc-2557-1e4612c59148
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: added peer, starting replication: peer=server-7cc1abf9-3541-33cc-2557-1e4612c59148
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: pipelining replication: peer="{Voter server-7cc1abf9-3541-33cc-2557-1e4612c59148 7cc1abf9-3541-33cc-2557-1e4612c59148}"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: pipelining replication: peer="{Voter server-7cc1abf9-3541-33cc-2557-1e4612c59148 7cc1abf9-3541-33cc-2557-1e4612c59148}"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: pipelining replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.578Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: pipelining replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
testLoggerAdapter verbose: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.578Z [WARN]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: rejecting vote request since our last term is greater: candidate=cd998d89-f625-c93a-5456-57496f6a21e7 last-term=2 last-candidate-term=1
    testing.go:149: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.578Z [WARN]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: rejecting vote request since our last term is greater: candidate=cd998d89-f625-c93a-5456-57496f6a21e7 last-term=2 last-candidate-term=1
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.686Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:05.579630867 +0000 UTC m=+13.731425015 (2 nodes), 70.809181ms from start of poll, 73.183863ms from cluster start. Timeout at 2024-01-08 10:38:05.685933285 +0000 UTC m=+13.837727401, 106.302386ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:05.686Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:05.579630867 +0000 UTC m=+13.731425015 (2 nodes), 70.809181ms from start of poll, 73.183863ms from cluster start. Timeout at 2024-01-08 10:38:05.685933285 +0000 UTC m=+13.837727401, 106.302386ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.686Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:05.686Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.793Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:05.686548919 +0000 UTC m=+13.838343047 (1 nodes), 3.578µs from start of poll, 180.101895ms from cluster start. Timeout at 2024-01-08 10:38:05.793814568 +0000 UTC m=+13.945608416, 107.265369ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:05.793Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:05.686548919 +0000 UTC m=+13.838343047 (1 nodes), 3.578µs from start of poll, 180.101895ms from cluster start. Timeout at 2024-01-08 10:38:05.793814568 +0000 UTC m=+13.945608416, 107.265369ms after stability
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.794Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to pipeline appendEntries: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}" error="append pipeline closed"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.794Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to pipeline appendEntries: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}" error="append pipeline closed"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.794Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: aborting pipeline replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.794Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: aborting pipeline replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.794Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to appendEntries to: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}" error="failed to connect to peer: 6b413bb1-5a87-f14d-40cb-58dabc00e143"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.794Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to appendEntries to: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}" error="failed to connect to peer: 6b413bb1-5a87-f14d-40cb-58dabc00e143"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.808Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to appendEntries to: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}" error="failed to connect to peer: 6b413bb1-5a87-f14d-40cb-58dabc00e143"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.808Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to appendEntries to: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}" error="failed to connect to peer: 6b413bb1-5a87-f14d-40cb-58dabc00e143"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.810Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to heartbeat to: peer=6b413bb1-5a87-f14d-40cb-58dabc00e143 error="failed to connect to peer: 6b413bb1-5a87-f14d-40cb-58dabc00e143"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.810Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to heartbeat to: peer=6b413bb1-5a87-f14d-40cb-58dabc00e143 error="failed to connect to peer: 6b413bb1-5a87-f14d-40cb-58dabc00e143"
    raft_test.go:1362: [INFO] Finished apply without behind follower
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.814Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: starting snapshot up to: index=102
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.814Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: starting snapshot up to: index=102
    testing.go:153: 2024-01-08T10:38:05.814Z [INFO]  creating new snapshot: path=/tmp/raft2395712373/snapshots/2-102-1704710285814.tmp
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.814Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: compacting logs: from=1 to=92
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.814Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: compacting logs: from=1 to=92
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.814Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: snapshot complete up to: index=102
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.814Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: snapshot complete up to: index=102
testLoggerAdapter verbose: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.814Z [ERROR] server-6b413bb1-5a87-f14d-40cb-58dabc00e143: failed to take snapshot: error="nothing new to snapshot"
    testing.go:149: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.814Z [ERROR] server-6b413bb1-5a87-f14d-40cb-58dabc00e143: failed to take snapshot: error="nothing new to snapshot"
testLoggerAdapter verbose: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.815Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: starting snapshot up to: index=68
    testing.go:149: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.815Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: starting snapshot up to: index=68
    testing.go:153: 2024-01-08T10:38:05.815Z [INFO]  creating new snapshot: path=/tmp/raft3810678984/snapshots/2-68-1704710285815.tmp
testLoggerAdapter verbose: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.815Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: compacting logs: from=1 to=68
    testing.go:149: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.815Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: compacting logs: from=1 to=68
testLoggerAdapter verbose: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.815Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: snapshot complete up to: index=68
    testing.go:149: server-7cc1abf9-3541-33cc-2557-1e4612c59148: 2024-01-08T10:38:05.815Z [INFO]  server-7cc1abf9-3541-33cc-2557-1e4612c59148: snapshot complete up to: index=68
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.822Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to get log: index=2 error="log not found"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.822Z [ERROR] server-cd998d89-f625-c93a-5456-57496f6a21e7: failed to get log: index=2 error="log not found"
    raft_test.go:1383: [INFO] Finished apply without behind follower
    testing.go:153: 2024-01-08T10:38:05.823Z [INFO]  creating new snapshot: path=/tmp/raft3679848351/snapshots/2-102-1704710285823.tmp
testLoggerAdapter verbose: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.823Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: copied to local snapshot: bytes=693
    testing.go:149: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.823Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: copied to local snapshot: bytes=693
testLoggerAdapter verbose: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.824Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: Installed remote snapshot
    testing.go:149: server-6b413bb1-5a87-f14d-40cb-58dabc00e143: 2024-01-08T10:38:05.824Z [INFO]  server-6b413bb1-5a87-f14d-40cb-58dabc00e143: Installed remote snapshot
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.824Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: pipelining replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.824Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: pipelining replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.845Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: aborting pipeline replication: peer="{Voter server-7cc1abf9-3541-33cc-2557-1e4612c59148 7cc1abf9-3541-33cc-2557-1e4612c59148}"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.845Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: aborting pipeline replication: peer="{Voter server-7cc1abf9-3541-33cc-2557-1e4612c59148 7cc1abf9-3541-33cc-2557-1e4612c59148}"
testLoggerAdapter verbose: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.846Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: aborting pipeline replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
    testing.go:149: server-cd998d89-f625-c93a-5456-57496f6a21e7: 2024-01-08T10:38:05.846Z [INFO]  server-cd998d89-f625-c93a-5456-57496f6a21e7: aborting pipeline replication: peer="{Voter server-6b413bb1-5a87-f14d-40cb-58dabc00e143 6b413bb1-5a87-f14d-40cb-58dabc00e143}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.35s)
=== RUN   TestRaft_ReJoinFollower
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.847Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.847Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.847Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.847Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:05.847Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:05.847Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:05.847Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:05.847Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.847Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering follower state: follower="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Follower]" leader=
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.847Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering follower state: follower="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Follower]" leader=
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.847Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering follower state: follower="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Follower]" leader=
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.847Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering follower state: follower="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Follower]" leader=
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:05.847Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: entering follower state: follower="Node at 152a1577-3c8d-5394-90a1-61403f0ab0a2 [Follower]" leader=
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:05.847Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: entering follower state: follower="Node at 152a1577-3c8d-5394-90a1-61403f0ab0a2 [Follower]" leader=
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.913Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.913Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.913Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering candidate state: node="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Candidate]" term=2
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.913Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering candidate state: node="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Candidate]" term=2
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: election won: tally=2
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: election won: tally=2
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering leader state: leader="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Leader]"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering leader state: leader="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Leader]"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering candidate state: node="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Candidate]" term=2
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-152a1577-3c8d-5394-90a1-61403f0ab0a2
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering candidate state: node="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Candidate]" term=2
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: duplicate requestVote for same term: term=2
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: duplicate requestVote for same term: term=2
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering follower state: follower="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Follower]" leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:05.914Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering follower state: follower="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Follower]" leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-152a1577-3c8d-5394-90a1-61403f0ab0a2
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.915Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.915Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:05.915Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:05.915Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.914Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.916Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:05.916Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.022Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:05.916776693 +0000 UTC m=+14.068570831 (1 nodes), 69.282592ms from start of poll, 69.719808ms from cluster start. Timeout at 2024-01-08 10:38:06.022283973 +0000 UTC m=+14.174078121, 105.50729ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:06.022Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:05.916776693 +0000 UTC m=+14.068570831 (1 nodes), 69.282592ms from start of poll, 69.719808ms from cluster start. Timeout at 2024-01-08 10:38:06.022283973 +0000 UTC m=+14.174078121, 105.50729ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.029Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:06.029Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.142Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:06.030107391 +0000 UTC m=+14.181901519 (2 nodes), 4.564µs from start of poll, 183.050496ms from cluster start. Timeout at 2024-01-08 10:38:06.141883097 +0000 UTC m=+14.293677223, 111.775704ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:06.142Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:06.030107391 +0000 UTC m=+14.181901519 (2 nodes), 4.564µs from start of poll, 183.050496ms from cluster start. Timeout at 2024-01-08 10:38:06.141883097 +0000 UTC m=+14.293677223, 111.775704ms after stability
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.142Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: updating configuration: command=RemoveServer server-id=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 server-addr= servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.142Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: updating configuration: command=RemoveServer server-id=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 server-addr= servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2}]"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.143Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: removed peer, stopping replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 last-index=3
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.143Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: removed peer, stopping replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 last-index=3
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.143Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.143Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.201Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: heartbeat timeout reached, starting election: last-leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.201Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: heartbeat timeout reached, starting election: last-leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.201Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering candidate state: node="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Candidate]" term=3
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.201Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering candidate state: node="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Candidate]" term=3
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.202Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.202Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.202Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.202Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.249Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: updating configuration: command=AddStaging server-id=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 server-addr=17d6bb27-8a9d-7e78-e2e5-f808df48d144 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144}]"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.249Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: updating configuration: command=AddStaging server-id=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 server-addr=17d6bb27-8a9d-7e78-e2e5-f808df48d144 servers="[{Suffrage:Voter ID:server-c834ac43-aa1a-f77a-9f09-22589df94c1e Address:c834ac43-aa1a-f77a-9f09-22589df94c1e} {Suffrage:Voter ID:server-152a1577-3c8d-5394-90a1-61403f0ab0a2 Address:152a1577-3c8d-5394-90a1-61403f0ab0a2} {Suffrage:Voter ID:server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 Address:17d6bb27-8a9d-7e78-e2e5-f808df48d144}]"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [ERROR] server-c834ac43-aa1a-f77a-9f09-22589df94c1e: peer has newer term, stopping replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [ERROR] server-c834ac43-aa1a-f77a-9f09-22589df94c1e: peer has newer term, stopping replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering follower state: follower="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Follower]" leader=
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering follower state: follower="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Follower]" leader=
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.250Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:06.250Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.250Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.265Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: Election timeout reached, restarting election
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.265Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: Election timeout reached, restarting election
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.265Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering candidate state: node="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Candidate]" term=4
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.265Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering candidate state: node="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Candidate]" term=4
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.266Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.266Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: rejecting vote request since we have a leader: from=17d6bb27-8a9d-7e78-e2e5-f808df48d144 leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.266Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: rejecting vote request since our last index is greater: candidate=17d6bb27-8a9d-7e78-e2e5-f808df48d144 last-index=4 last-candidate-index=3
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.266Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: rejecting vote request since our last index is greater: candidate=17d6bb27-8a9d-7e78-e2e5-f808df48d144 last-index=4 last-candidate-index=3
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.325Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: heartbeat timeout reached, starting election: last-leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.325Z [WARN]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: heartbeat timeout reached, starting election: last-leader=c834ac43-aa1a-f77a-9f09-22589df94c1e
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.325Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: entering candidate state: node="Node at 152a1577-3c8d-5394-90a1-61403f0ab0a2 [Candidate]" term=3
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.325Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: entering candidate state: node="Node at 152a1577-3c8d-5394-90a1-61403f0ab0a2 [Candidate]" term=3
testLoggerAdapter verbose: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.326Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: entering follower state: follower="Node at 152a1577-3c8d-5394-90a1-61403f0ab0a2 [Follower]" leader=
    testing.go:149: server-152a1577-3c8d-5394-90a1-61403f0ab0a2: 2024-01-08T10:38:06.326Z [INFO]  server-152a1577-3c8d-5394-90a1-61403f0ab0a2: entering follower state: follower="Node at 152a1577-3c8d-5394-90a1-61403f0ab0a2 [Follower]" leader=
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.329Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.329Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.329Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering candidate state: node="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Candidate]" term=5
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.329Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering candidate state: node="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Candidate]" term=5
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.330Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering follower state: follower="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Follower]" leader=
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.330Z [INFO]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: entering follower state: follower="Node at 17d6bb27-8a9d-7e78-e2e5-f808df48d144 [Follower]" leader=
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: election won: tally=2
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: election won: tally=2
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering leader state: leader="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Leader]"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: entering leader state: leader="Node at c834ac43-aa1a-f77a-9f09-22589df94c1e [Leader]"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-152a1577-3c8d-5394-90a1-61403f0ab0a2
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-152a1577-3c8d-5394-90a1-61403f0ab0a2
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: added peer, starting replication: peer=server-17d6bb27-8a9d-7e78-e2e5-f808df48d144
testLoggerAdapter verbose: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.330Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: failed to get previous log: previous-index=4 last-index=3 error="log not found"
    testing.go:149: server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: 2024-01-08T10:38:06.330Z [WARN]  server-17d6bb27-8a9d-7e78-e2e5-f808df48d144: failed to get previous log: previous-index=4 last-index=3 error="log not found"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: appendEntries rejected, sending older logs: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}" next=4
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [WARN]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: appendEntries rejected, sending older logs: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}" next=4
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.330Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: pipelining replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.437Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:06.330523313 +0000 UTC m=+14.482317164 (1 nodes), 80.174364ms from start of poll, 483.466141ms from cluster start. Timeout at 2024-01-08 10:38:06.437805611 +0000 UTC m=+14.589599462, 107.282298ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:06.437Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:06.330523313 +0000 UTC m=+14.482317164 (1 nodes), 80.174364ms from start of poll, 483.466141ms from cluster start. Timeout at 2024-01-08 10:38:06.437805611 +0000 UTC m=+14.589599462, 107.282298ms after stability
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.438Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.438Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-17d6bb27-8a9d-7e78-e2e5-f808df48d144 17d6bb27-8a9d-7e78-e2e5-f808df48d144}"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.438Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.438Z [INFO]  server-c834ac43-aa1a-f77a-9f09-22589df94c1e: aborting pipeline replication: peer="{Voter server-152a1577-3c8d-5394-90a1-61403f0ab0a2 152a1577-3c8d-5394-90a1-61403f0ab0a2}"
testLoggerAdapter verbose: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.489Z [ERROR] server-c834ac43-aa1a-f77a-9f09-22589df94c1e: failed to heartbeat to: peer=152a1577-3c8d-5394-90a1-61403f0ab0a2 error="command timed out"
    testing.go:149: server-c834ac43-aa1a-f77a-9f09-22589df94c1e: 2024-01-08T10:38:06.489Z [ERROR] server-c834ac43-aa1a-f77a-9f09-22589df94c1e: failed to heartbeat to: peer=152a1577-3c8d-5394-90a1-61403f0ab0a2 error="command timed out"
--- PASS: TestRaft_ReJoinFollower (0.64s)
=== RUN   TestRaft_LeaderLeaseExpire
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.490Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 Address:e64ee3e1-da7e-b774-e4d8-a37090cc7040} {Suffrage:Voter ID:server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 Address:c8f0a95d-2a4d-e13f-48c8-f79845ea7082}]"
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.490Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 Address:e64ee3e1-da7e-b774-e4d8-a37090cc7040} {Suffrage:Voter ID:server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 Address:c8f0a95d-2a4d-e13f-48c8-f79845ea7082}]"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.490Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 Address:e64ee3e1-da7e-b774-e4d8-a37090cc7040} {Suffrage:Voter ID:server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 Address:c8f0a95d-2a4d-e13f-48c8-f79845ea7082}]"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.490Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 Address:e64ee3e1-da7e-b774-e4d8-a37090cc7040} {Suffrage:Voter ID:server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 Address:c8f0a95d-2a4d-e13f-48c8-f79845ea7082}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.490Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:06.490Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.490Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering follower state: follower="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Follower]" leader=
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.490Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering follower state: follower="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Follower]" leader=
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.491Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering follower state: follower="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Follower]" leader=
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.491Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering follower state: follower="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Follower]" leader=
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.573Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.573Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.573Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=2
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.573Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=2
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.574Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.574Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.574Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=2
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.574Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=2
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.574Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: duplicate requestVote for same term: term=2
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.574Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: duplicate requestVote for same term: term=2
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.575Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: duplicate requestVote for same term: term=2
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.575Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: duplicate requestVote for same term: term=2
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.633Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: Election timeout reached, restarting election
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.633Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: Election timeout reached, restarting election
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.633Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=3
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.633Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=3
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.634Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering follower state: follower="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Follower]" leader=
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.634Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering follower state: follower="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Follower]" leader=
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: election won: tally=2
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: election won: tally=2
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering leader state: leader="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Leader]"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering leader state: leader="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Leader]"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: added peer, starting replication: peer=server-e64ee3e1-da7e-b774-e4d8-a37090cc7040
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: added peer, starting replication: peer=server-e64ee3e1-da7e-b774-e4d8-a37090cc7040
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: pipelining replication: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.634Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: pipelining replication: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.742Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:06.634361716 +0000 UTC m=+14.786155567 (1 nodes), 143.437404ms from start of poll, 143.778646ms from cluster start. Timeout at 2024-01-08 10:38:06.742013685 +0000 UTC m=+14.893807849, 107.652282ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:06.742Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:06.634361716 +0000 UTC m=+14.786155567 (1 nodes), 143.437404ms from start of poll, 143.778646ms from cluster start. Timeout at 2024-01-08 10:38:06.742013685 +0000 UTC m=+14.893807849, 107.652282ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.754Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:06.754Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.860Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:06.754471722 +0000 UTC m=+14.906265828 (1 nodes), 3.408µs from start of poll, 263.888907ms from cluster start. Timeout at 2024-01-08 10:38:06.859986759 +0000 UTC m=+15.011780896, 105.515068ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:06.860Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:06.754471722 +0000 UTC m=+14.906265828 (1 nodes), 3.408µs from start of poll, 263.888907ms from cluster start. Timeout at 2024-01-08 10:38:06.859986759 +0000 UTC m=+15.011780896, 105.515068ms after stability
    raft_test.go:1492: [INFO] Disconnecting Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Follower]
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.865Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to pipeline appendEntries: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="append pipeline closed"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.865Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to pipeline appendEntries: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="append pipeline closed"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.865Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to heartbeat to: peer=e64ee3e1-da7e-b774-e4d8-a37090cc7040 error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.865Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to heartbeat to: peer=e64ee3e1-da7e-b774-e4d8-a37090cc7040 error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.866Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: aborting pipeline replication: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.866Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: aborting pipeline replication: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.882Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to appendEntries to: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.882Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to appendEntries to: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.895Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to heartbeat to: peer=e64ee3e1-da7e-b774-e4d8-a37090cc7040 error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.895Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to heartbeat to: peer=e64ee3e1-da7e-b774-e4d8-a37090cc7040 error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.906Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to appendEntries to: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.906Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to appendEntries to: peer="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.909Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to contact: server-id=server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 time=51.345804ms
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.909Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to contact: server-id=server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 time=51.345804ms
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.910Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to contact quorum of nodes, stepping down
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.910Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to contact quorum of nodes, stepping down
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.910Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering follower state: follower="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Follower]" leader=
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.910Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering follower state: follower="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Follower]" leader=
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:06.910Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:06.910Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.954Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: heartbeat timeout reached, starting election: last-leader=c8f0a95d-2a4d-e13f-48c8-f79845ea7082
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.954Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: heartbeat timeout reached, starting election: last-leader=c8f0a95d-2a4d-e13f-48c8-f79845ea7082
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.954Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=4
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.954Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=4
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.954Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:06.954Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.994Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.994Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.994Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=4
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.994Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=4
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.994Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to make requestVote RPC: target="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:06.994Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to make requestVote RPC: target="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.022Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: Election timeout reached, restarting election
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.022Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: Election timeout reached, restarting election
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.022Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=5
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.022Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=5
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.022Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.022Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.089Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: Election timeout reached, restarting election
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.089Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: Election timeout reached, restarting election
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.089Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=5
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.089Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=5
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.090Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to make requestVote RPC: target="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.090Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to make requestVote RPC: target="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.101Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:06.995038847 +0000 UTC m=+15.146833021 (0 nodes), 84.87905ms from start of poll, 504.4561ms from cluster start. Timeout at 2024-01-08 10:38:07.101330799 +0000 UTC m=+15.253124934, 106.291913ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:07.101Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:06.995038847 +0000 UTC m=+15.146833021 (0 nodes), 84.87905ms from start of poll, 504.4561ms from cluster start. Timeout at 2024-01-08 10:38:07.101330799 +0000 UTC m=+15.253124934, 106.291913ms after stability
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.117Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: Election timeout reached, restarting election
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.117Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: Election timeout reached, restarting election
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.117Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=6
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.117Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=6
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.118Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.118Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.161Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: Election timeout reached, restarting election
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.161Z [WARN]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: Election timeout reached, restarting election
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.162Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=6
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.162Z [INFO]  server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: entering candidate state: node="Node at c8f0a95d-2a4d-e13f-48c8-f79845ea7082 [Candidate]" term=6
testLoggerAdapter verbose: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.162Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to make requestVote RPC: target="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
    testing.go:149: server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: 2024-01-08T10:38:07.162Z [ERROR] server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082: failed to make requestVote RPC: target="{Voter server-e64ee3e1-da7e-b774-e4d8-a37090cc7040 e64ee3e1-da7e-b774-e4d8-a37090cc7040}" error="failed to connect to peer: e64ee3e1-da7e-b774-e4d8-a37090cc7040"
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.190Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: Election timeout reached, restarting election
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.190Z [WARN]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: Election timeout reached, restarting election
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.191Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=7
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.191Z [INFO]  server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: entering candidate state: node="Node at e64ee3e1-da7e-b774-e4d8-a37090cc7040 [Candidate]" term=7
testLoggerAdapter verbose: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.191Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
    testing.go:149: server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: 2024-01-08T10:38:07.191Z [ERROR] server-e64ee3e1-da7e-b774-e4d8-a37090cc7040: failed to make requestVote RPC: target="{Voter server-c8f0a95d-2a4d-e13f-48c8-f79845ea7082 c8f0a95d-2a4d-e13f-48c8-f79845ea7082}" error="failed to connect to peer: c8f0a95d-2a4d-e13f-48c8-f79845ea7082"
--- PASS: TestRaft_LeaderLeaseExpire (0.72s)
=== RUN   TestRaft_Barrier
testLoggerAdapter verbose: server-a909e40e-0ad7-4866-c074-93b668e02bc7: 2024-01-08T10:38:07.212Z [INFO]  server-a909e40e-0ad7-4866-c074-93b668e02bc7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a909e40e-0ad7-4866-c074-93b668e02bc7 Address:a909e40e-0ad7-4866-c074-93b668e02bc7} {Suffrage:Voter ID:server-dd763fb6-a83d-f468-9c58-020ecef0822a Address:dd763fb6-a83d-f468-9c58-020ecef0822a} {Suffrage:Voter ID:server-c89db727-b52f-cb03-f79b-a3cf61045b7e Address:c89db727-b52f-cb03-f79b-a3cf61045b7e}]"
    testing.go:149: server-a909e40e-0ad7-4866-c074-93b668e02bc7: 2024-01-08T10:38:07.212Z [INFO]  server-a909e40e-0ad7-4866-c074-93b668e02bc7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a909e40e-0ad7-4866-c074-93b668e02bc7 Address:a909e40e-0ad7-4866-c074-93b668e02bc7} {Suffrage:Voter ID:server-dd763fb6-a83d-f468-9c58-020ecef0822a Address:dd763fb6-a83d-f468-9c58-020ecef0822a} {Suffrage:Voter ID:server-c89db727-b52f-cb03-f79b-a3cf61045b7e Address:c89db727-b52f-cb03-f79b-a3cf61045b7e}]"
testLoggerAdapter verbose: server-a909e40e-0ad7-4866-c074-93b668e02bc7: 2024-01-08T10:38:07.213Z [INFO]  server-a909e40e-0ad7-4866-c074-93b668e02bc7: entering follower state: follower="Node at a909e40e-0ad7-4866-c074-93b668e02bc7 [Follower]" leader=
    testing.go:149: server-a909e40e-0ad7-4866-c074-93b668e02bc7: 2024-01-08T10:38:07.213Z [INFO]  server-a909e40e-0ad7-4866-c074-93b668e02bc7: entering follower state: follower="Node at a909e40e-0ad7-4866-c074-93b668e02bc7 [Follower]" leader=
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.214Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a909e40e-0ad7-4866-c074-93b668e02bc7 Address:a909e40e-0ad7-4866-c074-93b668e02bc7} {Suffrage:Voter ID:server-dd763fb6-a83d-f468-9c58-020ecef0822a Address:dd763fb6-a83d-f468-9c58-020ecef0822a} {Suffrage:Voter ID:server-c89db727-b52f-cb03-f79b-a3cf61045b7e Address:c89db727-b52f-cb03-f79b-a3cf61045b7e}]"
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.214Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a909e40e-0ad7-4866-c074-93b668e02bc7 Address:a909e40e-0ad7-4866-c074-93b668e02bc7} {Suffrage:Voter ID:server-dd763fb6-a83d-f468-9c58-020ecef0822a Address:dd763fb6-a83d-f468-9c58-020ecef0822a} {Suffrage:Voter ID:server-c89db727-b52f-cb03-f79b-a3cf61045b7e Address:c89db727-b52f-cb03-f79b-a3cf61045b7e}]"
testLoggerAdapter verbose: server-c89db727-b52f-cb03-f79b-a3cf61045b7e: 2024-01-08T10:38:07.214Z [INFO]  server-c89db727-b52f-cb03-f79b-a3cf61045b7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a909e40e-0ad7-4866-c074-93b668e02bc7 Address:a909e40e-0ad7-4866-c074-93b668e02bc7} {Suffrage:Voter ID:server-dd763fb6-a83d-f468-9c58-020ecef0822a Address:dd763fb6-a83d-f468-9c58-020ecef0822a} {Suffrage:Voter ID:server-c89db727-b52f-cb03-f79b-a3cf61045b7e Address:c89db727-b52f-cb03-f79b-a3cf61045b7e}]"
    testing.go:149: server-c89db727-b52f-cb03-f79b-a3cf61045b7e: 2024-01-08T10:38:07.214Z [INFO]  server-c89db727-b52f-cb03-f79b-a3cf61045b7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a909e40e-0ad7-4866-c074-93b668e02bc7 Address:a909e40e-0ad7-4866-c074-93b668e02bc7} {Suffrage:Voter ID:server-dd763fb6-a83d-f468-9c58-020ecef0822a Address:dd763fb6-a83d-f468-9c58-020ecef0822a} {Suffrage:Voter ID:server-c89db727-b52f-cb03-f79b-a3cf61045b7e Address:c89db727-b52f-cb03-f79b-a3cf61045b7e}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.215Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:07.215Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.215Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: entering follower state: follower="Node at dd763fb6-a83d-f468-9c58-020ecef0822a [Follower]" leader=
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.215Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: entering follower state: follower="Node at dd763fb6-a83d-f468-9c58-020ecef0822a [Follower]" leader=
testLoggerAdapter verbose: server-c89db727-b52f-cb03-f79b-a3cf61045b7e: 2024-01-08T10:38:07.215Z [INFO]  server-c89db727-b52f-cb03-f79b-a3cf61045b7e: entering follower state: follower="Node at c89db727-b52f-cb03-f79b-a3cf61045b7e [Follower]" leader=
    testing.go:149: server-c89db727-b52f-cb03-f79b-a3cf61045b7e: 2024-01-08T10:38:07.215Z [INFO]  server-c89db727-b52f-cb03-f79b-a3cf61045b7e: entering follower state: follower="Node at c89db727-b52f-cb03-f79b-a3cf61045b7e [Follower]" leader=
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.282Z [WARN]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.282Z [WARN]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.282Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: entering candidate state: node="Node at dd763fb6-a83d-f468-9c58-020ecef0822a [Candidate]" term=2
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.282Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: entering candidate state: node="Node at dd763fb6-a83d-f468-9c58-020ecef0822a [Candidate]" term=2
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: election won: tally=2
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: election won: tally=2
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: entering leader state: leader="Node at dd763fb6-a83d-f468-9c58-020ecef0822a [Leader]"
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: entering leader state: leader="Node at dd763fb6-a83d-f468-9c58-020ecef0822a [Leader]"
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: added peer, starting replication: peer=server-a909e40e-0ad7-4866-c074-93b668e02bc7
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: added peer, starting replication: peer=server-a909e40e-0ad7-4866-c074-93b668e02bc7
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: added peer, starting replication: peer=server-c89db727-b52f-cb03-f79b-a3cf61045b7e
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.283Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: added peer, starting replication: peer=server-c89db727-b52f-cb03-f79b-a3cf61045b7e
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.284Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: pipelining replication: peer="{Voter server-c89db727-b52f-cb03-f79b-a3cf61045b7e c89db727-b52f-cb03-f79b-a3cf61045b7e}"
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.284Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: pipelining replication: peer="{Voter server-c89db727-b52f-cb03-f79b-a3cf61045b7e c89db727-b52f-cb03-f79b-a3cf61045b7e}"
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.285Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: pipelining replication: peer="{Voter server-a909e40e-0ad7-4866-c074-93b668e02bc7 a909e40e-0ad7-4866-c074-93b668e02bc7}"
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.285Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: pipelining replication: peer="{Voter server-a909e40e-0ad7-4866-c074-93b668e02bc7 a909e40e-0ad7-4866-c074-93b668e02bc7}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.390Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.284767458 +0000 UTC m=+15.436561592 (1 nodes), 69.444613ms from start of poll, 72.771043ms from cluster start. Timeout at 2024-01-08 10:38:07.390144979 +0000 UTC m=+15.541939092, 105.3775ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:07.390Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.284767458 +0000 UTC m=+15.436561592 (1 nodes), 69.444613ms from start of poll, 72.771043ms from cluster start. Timeout at 2024-01-08 10:38:07.390144979 +0000 UTC m=+15.541939092, 105.3775ms after stability
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.422Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: aborting pipeline replication: peer="{Voter server-c89db727-b52f-cb03-f79b-a3cf61045b7e c89db727-b52f-cb03-f79b-a3cf61045b7e}"
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.422Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: aborting pipeline replication: peer="{Voter server-c89db727-b52f-cb03-f79b-a3cf61045b7e c89db727-b52f-cb03-f79b-a3cf61045b7e}"
testLoggerAdapter verbose: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.423Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: aborting pipeline replication: peer="{Voter server-a909e40e-0ad7-4866-c074-93b668e02bc7 a909e40e-0ad7-4866-c074-93b668e02bc7}"
    testing.go:149: server-dd763fb6-a83d-f468-9c58-020ecef0822a: 2024-01-08T10:38:07.423Z [INFO]  server-dd763fb6-a83d-f468-9c58-020ecef0822a: aborting pipeline replication: peer="{Voter server-a909e40e-0ad7-4866-c074-93b668e02bc7 a909e40e-0ad7-4866-c074-93b668e02bc7}"
--- PASS: TestRaft_Barrier (0.21s)
=== RUN   TestRaft_VerifyLeader
testLoggerAdapter verbose: server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: 2024-01-08T10:38:07.426Z [INFO]  server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 Address:9a06e9f4-6df1-37c6-af33-ae7eae76ca73} {Suffrage:Voter ID:server-d2db0cf3-4585-1671-3512-bb0d14f751aa Address:d2db0cf3-4585-1671-3512-bb0d14f751aa} {Suffrage:Voter ID:server-7fe15e95-47aa-4c49-bff3-8188cc68453c Address:7fe15e95-47aa-4c49-bff3-8188cc68453c}]"
    testing.go:149: server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: 2024-01-08T10:38:07.426Z [INFO]  server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 Address:9a06e9f4-6df1-37c6-af33-ae7eae76ca73} {Suffrage:Voter ID:server-d2db0cf3-4585-1671-3512-bb0d14f751aa Address:d2db0cf3-4585-1671-3512-bb0d14f751aa} {Suffrage:Voter ID:server-7fe15e95-47aa-4c49-bff3-8188cc68453c Address:7fe15e95-47aa-4c49-bff3-8188cc68453c}]"
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.427Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 Address:9a06e9f4-6df1-37c6-af33-ae7eae76ca73} {Suffrage:Voter ID:server-d2db0cf3-4585-1671-3512-bb0d14f751aa Address:d2db0cf3-4585-1671-3512-bb0d14f751aa} {Suffrage:Voter ID:server-7fe15e95-47aa-4c49-bff3-8188cc68453c Address:7fe15e95-47aa-4c49-bff3-8188cc68453c}]"
testLoggerAdapter verbose: server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: 2024-01-08T10:38:07.427Z [INFO]  server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: entering follower state: follower="Node at 9a06e9f4-6df1-37c6-af33-ae7eae76ca73 [Follower]" leader=
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.427Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 Address:9a06e9f4-6df1-37c6-af33-ae7eae76ca73} {Suffrage:Voter ID:server-d2db0cf3-4585-1671-3512-bb0d14f751aa Address:d2db0cf3-4585-1671-3512-bb0d14f751aa} {Suffrage:Voter ID:server-7fe15e95-47aa-4c49-bff3-8188cc68453c Address:7fe15e95-47aa-4c49-bff3-8188cc68453c}]"
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.428Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: entering follower state: follower="Node at d2db0cf3-4585-1671-3512-bb0d14f751aa [Follower]" leader=
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.428Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: entering follower state: follower="Node at d2db0cf3-4585-1671-3512-bb0d14f751aa [Follower]" leader=
    testing.go:149: server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: 2024-01-08T10:38:07.427Z [INFO]  server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73: entering follower state: follower="Node at 9a06e9f4-6df1-37c6-af33-ae7eae76ca73 [Follower]" leader=
testLoggerAdapter verbose: server-7fe15e95-47aa-4c49-bff3-8188cc68453c: 2024-01-08T10:38:07.428Z [INFO]  server-7fe15e95-47aa-4c49-bff3-8188cc68453c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 Address:9a06e9f4-6df1-37c6-af33-ae7eae76ca73} {Suffrage:Voter ID:server-d2db0cf3-4585-1671-3512-bb0d14f751aa Address:d2db0cf3-4585-1671-3512-bb0d14f751aa} {Suffrage:Voter ID:server-7fe15e95-47aa-4c49-bff3-8188cc68453c Address:7fe15e95-47aa-4c49-bff3-8188cc68453c}]"
    testing.go:149: server-7fe15e95-47aa-4c49-bff3-8188cc68453c: 2024-01-08T10:38:07.428Z [INFO]  server-7fe15e95-47aa-4c49-bff3-8188cc68453c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 Address:9a06e9f4-6df1-37c6-af33-ae7eae76ca73} {Suffrage:Voter ID:server-d2db0cf3-4585-1671-3512-bb0d14f751aa Address:d2db0cf3-4585-1671-3512-bb0d14f751aa} {Suffrage:Voter ID:server-7fe15e95-47aa-4c49-bff3-8188cc68453c Address:7fe15e95-47aa-4c49-bff3-8188cc68453c}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.429Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:07.429Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-7fe15e95-47aa-4c49-bff3-8188cc68453c: 2024-01-08T10:38:07.429Z [INFO]  server-7fe15e95-47aa-4c49-bff3-8188cc68453c: entering follower state: follower="Node at 7fe15e95-47aa-4c49-bff3-8188cc68453c [Follower]" leader=
    testing.go:149: server-7fe15e95-47aa-4c49-bff3-8188cc68453c: 2024-01-08T10:38:07.429Z [INFO]  server-7fe15e95-47aa-4c49-bff3-8188cc68453c: entering follower state: follower="Node at 7fe15e95-47aa-4c49-bff3-8188cc68453c [Follower]" leader=
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.481Z [WARN]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.481Z [WARN]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.481Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: entering candidate state: node="Node at d2db0cf3-4585-1671-3512-bb0d14f751aa [Candidate]" term=2
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.481Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: entering candidate state: node="Node at d2db0cf3-4585-1671-3512-bb0d14f751aa [Candidate]" term=2
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: election won: tally=2
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: election won: tally=2
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: entering leader state: leader="Node at d2db0cf3-4585-1671-3512-bb0d14f751aa [Leader]"
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: entering leader state: leader="Node at d2db0cf3-4585-1671-3512-bb0d14f751aa [Leader]"
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: added peer, starting replication: peer=server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: added peer, starting replication: peer=server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: added peer, starting replication: peer=server-7fe15e95-47aa-4c49-bff3-8188cc68453c
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.482Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: added peer, starting replication: peer=server-7fe15e95-47aa-4c49-bff3-8188cc68453c
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.483Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: pipelining replication: peer="{Voter server-7fe15e95-47aa-4c49-bff3-8188cc68453c 7fe15e95-47aa-4c49-bff3-8188cc68453c}"
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.483Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: pipelining replication: peer="{Voter server-7fe15e95-47aa-4c49-bff3-8188cc68453c 7fe15e95-47aa-4c49-bff3-8188cc68453c}"
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.483Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: pipelining replication: peer="{Voter server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 9a06e9f4-6df1-37c6-af33-ae7eae76ca73}"
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.483Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: pipelining replication: peer="{Voter server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 9a06e9f4-6df1-37c6-af33-ae7eae76ca73}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.588Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.482482075 +0000 UTC m=+15.634276250 (1 nodes), 53.258956ms from start of poll, 56.071562ms from cluster start. Timeout at 2024-01-08 10:38:07.588140557 +0000 UTC m=+15.739934694, 105.658444ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:07.588Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.482482075 +0000 UTC m=+15.634276250 (1 nodes), 53.258956ms from start of poll, 56.071562ms from cluster start. Timeout at 2024-01-08 10:38:07.588140557 +0000 UTC m=+15.739934694, 105.658444ms after stability
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.592Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: aborting pipeline replication: peer="{Voter server-7fe15e95-47aa-4c49-bff3-8188cc68453c 7fe15e95-47aa-4c49-bff3-8188cc68453c}"
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.592Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: aborting pipeline replication: peer="{Voter server-7fe15e95-47aa-4c49-bff3-8188cc68453c 7fe15e95-47aa-4c49-bff3-8188cc68453c}"
testLoggerAdapter verbose: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.592Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: aborting pipeline replication: peer="{Voter server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 9a06e9f4-6df1-37c6-af33-ae7eae76ca73}"
    testing.go:149: server-d2db0cf3-4585-1671-3512-bb0d14f751aa: 2024-01-08T10:38:07.592Z [INFO]  server-d2db0cf3-4585-1671-3512-bb0d14f751aa: aborting pipeline replication: peer="{Voter server-9a06e9f4-6df1-37c6-af33-ae7eae76ca73 9a06e9f4-6df1-37c6-af33-ae7eae76ca73}"
--- PASS: TestRaft_VerifyLeader (0.17s)
=== RUN   TestRaft_VerifyLeader_Single
testLoggerAdapter verbose: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.599Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 Address:f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54}]"
    testing.go:149: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.599Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 Address:f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.599Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:07.599Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.600Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: entering follower state: follower="Node at f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 [Follower]" leader=
    testing.go:149: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.600Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: entering follower state: follower="Node at f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 [Follower]" leader=
testLoggerAdapter verbose: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.674Z [WARN]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.674Z [WARN]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.674Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: entering candidate state: node="Node at f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 [Candidate]" term=2
    testing.go:149: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.674Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: entering candidate state: node="Node at f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 [Candidate]" term=2
testLoggerAdapter verbose: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.675Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: election won: tally=1
    testing.go:149: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.675Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: election won: tally=1
testLoggerAdapter verbose: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.675Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: entering leader state: leader="Node at f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 [Leader]"
    testing.go:149: server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: 2024-01-08T10:38:07.675Z [INFO]  server-f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54: entering leader state: leader="Node at f3d4c3d2-4a15-4249-ecbc-e0aabf3ccb54 [Leader]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.781Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.67561028 +0000 UTC m=+15.827404414 (1 nodes), 75.476318ms from start of poll, 76.51451ms from cluster start. Timeout at 2024-01-08 10:38:07.781458445 +0000 UTC m=+15.933252579, 105.848165ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:07.781Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.67561028 +0000 UTC m=+15.827404414 (1 nodes), 75.476318ms from start of poll, 76.51451ms from cluster start. Timeout at 2024-01-08 10:38:07.781458445 +0000 UTC m=+15.933252579, 105.848165ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.18s)
=== RUN   TestRaft_VerifyLeader_Fail
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.784Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c69e841-fb9a-d527-764c-ea83335c4704 Address:9c69e841-fb9a-d527-764c-ea83335c4704} {Suffrage:Voter ID:server-b825f943-f310-52c0-7aa4-ae50180a9fee Address:b825f943-f310-52c0-7aa4-ae50180a9fee}]"
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.784Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c69e841-fb9a-d527-764c-ea83335c4704 Address:9c69e841-fb9a-d527-764c-ea83335c4704} {Suffrage:Voter ID:server-b825f943-f310-52c0-7aa4-ae50180a9fee Address:b825f943-f310-52c0-7aa4-ae50180a9fee}]"
testLoggerAdapter verbose: server-b825f943-f310-52c0-7aa4-ae50180a9fee: 2024-01-08T10:38:07.785Z [INFO]  server-b825f943-f310-52c0-7aa4-ae50180a9fee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c69e841-fb9a-d527-764c-ea83335c4704 Address:9c69e841-fb9a-d527-764c-ea83335c4704} {Suffrage:Voter ID:server-b825f943-f310-52c0-7aa4-ae50180a9fee Address:b825f943-f310-52c0-7aa4-ae50180a9fee}]"
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.785Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering follower state: follower="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Follower]" leader=
    testing.go:149: server-b825f943-f310-52c0-7aa4-ae50180a9fee: 2024-01-08T10:38:07.785Z [INFO]  server-b825f943-f310-52c0-7aa4-ae50180a9fee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c69e841-fb9a-d527-764c-ea83335c4704 Address:9c69e841-fb9a-d527-764c-ea83335c4704} {Suffrage:Voter ID:server-b825f943-f310-52c0-7aa4-ae50180a9fee Address:b825f943-f310-52c0-7aa4-ae50180a9fee}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.786Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.785Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering follower state: follower="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Follower]" leader=
testLoggerAdapter verbose: server-b825f943-f310-52c0-7aa4-ae50180a9fee: 2024-01-08T10:38:07.786Z [INFO]  server-b825f943-f310-52c0-7aa4-ae50180a9fee: entering follower state: follower="Node at b825f943-f310-52c0-7aa4-ae50180a9fee [Follower]" leader=
    testing.go:149: server-b825f943-f310-52c0-7aa4-ae50180a9fee: 2024-01-08T10:38:07.786Z [INFO]  server-b825f943-f310-52c0-7aa4-ae50180a9fee: entering follower state: follower="Node at b825f943-f310-52c0-7aa4-ae50180a9fee [Follower]" leader=
    testing.go:149: cluster: 2024-01-08T10:38:07.786Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.869Z [WARN]  server-9c69e841-fb9a-d527-764c-ea83335c4704: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.869Z [WARN]  server-9c69e841-fb9a-d527-764c-ea83335c4704: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.869Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering candidate state: node="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Candidate]" term=2
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.869Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering candidate state: node="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Candidate]" term=2
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.870Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: election won: tally=2
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.870Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: election won: tally=2
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.870Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering leader state: leader="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Leader]"
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.870Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering leader state: leader="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Leader]"
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.870Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: added peer, starting replication: peer=server-b825f943-f310-52c0-7aa4-ae50180a9fee
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.870Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: added peer, starting replication: peer=server-b825f943-f310-52c0-7aa4-ae50180a9fee
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.871Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: pipelining replication: peer="{Voter server-b825f943-f310-52c0-7aa4-ae50180a9fee b825f943-f310-52c0-7aa4-ae50180a9fee}"
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:07.871Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: pipelining replication: peer="{Voter server-b825f943-f310-52c0-7aa4-ae50180a9fee b825f943-f310-52c0-7aa4-ae50180a9fee}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.976Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.871354104 +0000 UTC m=+16.023148263 (1 nodes), 84.256402ms from start of poll, 87.073938ms from cluster start. Timeout at 2024-01-08 10:38:07.976631059 +0000 UTC m=+16.128425197, 105.276934ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:07.976Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:07.871354104 +0000 UTC m=+16.023148263 (1 nodes), 84.256402ms from start of poll, 87.073938ms from cluster start. Timeout at 2024-01-08 10:38:07.976631059 +0000 UTC m=+16.128425197, 105.276934ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:07.977Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:07.977Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.083Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:07.977375547 +0000 UTC m=+16.129169671 (1 nodes), 3.079µs from start of poll, 193.095346ms from cluster start. Timeout at 2024-01-08 10:38:08.082610748 +0000 UTC m=+16.234404897, 105.235226ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:08.083Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:07.977375547 +0000 UTC m=+16.129169671 (1 nodes), 3.079µs from start of poll, 193.095346ms from cluster start. Timeout at 2024-01-08 10:38:08.082610748 +0000 UTC m=+16.234404897, 105.235226ms after stability
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:08.083Z [WARN]  server-9c69e841-fb9a-d527-764c-ea83335c4704: new leader elected, stepping down
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:08.083Z [WARN]  server-9c69e841-fb9a-d527-764c-ea83335c4704: new leader elected, stepping down
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:08.083Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering follower state: follower="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Follower]" leader=
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:08.083Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: entering follower state: follower="Node at 9c69e841-fb9a-d527-764c-ea83335c4704 [Follower]" leader=
testLoggerAdapter verbose: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:08.083Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: aborting pipeline replication: peer="{Voter server-b825f943-f310-52c0-7aa4-ae50180a9fee b825f943-f310-52c0-7aa4-ae50180a9fee}"
    testing.go:149: server-9c69e841-fb9a-d527-764c-ea83335c4704: 2024-01-08T10:38:08.083Z [INFO]  server-9c69e841-fb9a-d527-764c-ea83335c4704: aborting pipeline replication: peer="{Voter server-b825f943-f310-52c0-7aa4-ae50180a9fee b825f943-f310-52c0-7aa4-ae50180a9fee}"
--- PASS: TestRaft_VerifyLeader_Fail (0.30s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.087Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc7e76de-322a-da98-37be-8643c949617b Address:fc7e76de-322a-da98-37be-8643c949617b} {Suffrage:Voter ID:server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 Address:dcab4873-8ebb-f008-f54c-f6bf2b6d6a48} {Suffrage:Voter ID:server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 Address:576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}]"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.087Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc7e76de-322a-da98-37be-8643c949617b Address:fc7e76de-322a-da98-37be-8643c949617b} {Suffrage:Voter ID:server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 Address:dcab4873-8ebb-f008-f54c-f6bf2b6d6a48} {Suffrage:Voter ID:server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 Address:576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}]"
testLoggerAdapter verbose: server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: 2024-01-08T10:38:08.088Z [INFO]  server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc7e76de-322a-da98-37be-8643c949617b Address:fc7e76de-322a-da98-37be-8643c949617b} {Suffrage:Voter ID:server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 Address:dcab4873-8ebb-f008-f54c-f6bf2b6d6a48} {Suffrage:Voter ID:server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 Address:576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}]"
    testing.go:149: server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: 2024-01-08T10:38:08.088Z [INFO]  server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc7e76de-322a-da98-37be-8643c949617b Address:fc7e76de-322a-da98-37be-8643c949617b} {Suffrage:Voter ID:server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 Address:dcab4873-8ebb-f008-f54c-f6bf2b6d6a48} {Suffrage:Voter ID:server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 Address:576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}]"
testLoggerAdapter verbose: server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: 2024-01-08T10:38:08.089Z [INFO]  server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc7e76de-322a-da98-37be-8643c949617b Address:fc7e76de-322a-da98-37be-8643c949617b} {Suffrage:Voter ID:server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 Address:dcab4873-8ebb-f008-f54c-f6bf2b6d6a48} {Suffrage:Voter ID:server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 Address:576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}]"
    testing.go:149: server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: 2024-01-08T10:38:08.089Z [INFO]  server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc7e76de-322a-da98-37be-8643c949617b Address:fc7e76de-322a-da98-37be-8643c949617b} {Suffrage:Voter ID:server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 Address:dcab4873-8ebb-f008-f54c-f6bf2b6d6a48} {Suffrage:Voter ID:server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 Address:576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.089Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:08.089Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.089Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: entering follower state: follower="Node at fc7e76de-322a-da98-37be-8643c949617b [Follower]" leader=
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.089Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: entering follower state: follower="Node at fc7e76de-322a-da98-37be-8643c949617b [Follower]" leader=
testLoggerAdapter verbose: server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: 2024-01-08T10:38:08.090Z [INFO]  server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: entering follower state: follower="Node at 576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 [Follower]" leader=
    testing.go:149: server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: 2024-01-08T10:38:08.090Z [INFO]  server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2: entering follower state: follower="Node at 576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 [Follower]" leader=
testLoggerAdapter verbose: server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: 2024-01-08T10:38:08.091Z [INFO]  server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: entering follower state: follower="Node at dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 [Follower]" leader=
    testing.go:149: server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: 2024-01-08T10:38:08.091Z [INFO]  server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48: entering follower state: follower="Node at dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 [Follower]" leader=
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.152Z [WARN]  server-fc7e76de-322a-da98-37be-8643c949617b: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.152Z [WARN]  server-fc7e76de-322a-da98-37be-8643c949617b: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: entering candidate state: node="Node at fc7e76de-322a-da98-37be-8643c949617b [Candidate]" term=2
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: entering candidate state: node="Node at fc7e76de-322a-da98-37be-8643c949617b [Candidate]" term=2
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: election won: tally=2
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: election won: tally=2
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: entering leader state: leader="Node at fc7e76de-322a-da98-37be-8643c949617b [Leader]"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: entering leader state: leader="Node at fc7e76de-322a-da98-37be-8643c949617b [Leader]"
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: added peer, starting replication: peer=server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.153Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: added peer, starting replication: peer=server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.154Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: added peer, starting replication: peer=server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.154Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: added peer, starting replication: peer=server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.154Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: pipelining replication: peer="{Voter server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.154Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: pipelining replication: peer="{Voter server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}"
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.155Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: pipelining replication: peer="{Voter server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 dcab4873-8ebb-f008-f54c-f6bf2b6d6a48}"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.155Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: pipelining replication: peer="{Voter server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 dcab4873-8ebb-f008-f54c-f6bf2b6d6a48}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.262Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:08.155626558 +0000 UTC m=+16.307420696 (1 nodes), 66.243774ms from start of poll, 68.22877ms from cluster start. Timeout at 2024-01-08 10:38:08.262253569 +0000 UTC m=+16.414047693, 106.626997ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:08.262Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:08.155626558 +0000 UTC m=+16.307420696 (1 nodes), 66.243774ms from start of poll, 68.22877ms from cluster start. Timeout at 2024-01-08 10:38:08.262253569 +0000 UTC m=+16.414047693, 106.626997ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.268Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:08.268Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.375Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:08.268699729 +0000 UTC m=+16.420493837 (2 nodes), 4.705µs from start of poll, 181.301911ms from cluster start. Timeout at 2024-01-08 10:38:08.37553774 +0000 UTC m=+16.527331862, 106.838025ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:08.375Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:08.268699729 +0000 UTC m=+16.420493837 (2 nodes), 4.705µs from start of poll, 181.301911ms from cluster start. Timeout at 2024-01-08 10:38:08.37553774 +0000 UTC m=+16.527331862, 106.838025ms after stability
    raft_test.go:1647: [INFO] Disconnecting Node at dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 [Follower]
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.376Z [ERROR] server-fc7e76de-322a-da98-37be-8643c949617b: failed to heartbeat to: peer=dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 error="failed to connect to peer: dcab4873-8ebb-f008-f54c-f6bf2b6d6a48"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.376Z [ERROR] server-fc7e76de-322a-da98-37be-8643c949617b: failed to heartbeat to: peer=dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 error="failed to connect to peer: dcab4873-8ebb-f008-f54c-f6bf2b6d6a48"
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.377Z [ERROR] server-fc7e76de-322a-da98-37be-8643c949617b: failed to pipeline appendEntries: peer="{Voter server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 dcab4873-8ebb-f008-f54c-f6bf2b6d6a48}" error="append pipeline closed"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.377Z [ERROR] server-fc7e76de-322a-da98-37be-8643c949617b: failed to pipeline appendEntries: peer="{Voter server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 dcab4873-8ebb-f008-f54c-f6bf2b6d6a48}" error="append pipeline closed"
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.377Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: aborting pipeline replication: peer="{Voter server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 dcab4873-8ebb-f008-f54c-f6bf2b6d6a48}"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.377Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: aborting pipeline replication: peer="{Voter server-dcab4873-8ebb-f008-f54c-f6bf2b6d6a48 dcab4873-8ebb-f008-f54c-f6bf2b6d6a48}"
testLoggerAdapter verbose: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.377Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: aborting pipeline replication: peer="{Voter server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}"
    testing.go:149: server-fc7e76de-322a-da98-37be-8643c949617b: 2024-01-08T10:38:08.377Z [INFO]  server-fc7e76de-322a-da98-37be-8643c949617b: aborting pipeline replication: peer="{Voter server-576e3b92-8e15-2a9a-62a1-40b5ab4b5da2 576e3b92-8e15-2a9a-62a1-40b5ab4b5da2}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s)
=== RUN   TestRaft_StartAsLeader
testLoggerAdapter verbose: server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: 2024-01-08T10:38:08.381Z [INFO]  server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06783dd9-7d4f-27a4-4741-f1635ea4f74f Address:06783dd9-7d4f-27a4-4741-f1635ea4f74f}]"
    testing.go:149: server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: 2024-01-08T10:38:08.381Z [INFO]  server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06783dd9-7d4f-27a4-4741-f1635ea4f74f Address:06783dd9-7d4f-27a4-4741-f1635ea4f74f}]"
testLoggerAdapter verbose: server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: 2024-01-08T10:38:08.381Z [INFO]  server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: entering leader state: leader="Node at 06783dd9-7d4f-27a4-4741-f1635ea4f74f [Leader]"
    testing.go:149: server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: 2024-01-08T10:38:08.381Z [INFO]  server-06783dd9-7d4f-27a4-4741-f1635ea4f74f: entering leader state: leader="Node at 06783dd9-7d4f-27a4-4741-f1635ea4f74f [Leader]"
--- PASS: TestRaft_StartAsLeader (0.00s)
=== RUN   TestRaft_NotifyCh
testLoggerAdapter verbose: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.384Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-589315db-85e9-7634-7828-e72f469725f6 Address:589315db-85e9-7634-7828-e72f469725f6}]"
    testing.go:149: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.384Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-589315db-85e9-7634-7828-e72f469725f6 Address:589315db-85e9-7634-7828-e72f469725f6}]"
testLoggerAdapter verbose: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.385Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: entering follower state: follower="Node at 589315db-85e9-7634-7828-e72f469725f6 [Follower]" leader=
    testing.go:149: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.385Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: entering follower state: follower="Node at 589315db-85e9-7634-7828-e72f469725f6 [Follower]" leader=
testLoggerAdapter verbose: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [WARN]  server-589315db-85e9-7634-7828-e72f469725f6: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [WARN]  server-589315db-85e9-7634-7828-e72f469725f6: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: entering candidate state: node="Node at 589315db-85e9-7634-7828-e72f469725f6 [Candidate]" term=2
    testing.go:149: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: entering candidate state: node="Node at 589315db-85e9-7634-7828-e72f469725f6 [Candidate]" term=2
testLoggerAdapter verbose: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: election won: tally=1
    testing.go:149: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: election won: tally=1
testLoggerAdapter verbose: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: entering leader state: leader="Node at 589315db-85e9-7634-7828-e72f469725f6 [Leader]"
    testing.go:149: server-589315db-85e9-7634-7828-e72f469725f6: 2024-01-08T10:38:08.476Z [INFO]  server-589315db-85e9-7634-7828-e72f469725f6: entering leader state: leader="Node at 589315db-85e9-7634-7828-e72f469725f6 [Leader]"
--- PASS: TestRaft_NotifyCh (0.09s)
=== RUN   TestRaft_Voting
testLoggerAdapter verbose: server-81277984-cf6e-3452-2f85-a634a8926c27: 2024-01-08T10:38:08.480Z [INFO]  server-81277984-cf6e-3452-2f85-a634a8926c27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81277984-cf6e-3452-2f85-a634a8926c27 Address:81277984-cf6e-3452-2f85-a634a8926c27} {Suffrage:Voter ID:server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac Address:eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac} {Suffrage:Voter ID:server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f Address:57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}]"
    testing.go:149: server-81277984-cf6e-3452-2f85-a634a8926c27: 2024-01-08T10:38:08.480Z [INFO]  server-81277984-cf6e-3452-2f85-a634a8926c27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81277984-cf6e-3452-2f85-a634a8926c27 Address:81277984-cf6e-3452-2f85-a634a8926c27} {Suffrage:Voter ID:server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac Address:eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac} {Suffrage:Voter ID:server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f Address:57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}]"
testLoggerAdapter verbose: server-81277984-cf6e-3452-2f85-a634a8926c27: 2024-01-08T10:38:08.481Z [INFO]  server-81277984-cf6e-3452-2f85-a634a8926c27: entering follower state: follower="Node at 81277984-cf6e-3452-2f85-a634a8926c27 [Follower]" leader=
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.482Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81277984-cf6e-3452-2f85-a634a8926c27 Address:81277984-cf6e-3452-2f85-a634a8926c27} {Suffrage:Voter ID:server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac Address:eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac} {Suffrage:Voter ID:server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f Address:57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}]"
    testing.go:149: server-81277984-cf6e-3452-2f85-a634a8926c27: 2024-01-08T10:38:08.481Z [INFO]  server-81277984-cf6e-3452-2f85-a634a8926c27: entering follower state: follower="Node at 81277984-cf6e-3452-2f85-a634a8926c27 [Follower]" leader=
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.482Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81277984-cf6e-3452-2f85-a634a8926c27 Address:81277984-cf6e-3452-2f85-a634a8926c27} {Suffrage:Voter ID:server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac Address:eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac} {Suffrage:Voter ID:server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f Address:57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}]"
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.482Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: entering follower state: follower="Node at eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac [Follower]" leader=
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.482Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: entering follower state: follower="Node at eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac [Follower]" leader=
testLoggerAdapter verbose: server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: 2024-01-08T10:38:08.483Z [INFO]  server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81277984-cf6e-3452-2f85-a634a8926c27 Address:81277984-cf6e-3452-2f85-a634a8926c27} {Suffrage:Voter ID:server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac Address:eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac} {Suffrage:Voter ID:server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f Address:57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}]"
    testing.go:149: server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: 2024-01-08T10:38:08.483Z [INFO]  server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81277984-cf6e-3452-2f85-a634a8926c27 Address:81277984-cf6e-3452-2f85-a634a8926c27} {Suffrage:Voter ID:server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac Address:eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac} {Suffrage:Voter ID:server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f Address:57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.483Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:08.483Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: 2024-01-08T10:38:08.483Z [INFO]  server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: entering follower state: follower="Node at 57b1cab5-0192-b351-54d4-9cd0ac0d4b3f [Follower]" leader=
    testing.go:149: server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: 2024-01-08T10:38:08.483Z [INFO]  server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: entering follower state: follower="Node at 57b1cab5-0192-b351-54d4-9cd0ac0d4b3f [Follower]" leader=
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.562Z [WARN]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.562Z [WARN]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.562Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: entering candidate state: node="Node at eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac [Candidate]" term=2
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.562Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: entering candidate state: node="Node at eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac [Candidate]" term=2
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.562Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: election won: tally=2
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.562Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: election won: tally=2
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: entering leader state: leader="Node at eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac [Leader]"
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: entering leader state: leader="Node at eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac [Leader]"
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: added peer, starting replication: peer=server-81277984-cf6e-3452-2f85-a634a8926c27
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: added peer, starting replication: peer=server-81277984-cf6e-3452-2f85-a634a8926c27
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: added peer, starting replication: peer=server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: added peer, starting replication: peer=server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: pipelining replication: peer="{Voter server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f 57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}"
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.563Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: pipelining replication: peer="{Voter server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f 57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}"
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.564Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: pipelining replication: peer="{Voter server-81277984-cf6e-3452-2f85-a634a8926c27 81277984-cf6e-3452-2f85-a634a8926c27}"
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.564Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: pipelining replication: peer="{Voter server-81277984-cf6e-3452-2f85-a634a8926c27 81277984-cf6e-3452-2f85-a634a8926c27}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.670Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:08.565031232 +0000 UTC m=+16.716825402 (2 nodes), 81.299025ms from start of poll, 84.927466ms from cluster start. Timeout at 2024-01-08 10:38:08.670050389 +0000 UTC m=+16.821844509, 105.019107ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:08.670Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:08.565031232 +0000 UTC m=+16.716825402 (2 nodes), 81.299025ms from start of poll, 84.927466ms from cluster start. Timeout at 2024-01-08 10:38:08.670050389 +0000 UTC m=+16.821844509, 105.019107ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.670Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:08.670Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.776Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:08.670663994 +0000 UTC m=+16.822458106 (1 nodes), 3.267µs from start of poll, 190.56017ms from cluster start. Timeout at 2024-01-08 10:38:08.775892195 +0000 UTC m=+16.927686302, 105.228196ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:08.776Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:08.670663994 +0000 UTC m=+16.822458106 (1 nodes), 3.267µs from start of poll, 190.56017ms from cluster start. Timeout at 2024-01-08 10:38:08.775892195 +0000 UTC m=+16.927686302, 105.228196ms after stability
testLoggerAdapter verbose: server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: 2024-01-08T10:38:08.776Z [WARN]  server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: rejecting vote request since we have a leader: from=81277984-cf6e-3452-2f85-a634a8926c27 leader=eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac
    testing.go:149: server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: 2024-01-08T10:38:08.776Z [WARN]  server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f: rejecting vote request since we have a leader: from=81277984-cf6e-3452-2f85-a634a8926c27 leader=eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.777Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: aborting pipeline replication: peer="{Voter server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f 57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}"
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.777Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: aborting pipeline replication: peer="{Voter server-57b1cab5-0192-b351-54d4-9cd0ac0d4b3f 57b1cab5-0192-b351-54d4-9cd0ac0d4b3f}"
testLoggerAdapter verbose: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.777Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: aborting pipeline replication: peer="{Voter server-81277984-cf6e-3452-2f85-a634a8926c27 81277984-cf6e-3452-2f85-a634a8926c27}"
    testing.go:149: server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: 2024-01-08T10:38:08.777Z [INFO]  server-eb1ad3c0-6db3-0384-4aa8-19c0d932e9ac: aborting pipeline replication: peer="{Voter server-81277984-cf6e-3452-2f85-a634a8926c27 81277984-cf6e-3452-2f85-a634a8926c27}"
--- PASS: TestRaft_Voting (0.30s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
testLoggerAdapter verbose: server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: 2024-01-08T10:38:08.782Z [INFO]  server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 Address:2fa038e7-29ef-37fc-95fb-c3d54d7ed694} {Suffrage:Voter ID:server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d Address:ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d} {Suffrage:Voter ID:server-a8652d3e-1813-2801-62af-3ceee35f1025 Address:a8652d3e-1813-2801-62af-3ceee35f1025}]"
    testing.go:149: server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: 2024-01-08T10:38:08.782Z [INFO]  server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 Address:2fa038e7-29ef-37fc-95fb-c3d54d7ed694} {Suffrage:Voter ID:server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d Address:ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d} {Suffrage:Voter ID:server-a8652d3e-1813-2801-62af-3ceee35f1025 Address:a8652d3e-1813-2801-62af-3ceee35f1025}]"
testLoggerAdapter verbose: server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: 2024-01-08T10:38:08.782Z [INFO]  server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: entering follower state: follower="Node at 2fa038e7-29ef-37fc-95fb-c3d54d7ed694 [Follower]" leader=
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.783Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 Address:2fa038e7-29ef-37fc-95fb-c3d54d7ed694} {Suffrage:Voter ID:server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d Address:ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d} {Suffrage:Voter ID:server-a8652d3e-1813-2801-62af-3ceee35f1025 Address:a8652d3e-1813-2801-62af-3ceee35f1025}]"
    testing.go:149: server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: 2024-01-08T10:38:08.782Z [INFO]  server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694: entering follower state: follower="Node at 2fa038e7-29ef-37fc-95fb-c3d54d7ed694 [Follower]" leader=
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.783Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 Address:2fa038e7-29ef-37fc-95fb-c3d54d7ed694} {Suffrage:Voter ID:server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d Address:ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d} {Suffrage:Voter ID:server-a8652d3e-1813-2801-62af-3ceee35f1025 Address:a8652d3e-1813-2801-62af-3ceee35f1025}]"
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.783Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: entering follower state: follower="Node at ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d [Follower]" leader=
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.783Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: entering follower state: follower="Node at ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d [Follower]" leader=
testLoggerAdapter verbose: server-a8652d3e-1813-2801-62af-3ceee35f1025: 2024-01-08T10:38:08.783Z [INFO]  server-a8652d3e-1813-2801-62af-3ceee35f1025: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 Address:2fa038e7-29ef-37fc-95fb-c3d54d7ed694} {Suffrage:Voter ID:server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d Address:ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d} {Suffrage:Voter ID:server-a8652d3e-1813-2801-62af-3ceee35f1025 Address:a8652d3e-1813-2801-62af-3ceee35f1025}]"
    testing.go:149: server-a8652d3e-1813-2801-62af-3ceee35f1025: 2024-01-08T10:38:08.783Z [INFO]  server-a8652d3e-1813-2801-62af-3ceee35f1025: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 Address:2fa038e7-29ef-37fc-95fb-c3d54d7ed694} {Suffrage:Voter ID:server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d Address:ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d} {Suffrage:Voter ID:server-a8652d3e-1813-2801-62af-3ceee35f1025 Address:a8652d3e-1813-2801-62af-3ceee35f1025}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.784Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:08.784Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-a8652d3e-1813-2801-62af-3ceee35f1025: 2024-01-08T10:38:08.784Z [INFO]  server-a8652d3e-1813-2801-62af-3ceee35f1025: entering follower state: follower="Node at a8652d3e-1813-2801-62af-3ceee35f1025 [Follower]" leader=
    testing.go:149: server-a8652d3e-1813-2801-62af-3ceee35f1025: 2024-01-08T10:38:08.784Z [INFO]  server-a8652d3e-1813-2801-62af-3ceee35f1025: entering follower state: follower="Node at a8652d3e-1813-2801-62af-3ceee35f1025 [Follower]" leader=
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.858Z [WARN]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.858Z [WARN]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.858Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: entering candidate state: node="Node at ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d [Candidate]" term=2
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.858Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: entering candidate state: node="Node at ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d [Candidate]" term=2
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: election won: tally=2
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: election won: tally=2
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: entering leader state: leader="Node at ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d [Leader]"
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: entering leader state: leader="Node at ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d [Leader]"
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: added peer, starting replication: peer=server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: added peer, starting replication: peer=server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: added peer, starting replication: peer=server-a8652d3e-1813-2801-62af-3ceee35f1025
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.859Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: added peer, starting replication: peer=server-a8652d3e-1813-2801-62af-3ceee35f1025
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.860Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: pipelining replication: peer="{Voter server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 2fa038e7-29ef-37fc-95fb-c3d54d7ed694}"
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.860Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: pipelining replication: peer="{Voter server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 2fa038e7-29ef-37fc-95fb-c3d54d7ed694}"
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.860Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: pipelining replication: peer="{Voter server-a8652d3e-1813-2801-62af-3ceee35f1025 a8652d3e-1813-2801-62af-3ceee35f1025}"
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:08.860Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: pipelining replication: peer="{Voter server-a8652d3e-1813-2801-62af-3ceee35f1025 a8652d3e-1813-2801-62af-3ceee35f1025}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.966Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:08.859546107 +0000 UTC m=+17.011340244 (2 nodes), 75.008564ms from start of poll, 78.521275ms from cluster start. Timeout at 2024-01-08 10:38:08.965981199 +0000 UTC m=+17.117775316, 106.435072ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:08.966Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:08.859546107 +0000 UTC m=+17.011340244 (2 nodes), 75.008564ms from start of poll, 78.521275ms from cluster start. Timeout at 2024-01-08 10:38:08.965981199 +0000 UTC m=+17.117775316, 106.435072ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:08.966Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:08.966Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.072Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:08.966709538 +0000 UTC m=+17.118503654 (1 nodes), 3.166µs from start of poll, 185.684685ms from cluster start. Timeout at 2024-01-08 10:38:09.072692883 +0000 UTC m=+17.224487043, 105.983389ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.072Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:08.966709538 +0000 UTC m=+17.118503654 (1 nodes), 3.166µs from start of poll, 185.684685ms from cluster start. Timeout at 2024-01-08 10:38:09.072692883 +0000 UTC m=+17.224487043, 105.983389ms after stability
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:09.073Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: aborting pipeline replication: peer="{Voter server-a8652d3e-1813-2801-62af-3ceee35f1025 a8652d3e-1813-2801-62af-3ceee35f1025}"
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:09.073Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: aborting pipeline replication: peer="{Voter server-a8652d3e-1813-2801-62af-3ceee35f1025 a8652d3e-1813-2801-62af-3ceee35f1025}"
testLoggerAdapter verbose: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:09.073Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: aborting pipeline replication: peer="{Voter server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 2fa038e7-29ef-37fc-95fb-c3d54d7ed694}"
    testing.go:149: server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: 2024-01-08T10:38:09.073Z [INFO]  server-ef0b8cc6-1c5c-9595-a5c5-3ed6e7e8f45d: aborting pipeline replication: peer="{Voter server-2fa038e7-29ef-37fc-95fb-c3d54d7ed694 2fa038e7-29ef-37fc-95fb-c3d54d7ed694}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.30s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.078Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: initial configuration: index=1 servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97}]"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.078Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: initial configuration: index=1 servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97}]"
testLoggerAdapter verbose: 1cac2c58-f503-a02c-4055-81f1d9ef4b97: 2024-01-08T10:38:09.079Z [INFO]  1cac2c58-f503-a02c-4055-81f1d9ef4b97: initial configuration: index=1 servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97}]"
    testing.go:149: 1cac2c58-f503-a02c-4055-81f1d9ef4b97: 2024-01-08T10:38:09.079Z [INFO]  1cac2c58-f503-a02c-4055-81f1d9ef4b97: initial configuration: index=1 servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97}]"
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.079Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: entering follower state: follower="Node at 42460d72-c68b-3f8e-98b1-a7c039f9d77c [Follower]" leader=
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.079Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: entering follower state: follower="Node at 42460d72-c68b-3f8e-98b1-a7c039f9d77c [Follower]" leader=
testLoggerAdapter verbose: 1cac2c58-f503-a02c-4055-81f1d9ef4b97: 2024-01-08T10:38:09.079Z [INFO]  1cac2c58-f503-a02c-4055-81f1d9ef4b97: entering follower state: follower="Node at 1cac2c58-f503-a02c-4055-81f1d9ef4b97 [Follower]" leader=
testLoggerAdapter verbose: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.079Z [INFO]  274ff229-d96c-710d-285b-11d1c2b0f378: initial configuration: index=0 servers=[]
    testing.go:149: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.079Z [INFO]  274ff229-d96c-710d-285b-11d1c2b0f378: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.080Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: 1cac2c58-f503-a02c-4055-81f1d9ef4b97: 2024-01-08T10:38:09.079Z [INFO]  1cac2c58-f503-a02c-4055-81f1d9ef4b97: entering follower state: follower="Node at 1cac2c58-f503-a02c-4055-81f1d9ef4b97 [Follower]" leader=
    testing.go:149: cluster: 2024-01-08T10:38:09.080Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.080Z [INFO]  274ff229-d96c-710d-285b-11d1c2b0f378: entering follower state: follower="Node at 274ff229-d96c-710d-285b-11d1c2b0f378 [Follower]" leader=
    testing.go:149: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.080Z [INFO]  274ff229-d96c-710d-285b-11d1c2b0f378: entering follower state: follower="Node at 274ff229-d96c-710d-285b-11d1c2b0f378 [Follower]" leader=
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.146Z [WARN]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.146Z [WARN]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.146Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: entering candidate state: node="Node at 42460d72-c68b-3f8e-98b1-a7c039f9d77c [Candidate]" term=2
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.146Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: entering candidate state: node="Node at 42460d72-c68b-3f8e-98b1-a7c039f9d77c [Candidate]" term=2
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.147Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: election won: tally=2
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.147Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: election won: tally=2
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.147Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: entering leader state: leader="Node at 42460d72-c68b-3f8e-98b1-a7c039f9d77c [Leader]"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.147Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: entering leader state: leader="Node at 42460d72-c68b-3f8e-98b1-a7c039f9d77c [Leader]"
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.148Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: added peer, starting replication: peer=1cac2c58-f503-a02c-4055-81f1d9ef4b97
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.148Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: added peer, starting replication: peer=1cac2c58-f503-a02c-4055-81f1d9ef4b97
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.148Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: pipelining replication: peer="{Voter 1cac2c58-f503-a02c-4055-81f1d9ef4b97 1cac2c58-f503-a02c-4055-81f1d9ef4b97}"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.148Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: pipelining replication: peer="{Voter 1cac2c58-f503-a02c-4055-81f1d9ef4b97 1cac2c58-f503-a02c-4055-81f1d9ef4b97}"
testLoggerAdapter verbose: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.165Z [WARN]  274ff229-d96c-710d-285b-11d1c2b0f378: no known peers, aborting election
    testing.go:149: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.165Z [WARN]  274ff229-d96c-710d-285b-11d1c2b0f378: no known peers, aborting election
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.254Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.148436747 +0000 UTC m=+17.300230921 (1 nodes), 67.835869ms from start of poll, 71.209507ms from cluster start. Timeout at 2024-01-08 10:38:09.254597882 +0000 UTC m=+17.406392004, 106.161083ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.254Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.148436747 +0000 UTC m=+17.300230921 (1 nodes), 67.835869ms from start of poll, 71.209507ms from cluster start. Timeout at 2024-01-08 10:38:09.254597882 +0000 UTC m=+17.406392004, 106.161083ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.255Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:09.255Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.360Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.255349213 +0000 UTC m=+17.407143318 (1 nodes), 3.715µs from start of poll, 178.121904ms from cluster start. Timeout at 2024-01-08 10:38:09.360406708 +0000 UTC m=+17.512200846, 105.057528ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.360Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.255349213 +0000 UTC m=+17.407143318 (1 nodes), 3.715µs from start of poll, 178.121904ms from cluster start. Timeout at 2024-01-08 10:38:09.360406708 +0000 UTC m=+17.512200846, 105.057528ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.361Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:09.361Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.470Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.361353918 +0000 UTC m=+17.513148033 (1 nodes), 3.798µs from start of poll, 284.126619ms from cluster start. Timeout at 2024-01-08 10:38:09.469918441 +0000 UTC m=+17.621712589, 108.564556ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.470Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.361353918 +0000 UTC m=+17.513148033 (1 nodes), 3.798µs from start of poll, 284.126619ms from cluster start. Timeout at 2024-01-08 10:38:09.469918441 +0000 UTC m=+17.621712589, 108.564556ms after stability
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.470Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: updating configuration: command=AddStaging server-id=274ff229-d96c-710d-285b-11d1c2b0f378 server-addr=274ff229-d96c-710d-285b-11d1c2b0f378 servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97} {Suffrage:Voter ID:274ff229-d96c-710d-285b-11d1c2b0f378 Address:274ff229-d96c-710d-285b-11d1c2b0f378}]"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.470Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: updating configuration: command=AddStaging server-id=274ff229-d96c-710d-285b-11d1c2b0f378 server-addr=274ff229-d96c-710d-285b-11d1c2b0f378 servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97} {Suffrage:Voter ID:274ff229-d96c-710d-285b-11d1c2b0f378 Address:274ff229-d96c-710d-285b-11d1c2b0f378}]"
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.471Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: added peer, starting replication: peer=274ff229-d96c-710d-285b-11d1c2b0f378
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.471Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: added peer, starting replication: peer=274ff229-d96c-710d-285b-11d1c2b0f378
testLoggerAdapter verbose: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.472Z [WARN]  274ff229-d96c-710d-285b-11d1c2b0f378: failed to get previous log: previous-index=3 last-index=0 error="log not found"
    testing.go:149: 274ff229-d96c-710d-285b-11d1c2b0f378: 2024-01-08T10:38:09.472Z [WARN]  274ff229-d96c-710d-285b-11d1c2b0f378: failed to get previous log: previous-index=3 last-index=0 error="log not found"
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.472Z [WARN]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: appendEntries rejected, sending older logs: peer="{Voter 274ff229-d96c-710d-285b-11d1c2b0f378 274ff229-d96c-710d-285b-11d1c2b0f378}" next=1
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.472Z [WARN]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: appendEntries rejected, sending older logs: peer="{Voter 274ff229-d96c-710d-285b-11d1c2b0f378 274ff229-d96c-710d-285b-11d1c2b0f378}" next=1
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.472Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: pipelining replication: peer="{Voter 274ff229-d96c-710d-285b-11d1c2b0f378 274ff229-d96c-710d-285b-11d1c2b0f378}"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.472Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: pipelining replication: peer="{Voter 274ff229-d96c-710d-285b-11d1c2b0f378 274ff229-d96c-710d-285b-11d1c2b0f378}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.478Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:09.478Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.584Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.478355191 +0000 UTC m=+17.630149315 (1 nodes), 3.162µs from start of poll, 401.127901ms from cluster start. Timeout at 2024-01-08 10:38:09.584498811 +0000 UTC m=+17.736292949, 106.143634ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.584Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.478355191 +0000 UTC m=+17.630149315 (1 nodes), 3.162µs from start of poll, 401.127901ms from cluster start. Timeout at 2024-01-08 10:38:09.584498811 +0000 UTC m=+17.736292949, 106.143634ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.584Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:09.584Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.692Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.585648147 +0000 UTC m=+17.737442271 (1 nodes), 3.755µs from start of poll, 508.420857ms from cluster start. Timeout at 2024-01-08 10:38:09.691920519 +0000 UTC m=+17.843714653, 106.272382ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.692Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.585648147 +0000 UTC m=+17.737442271 (1 nodes), 3.755µs from start of poll, 508.420857ms from cluster start. Timeout at 2024-01-08 10:38:09.691920519 +0000 UTC m=+17.843714653, 106.272382ms after stability
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.692Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: updating configuration: command=RemoveServer server-id=274ff229-d96c-710d-285b-11d1c2b0f378 server-addr= servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97}]"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.692Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: updating configuration: command=RemoveServer server-id=274ff229-d96c-710d-285b-11d1c2b0f378 server-addr= servers="[{Suffrage:Voter ID:42460d72-c68b-3f8e-98b1-a7c039f9d77c Address:42460d72-c68b-3f8e-98b1-a7c039f9d77c} {Suffrage:Voter ID:1cac2c58-f503-a02c-4055-81f1d9ef4b97 Address:1cac2c58-f503-a02c-4055-81f1d9ef4b97}]"
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.692Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: removed peer, stopping replication: peer=274ff229-d96c-710d-285b-11d1c2b0f378 last-index=4
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.692Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: removed peer, stopping replication: peer=274ff229-d96c-710d-285b-11d1c2b0f378 last-index=4
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.693Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: aborting pipeline replication: peer="{Voter 274ff229-d96c-710d-285b-11d1c2b0f378 274ff229-d96c-710d-285b-11d1c2b0f378}"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.693Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: aborting pipeline replication: peer="{Voter 274ff229-d96c-710d-285b-11d1c2b0f378 274ff229-d96c-710d-285b-11d1c2b0f378}"
testLoggerAdapter verbose: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.693Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: aborting pipeline replication: peer="{Voter 1cac2c58-f503-a02c-4055-81f1d9ef4b97 1cac2c58-f503-a02c-4055-81f1d9ef4b97}"
    testing.go:149: 42460d72-c68b-3f8e-98b1-a7c039f9d77c: 2024-01-08T10:38:09.693Z [INFO]  42460d72-c68b-3f8e-98b1-a7c039f9d77c: aborting pipeline replication: peer="{Voter 1cac2c58-f503-a02c-4055-81f1d9ef4b97 1cac2c58-f503-a02c-4055-81f1d9ef4b97}"
--- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.62s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
testLoggerAdapter verbose: 72508752-429a-0774-8739-0022d01709dc: 2024-01-08T10:38:09.698Z [INFO]  72508752-429a-0774-8739-0022d01709dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:72508752-429a-0774-8739-0022d01709dc Address:72508752-429a-0774-8739-0022d01709dc} {Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697}]"
    testing.go:149: 72508752-429a-0774-8739-0022d01709dc: 2024-01-08T10:38:09.698Z [INFO]  72508752-429a-0774-8739-0022d01709dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:72508752-429a-0774-8739-0022d01709dc Address:72508752-429a-0774-8739-0022d01709dc} {Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697}]"
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.698Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: initial configuration: index=1 servers="[{Suffrage:Voter ID:72508752-429a-0774-8739-0022d01709dc Address:72508752-429a-0774-8739-0022d01709dc} {Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697}]"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.698Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: initial configuration: index=1 servers="[{Suffrage:Voter ID:72508752-429a-0774-8739-0022d01709dc Address:72508752-429a-0774-8739-0022d01709dc} {Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.699Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:09.699Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.699Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: entering follower state: follower="Node at e8840dec-fdb5-2092-7b67-febd91c76697 [Follower]" leader=
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.699Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: entering follower state: follower="Node at e8840dec-fdb5-2092-7b67-febd91c76697 [Follower]" leader=
testLoggerAdapter verbose: 72508752-429a-0774-8739-0022d01709dc: 2024-01-08T10:38:09.699Z [INFO]  72508752-429a-0774-8739-0022d01709dc: entering follower state: follower="Node at 72508752-429a-0774-8739-0022d01709dc [Follower]" leader=
    testing.go:149: 72508752-429a-0774-8739-0022d01709dc: 2024-01-08T10:38:09.699Z [INFO]  72508752-429a-0774-8739-0022d01709dc: entering follower state: follower="Node at 72508752-429a-0774-8739-0022d01709dc [Follower]" leader=
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.772Z [WARN]  e8840dec-fdb5-2092-7b67-febd91c76697: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.772Z [WARN]  e8840dec-fdb5-2092-7b67-febd91c76697: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.773Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: entering candidate state: node="Node at e8840dec-fdb5-2092-7b67-febd91c76697 [Candidate]" term=2
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.773Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: entering candidate state: node="Node at e8840dec-fdb5-2092-7b67-febd91c76697 [Candidate]" term=2
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.773Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: election won: tally=2
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.773Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: election won: tally=2
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.774Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: entering leader state: leader="Node at e8840dec-fdb5-2092-7b67-febd91c76697 [Leader]"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.774Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: entering leader state: leader="Node at e8840dec-fdb5-2092-7b67-febd91c76697 [Leader]"
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.774Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: added peer, starting replication: peer=72508752-429a-0774-8739-0022d01709dc
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.774Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: added peer, starting replication: peer=72508752-429a-0774-8739-0022d01709dc
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.774Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: pipelining replication: peer="{Voter 72508752-429a-0774-8739-0022d01709dc 72508752-429a-0774-8739-0022d01709dc}"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.774Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: pipelining replication: peer="{Voter 72508752-429a-0774-8739-0022d01709dc 72508752-429a-0774-8739-0022d01709dc}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.882Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:09.775223536 +0000 UTC m=+17.927017694 (1 nodes), 76.00369ms from start of poll, 77.658646ms from cluster start. Timeout at 2024-01-08 10:38:09.882139632 +0000 UTC m=+18.033933744, 106.91605ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.882Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:09.775223536 +0000 UTC m=+17.927017694 (1 nodes), 76.00369ms from start of poll, 77.658646ms from cluster start. Timeout at 2024-01-08 10:38:09.882139632 +0000 UTC m=+18.033933744, 106.91605ms after stability
testLoggerAdapter verbose: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.883Z [INFO]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: initial configuration: index=0 servers=[]
    testing.go:149: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.883Z [INFO]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: initial configuration: index=0 servers=[]
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.884Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:09.884Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.884Z [INFO]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: entering follower state: follower="Node at 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 [Follower]" leader=
    testing.go:149: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.884Z [INFO]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: entering follower state: follower="Node at 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 [Follower]" leader=
testLoggerAdapter verbose: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.986Z [WARN]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: no known peers, aborting election
    testing.go:149: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.986Z [WARN]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: no known peers, aborting election
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.990Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.884434755 +0000 UTC m=+18.036228869 (1 nodes), 2.736µs from start of poll, 186.869821ms from cluster start. Timeout at 2024-01-08 10:38:09.990604844 +0000 UTC m=+18.142398962, 106.170093ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:09.990Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.884434755 +0000 UTC m=+18.036228869 (1 nodes), 2.736µs from start of poll, 186.869821ms from cluster start. Timeout at 2024-01-08 10:38:09.990604844 +0000 UTC m=+18.142398962, 106.170093ms after stability
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.991Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: updating configuration: command=AddStaging server-id=server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 server-addr=6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 servers="[{Suffrage:Voter ID:72508752-429a-0774-8739-0022d01709dc Address:72508752-429a-0774-8739-0022d01709dc} {Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697} {Suffrage:Voter ID:server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 Address:6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}]"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.991Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: updating configuration: command=AddStaging server-id=server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 server-addr=6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 servers="[{Suffrage:Voter ID:72508752-429a-0774-8739-0022d01709dc Address:72508752-429a-0774-8739-0022d01709dc} {Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697} {Suffrage:Voter ID:server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 Address:6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}]"
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.991Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: added peer, starting replication: peer=server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.991Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: added peer, starting replication: peer=server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2
testLoggerAdapter verbose: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.992Z [WARN]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: failed to get previous log: previous-index=3 last-index=0 error="log not found"
    testing.go:149: server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: 2024-01-08T10:38:09.992Z [WARN]  server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2: failed to get previous log: previous-index=3 last-index=0 error="log not found"
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.992Z [WARN]  e8840dec-fdb5-2092-7b67-febd91c76697: appendEntries rejected, sending older logs: peer="{Voter server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}" next=1
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.992Z [WARN]  e8840dec-fdb5-2092-7b67-febd91c76697: appendEntries rejected, sending older logs: peer="{Voter server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}" next=1
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.993Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: pipelining replication: peer="{Voter server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:09.993Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: pipelining replication: peer="{Voter server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:09.998Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:09.998Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.104Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.998733888 +0000 UTC m=+18.150528008 (1 nodes), 3.263µs from start of poll, 301.16896ms from cluster start. Timeout at 2024-01-08 10:38:10.103791628 +0000 UTC m=+18.255585742, 105.057734ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:10.104Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:09.998733888 +0000 UTC m=+18.150528008 (1 nodes), 3.263µs from start of poll, 301.16896ms from cluster start. Timeout at 2024-01-08 10:38:10.103791628 +0000 UTC m=+18.255585742, 105.057734ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.104Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:10.104Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.210Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.104608976 +0000 UTC m=+18.256403102 (1 nodes), 3.794µs from start of poll, 407.044054ms from cluster start. Timeout at 2024-01-08 10:38:10.210444407 +0000 UTC m=+18.362238554, 105.835452ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:10.210Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.104608976 +0000 UTC m=+18.256403102 (1 nodes), 3.794µs from start of poll, 407.044054ms from cluster start. Timeout at 2024-01-08 10:38:10.210444407 +0000 UTC m=+18.362238554, 105.835452ms after stability
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.210Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: updating configuration: command=RemoveServer server-id=72508752-429a-0774-8739-0022d01709dc server-addr= servers="[{Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697} {Suffrage:Voter ID:server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 Address:6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}]"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.210Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: updating configuration: command=RemoveServer server-id=72508752-429a-0774-8739-0022d01709dc server-addr= servers="[{Suffrage:Voter ID:e8840dec-fdb5-2092-7b67-febd91c76697 Address:e8840dec-fdb5-2092-7b67-febd91c76697} {Suffrage:Voter ID:server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 Address:6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}]"
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.211Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: removed peer, stopping replication: peer=72508752-429a-0774-8739-0022d01709dc last-index=4
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.211Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: removed peer, stopping replication: peer=72508752-429a-0774-8739-0022d01709dc last-index=4
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.212Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: aborting pipeline replication: peer="{Voter 72508752-429a-0774-8739-0022d01709dc 72508752-429a-0774-8739-0022d01709dc}"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.212Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: aborting pipeline replication: peer="{Voter 72508752-429a-0774-8739-0022d01709dc 72508752-429a-0774-8739-0022d01709dc}"
testLoggerAdapter verbose: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.212Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: aborting pipeline replication: peer="{Voter server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}"
    testing.go:149: e8840dec-fdb5-2092-7b67-febd91c76697: 2024-01-08T10:38:10.212Z [INFO]  e8840dec-fdb5-2092-7b67-febd91c76697: aborting pipeline replication: peer="{Voter server-6913f6f8-c7e8-4e79-16ff-cd86e8a571a2 6913f6f8-c7e8-4e79-16ff-cd86e8a571a2}"
--- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.53s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.225Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53e42150-8fb1-53f1-4673-90760bc46e69 Address:53e42150-8fb1-53f1-4673-90760bc46e69} {Suffrage:Voter ID:server-e89fb56d-efec-5e6c-e0cb-da20735694f1 Address:e89fb56d-efec-5e6c-e0cb-da20735694f1} {Suffrage:Voter ID:server-57f024c7-3c00-7cad-990a-630c1a980232 Address:57f024c7-3c00-7cad-990a-630c1a980232}]"
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.225Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53e42150-8fb1-53f1-4673-90760bc46e69 Address:53e42150-8fb1-53f1-4673-90760bc46e69} {Suffrage:Voter ID:server-e89fb56d-efec-5e6c-e0cb-da20735694f1 Address:e89fb56d-efec-5e6c-e0cb-da20735694f1} {Suffrage:Voter ID:server-57f024c7-3c00-7cad-990a-630c1a980232 Address:57f024c7-3c00-7cad-990a-630c1a980232}]"
testLoggerAdapter verbose: server-e89fb56d-efec-5e6c-e0cb-da20735694f1: 2024-01-08T10:38:10.235Z [INFO]  server-e89fb56d-efec-5e6c-e0cb-da20735694f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53e42150-8fb1-53f1-4673-90760bc46e69 Address:53e42150-8fb1-53f1-4673-90760bc46e69} {Suffrage:Voter ID:server-e89fb56d-efec-5e6c-e0cb-da20735694f1 Address:e89fb56d-efec-5e6c-e0cb-da20735694f1} {Suffrage:Voter ID:server-57f024c7-3c00-7cad-990a-630c1a980232 Address:57f024c7-3c00-7cad-990a-630c1a980232}]"
    testing.go:149: server-e89fb56d-efec-5e6c-e0cb-da20735694f1: 2024-01-08T10:38:10.235Z [INFO]  server-e89fb56d-efec-5e6c-e0cb-da20735694f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53e42150-8fb1-53f1-4673-90760bc46e69 Address:53e42150-8fb1-53f1-4673-90760bc46e69} {Suffrage:Voter ID:server-e89fb56d-efec-5e6c-e0cb-da20735694f1 Address:e89fb56d-efec-5e6c-e0cb-da20735694f1} {Suffrage:Voter ID:server-57f024c7-3c00-7cad-990a-630c1a980232 Address:57f024c7-3c00-7cad-990a-630c1a980232}]"
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.236Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53e42150-8fb1-53f1-4673-90760bc46e69 Address:53e42150-8fb1-53f1-4673-90760bc46e69} {Suffrage:Voter ID:server-e89fb56d-efec-5e6c-e0cb-da20735694f1 Address:e89fb56d-efec-5e6c-e0cb-da20735694f1} {Suffrage:Voter ID:server-57f024c7-3c00-7cad-990a-630c1a980232 Address:57f024c7-3c00-7cad-990a-630c1a980232}]"
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.236Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53e42150-8fb1-53f1-4673-90760bc46e69 Address:53e42150-8fb1-53f1-4673-90760bc46e69} {Suffrage:Voter ID:server-e89fb56d-efec-5e6c-e0cb-da20735694f1 Address:e89fb56d-efec-5e6c-e0cb-da20735694f1} {Suffrage:Voter ID:server-57f024c7-3c00-7cad-990a-630c1a980232 Address:57f024c7-3c00-7cad-990a-630c1a980232}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.236Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:10.236Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.236Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: entering follower state: follower="Node at 53e42150-8fb1-53f1-4673-90760bc46e69 [Follower]" leader=
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.236Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: entering follower state: follower="Node at 53e42150-8fb1-53f1-4673-90760bc46e69 [Follower]" leader=
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.236Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering follower state: follower="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Follower]" leader=
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.236Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering follower state: follower="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Follower]" leader=
testLoggerAdapter verbose: server-e89fb56d-efec-5e6c-e0cb-da20735694f1: 2024-01-08T10:38:10.236Z [INFO]  server-e89fb56d-efec-5e6c-e0cb-da20735694f1: entering follower state: follower="Node at e89fb56d-efec-5e6c-e0cb-da20735694f1 [Follower]" leader=
    testing.go:149: server-e89fb56d-efec-5e6c-e0cb-da20735694f1: 2024-01-08T10:38:10.236Z [INFO]  server-e89fb56d-efec-5e6c-e0cb-da20735694f1: entering follower state: follower="Node at e89fb56d-efec-5e6c-e0cb-da20735694f1 [Follower]" leader=
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.298Z [WARN]  server-57f024c7-3c00-7cad-990a-630c1a980232: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.298Z [WARN]  server-57f024c7-3c00-7cad-990a-630c1a980232: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.298Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering candidate state: node="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Candidate]" term=2
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.298Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering candidate state: node="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Candidate]" term=2
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: election won: tally=2
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: election won: tally=2
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering leader state: leader="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Leader]"
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering leader state: leader="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Leader]"
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: added peer, starting replication: peer=server-53e42150-8fb1-53f1-4673-90760bc46e69
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: added peer, starting replication: peer=server-53e42150-8fb1-53f1-4673-90760bc46e69
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: added peer, starting replication: peer=server-e89fb56d-efec-5e6c-e0cb-da20735694f1
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.299Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: added peer, starting replication: peer=server-e89fb56d-efec-5e6c-e0cb-da20735694f1
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.300Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: pipelining replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.300Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: pipelining replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.301Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: pipelining replication: peer="{Voter server-53e42150-8fb1-53f1-4673-90760bc46e69 53e42150-8fb1-53f1-4673-90760bc46e69}"
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.301Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: pipelining replication: peer="{Voter server-53e42150-8fb1-53f1-4673-90760bc46e69 53e42150-8fb1-53f1-4673-90760bc46e69}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.406Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.301373085 +0000 UTC m=+18.453167241 (1 nodes), 64.863093ms from start of poll, 76.509505ms from cluster start. Timeout at 2024-01-08 10:38:10.406387717 +0000 UTC m=+18.558181859, 105.014618ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:10.406Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.301373085 +0000 UTC m=+18.453167241 (1 nodes), 64.863093ms from start of poll, 76.509505ms from cluster start. Timeout at 2024-01-08 10:38:10.406387717 +0000 UTC m=+18.558181859, 105.014618ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.406Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:10.406Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.518Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.406989113 +0000 UTC m=+18.558783232 (1 nodes), 3.435µs from start of poll, 182.125496ms from cluster start. Timeout at 2024-01-08 10:38:10.518476561 +0000 UTC m=+18.670270703, 111.487471ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:10.518Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.406989113 +0000 UTC m=+18.558783232 (1 nodes), 3.435µs from start of poll, 182.125496ms from cluster start. Timeout at 2024-01-08 10:38:10.518476561 +0000 UTC m=+18.670270703, 111.487471ms after stability
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.519Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: entering candidate state: node="Node at 53e42150-8fb1-53f1-4673-90760bc46e69 [Candidate]" term=3
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.519Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: entering candidate state: node="Node at 53e42150-8fb1-53f1-4673-90760bc46e69 [Candidate]" term=3
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.520Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering follower state: follower="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Follower]" leader=
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.520Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: entering follower state: follower="Node at 57f024c7-3c00-7cad-990a-630c1a980232 [Follower]" leader=
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.520Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: aborting pipeline replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.520Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: aborting pipeline replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.520Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:10.520Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.521Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: election won: tally=2
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.521Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: election won: tally=2
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.521Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: entering leader state: leader="Node at 53e42150-8fb1-53f1-4673-90760bc46e69 [Leader]"
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.521Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: entering leader state: leader="Node at 53e42150-8fb1-53f1-4673-90760bc46e69 [Leader]"
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.521Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: added peer, starting replication: peer=server-e89fb56d-efec-5e6c-e0cb-da20735694f1
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.521Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: added peer, starting replication: peer=server-e89fb56d-efec-5e6c-e0cb-da20735694f1
testLoggerAdapter verbose: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.522Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: aborting pipeline replication: peer="{Voter server-53e42150-8fb1-53f1-4673-90760bc46e69 53e42150-8fb1-53f1-4673-90760bc46e69}"
    testing.go:149: server-57f024c7-3c00-7cad-990a-630c1a980232: 2024-01-08T10:38:10.522Z [INFO]  server-57f024c7-3c00-7cad-990a-630c1a980232: aborting pipeline replication: peer="{Voter server-53e42150-8fb1-53f1-4673-90760bc46e69 53e42150-8fb1-53f1-4673-90760bc46e69}"
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.522Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: added peer, starting replication: peer=server-57f024c7-3c00-7cad-990a-630c1a980232
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.522Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: added peer, starting replication: peer=server-57f024c7-3c00-7cad-990a-630c1a980232
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.522Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: pipelining replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.522Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: pipelining replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.523Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: pipelining replication: peer="{Voter server-57f024c7-3c00-7cad-990a-630c1a980232 57f024c7-3c00-7cad-990a-630c1a980232}"
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.523Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: pipelining replication: peer="{Voter server-57f024c7-3c00-7cad-990a-630c1a980232 57f024c7-3c00-7cad-990a-630c1a980232}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.630Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.522617412 +0000 UTC m=+18.674411582 (1 nodes), 1.787399ms from start of poll, 297.753846ms from cluster start. Timeout at 2024-01-08 10:38:10.630376511 +0000 UTC m=+18.782170657, 107.759075ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:10.630Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.522617412 +0000 UTC m=+18.674411582 (1 nodes), 1.787399ms from start of poll, 297.753846ms from cluster start. Timeout at 2024-01-08 10:38:10.630376511 +0000 UTC m=+18.782170657, 107.759075ms after stability
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.631Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: aborting pipeline replication: peer="{Voter server-57f024c7-3c00-7cad-990a-630c1a980232 57f024c7-3c00-7cad-990a-630c1a980232}"
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.631Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: aborting pipeline replication: peer="{Voter server-57f024c7-3c00-7cad-990a-630c1a980232 57f024c7-3c00-7cad-990a-630c1a980232}"
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.631Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: aborting pipeline replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.631Z [INFO]  server-53e42150-8fb1-53f1-4673-90760bc46e69: aborting pipeline replication: peer="{Voter server-e89fb56d-efec-5e6c-e0cb-da20735694f1 e89fb56d-efec-5e6c-e0cb-da20735694f1}"
testLoggerAdapter verbose: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.681Z [ERROR] server-53e42150-8fb1-53f1-4673-90760bc46e69: failed to heartbeat to: peer=e89fb56d-efec-5e6c-e0cb-da20735694f1 error="command timed out"
    testing.go:149: server-53e42150-8fb1-53f1-4673-90760bc46e69: 2024-01-08T10:38:10.681Z [ERROR] server-53e42150-8fb1-53f1-4673-90760bc46e69: failed to heartbeat to: peer=e89fb56d-efec-5e6c-e0cb-da20735694f1 error="command timed out"
--- PASS: TestRaft_LeadershipTransfer (0.46s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
testLoggerAdapter verbose: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.686Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8042552-6754-a1be-d878-18ef09730980 Address:f8042552-6754-a1be-d878-18ef09730980}]"
    testing.go:149: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.686Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8042552-6754-a1be-d878-18ef09730980 Address:f8042552-6754-a1be-d878-18ef09730980}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.687Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:10.687Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.687Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: entering follower state: follower="Node at f8042552-6754-a1be-d878-18ef09730980 [Follower]" leader=
    testing.go:149: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.687Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: entering follower state: follower="Node at f8042552-6754-a1be-d878-18ef09730980 [Follower]" leader=
testLoggerAdapter verbose: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.785Z [WARN]  server-f8042552-6754-a1be-d878-18ef09730980: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.785Z [WARN]  server-f8042552-6754-a1be-d878-18ef09730980: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.786Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: entering candidate state: node="Node at f8042552-6754-a1be-d878-18ef09730980 [Candidate]" term=2
    testing.go:149: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.786Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: entering candidate state: node="Node at f8042552-6754-a1be-d878-18ef09730980 [Candidate]" term=2
testLoggerAdapter verbose: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.786Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: election won: tally=1
    testing.go:149: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.786Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: election won: tally=1
testLoggerAdapter verbose: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.786Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: entering leader state: leader="Node at f8042552-6754-a1be-d878-18ef09730980 [Leader]"
    testing.go:149: server-f8042552-6754-a1be-d878-18ef09730980: 2024-01-08T10:38:10.786Z [INFO]  server-f8042552-6754-a1be-d878-18ef09730980: entering leader state: leader="Node at f8042552-6754-a1be-d878-18ef09730980 [Leader]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.892Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.787091033 +0000 UTC m=+18.938885172 (1 nodes), 99.902193ms from start of poll, 101.896625ms from cluster start. Timeout at 2024-01-08 10:38:10.892329999 +0000 UTC m=+19.044124143, 105.238971ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:10.892Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.787091033 +0000 UTC m=+18.938885172 (1 nodes), 99.902193ms from start of poll, 101.896625ms from cluster start. Timeout at 2024-01-08 10:38:10.892329999 +0000 UTC m=+19.044124143, 105.238971ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.21s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:10.900Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:10.900Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:10.902Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: entering follower state: follower="Node at f0272111-5736-6156-bf82-8f394c205f0e [Follower]" leader=
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:10.902Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: entering follower state: follower="Node at f0272111-5736-6156-bf82-8f394c205f0e [Follower]" leader=
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.903Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.903Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: server-54ba4e12-b2e3-2056-c415-bac8d7baa320: 2024-01-08T10:38:10.904Z [INFO]  server-54ba4e12-b2e3-2056-c415-bac8d7baa320: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
    testing.go:149: server-54ba4e12-b2e3-2056-c415-bac8d7baa320: 2024-01-08T10:38:10.904Z [INFO]  server-54ba4e12-b2e3-2056-c415-bac8d7baa320: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.904Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering follower state: follower="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Follower]" leader=
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.904Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering follower state: follower="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Follower]" leader=
testLoggerAdapter verbose: server-54ba4e12-b2e3-2056-c415-bac8d7baa320: 2024-01-08T10:38:10.904Z [INFO]  server-54ba4e12-b2e3-2056-c415-bac8d7baa320: entering follower state: follower="Node at 54ba4e12-b2e3-2056-c415-bac8d7baa320 [Follower]" leader=
    testing.go:149: server-54ba4e12-b2e3-2056-c415-bac8d7baa320: 2024-01-08T10:38:10.904Z [INFO]  server-54ba4e12-b2e3-2056-c415-bac8d7baa320: entering follower state: follower="Node at 54ba4e12-b2e3-2056-c415-bac8d7baa320 [Follower]" leader=
testLoggerAdapter verbose: server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: 2024-01-08T10:38:10.905Z [INFO]  server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
    testing.go:149: server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: 2024-01-08T10:38:10.905Z [INFO]  server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: server-c9ffd001-affd-4795-8b0f-0b693a7c3307: 2024-01-08T10:38:10.906Z [INFO]  server-c9ffd001-affd-4795-8b0f-0b693a7c3307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
    testing.go:149: server-c9ffd001-affd-4795-8b0f-0b693a7c3307: 2024-01-08T10:38:10.906Z [INFO]  server-c9ffd001-affd-4795-8b0f-0b693a7c3307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: server-f1a4d6f2-352f-9423-653f-256aeaea35fc: 2024-01-08T10:38:10.907Z [INFO]  server-f1a4d6f2-352f-9423-653f-256aeaea35fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
    testing.go:149: server-f1a4d6f2-352f-9423-653f-256aeaea35fc: 2024-01-08T10:38:10.907Z [INFO]  server-f1a4d6f2-352f-9423-653f-256aeaea35fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: 2024-01-08T10:38:10.908Z [INFO]  server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: server-c9ffd001-affd-4795-8b0f-0b693a7c3307: 2024-01-08T10:38:10.910Z [INFO]  server-c9ffd001-affd-4795-8b0f-0b693a7c3307: entering follower state: follower="Node at c9ffd001-affd-4795-8b0f-0b693a7c3307 [Follower]" leader=
testLoggerAdapter verbose: server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: 2024-01-08T10:38:10.910Z [INFO]  server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: entering follower state: follower="Node at 31d75051-ef55-e8ca-8cae-155f2ef7e5ec [Follower]" leader=
testLoggerAdapter verbose: server-f1a4d6f2-352f-9423-653f-256aeaea35fc: 2024-01-08T10:38:10.911Z [INFO]  server-f1a4d6f2-352f-9423-653f-256aeaea35fc: entering follower state: follower="Node at f1a4d6f2-352f-9423-653f-256aeaea35fc [Follower]" leader=
    testing.go:149: server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: 2024-01-08T10:38:10.908Z [INFO]  server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0272111-5736-6156-bf82-8f394c205f0e Address:f0272111-5736-6156-bf82-8f394c205f0e} {Suffrage:Voter ID:server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 Address:55bea5a6-dfbf-5309-cf6e-0aa3038778f3} {Suffrage:Voter ID:server-54ba4e12-b2e3-2056-c415-bac8d7baa320 Address:54ba4e12-b2e3-2056-c415-bac8d7baa320} {Suffrage:Voter ID:server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec Address:31d75051-ef55-e8ca-8cae-155f2ef7e5ec} {Suffrage:Voter ID:server-c9ffd001-affd-4795-8b0f-0b693a7c3307 Address:c9ffd001-affd-4795-8b0f-0b693a7c3307} {Suffrage:Voter ID:server-f1a4d6f2-352f-9423-653f-256aeaea35fc Address:f1a4d6f2-352f-9423-653f-256aeaea35fc} {Suffrage:Voter ID:server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab Address:be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:10.912Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:10.912Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: server-c9ffd001-affd-4795-8b0f-0b693a7c3307: 2024-01-08T10:38:10.910Z [INFO]  server-c9ffd001-affd-4795-8b0f-0b693a7c3307: entering follower state: follower="Node at c9ffd001-affd-4795-8b0f-0b693a7c3307 [Follower]" leader=
    testing.go:149: server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: 2024-01-08T10:38:10.910Z [INFO]  server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec: entering follower state: follower="Node at 31d75051-ef55-e8ca-8cae-155f2ef7e5ec [Follower]" leader=
    testing.go:149: server-f1a4d6f2-352f-9423-653f-256aeaea35fc: 2024-01-08T10:38:10.911Z [INFO]  server-f1a4d6f2-352f-9423-653f-256aeaea35fc: entering follower state: follower="Node at f1a4d6f2-352f-9423-653f-256aeaea35fc [Follower]" leader=
testLoggerAdapter verbose: server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: 2024-01-08T10:38:10.913Z [INFO]  server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: entering follower state: follower="Node at be18fb4a-25ac-1d45-92a6-24eb0b1d5bab [Follower]" leader=
    testing.go:149: server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: 2024-01-08T10:38:10.913Z [INFO]  server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab: entering follower state: follower="Node at be18fb4a-25ac-1d45-92a6-24eb0b1d5bab [Follower]" leader=
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.961Z [WARN]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.961Z [WARN]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.961Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering candidate state: node="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Candidate]" term=2
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.961Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering candidate state: node="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Candidate]" term=2
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.962Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: election won: tally=4
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.962Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: election won: tally=4
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering leader state: leader="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Leader]"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering leader state: leader="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Leader]"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-f0272111-5736-6156-bf82-8f394c205f0e
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-f0272111-5736-6156-bf82-8f394c205f0e
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-54ba4e12-b2e3-2056-c415-bac8d7baa320
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-54ba4e12-b2e3-2056-c415-bac8d7baa320
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-c9ffd001-affd-4795-8b0f-0b693a7c3307
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-c9ffd001-affd-4795-8b0f-0b693a7c3307
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-f1a4d6f2-352f-9423-653f-256aeaea35fc
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.963Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-f1a4d6f2-352f-9423-653f-256aeaea35fc
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: added peer, starting replication: peer=server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-f0272111-5736-6156-bf82-8f394c205f0e f0272111-5736-6156-bf82-8f394c205f0e}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-f0272111-5736-6156-bf82-8f394c205f0e f0272111-5736-6156-bf82-8f394c205f0e}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.965Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.965Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.965Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.965Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.965Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.965Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:10.964Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: pipelining replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.070Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.963442759 +0000 UTC m=+19.115236923 (1 nodes), 50.536716ms from start of poll, 63.481965ms from cluster start. Timeout at 2024-01-08 10:38:11.070192466 +0000 UTC m=+19.221986590, 106.749667ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:11.070Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:10.963442759 +0000 UTC m=+19.115236923 (1 nodes), 50.536716ms from start of poll, 63.481965ms from cluster start. Timeout at 2024-01-08 10:38:11.070192466 +0000 UTC m=+19.221986590, 106.749667ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.071Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:11.071Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.198Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:11.071325663 +0000 UTC m=+19.223119791 (6 nodes), 8.959µs from start of poll, 171.364833ms from cluster start. Timeout at 2024-01-08 10:38:11.198139439 +0000 UTC m=+19.349933588, 126.813797ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:11.198Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:11.071325663 +0000 UTC m=+19.223119791 (6 nodes), 8.959µs from start of poll, 171.364833ms from cluster start. Timeout at 2024-01-08 10:38:11.198139439 +0000 UTC m=+19.349933588, 126.813797ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.198Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:11.198Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.306Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.198944773 +0000 UTC m=+19.350738881 (1 nodes), 5.119µs from start of poll, 298.983923ms from cluster start. Timeout at 2024-01-08 10:38:11.306263114 +0000 UTC m=+19.458057264, 107.318383ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:11.306Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.198944773 +0000 UTC m=+19.350738881 (1 nodes), 5.119µs from start of poll, 298.983923ms from cluster start. Timeout at 2024-01-08 10:38:11.306263114 +0000 UTC m=+19.458057264, 107.318383ms after stability
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.307Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: entering candidate state: node="Node at f0272111-5736-6156-bf82-8f394c205f0e [Candidate]" term=3
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.307Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: entering candidate state: node="Node at f0272111-5736-6156-bf82-8f394c205f0e [Candidate]" term=3
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.307Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:11.307Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.308Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering follower state: follower="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Follower]" leader=
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.308Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: entering follower state: follower="Node at 55bea5a6-dfbf-5309-cf6e-0aa3038778f3 [Follower]" leader=
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.308Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.308Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.309Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: election won: tally=4
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.309Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: election won: tally=4
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.309Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: entering leader state: leader="Node at f0272111-5736-6156-bf82-8f394c205f0e [Leader]"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.309Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: entering leader state: leader="Node at f0272111-5736-6156-bf82-8f394c205f0e [Leader]"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.309Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-f0272111-5736-6156-bf82-8f394c205f0e f0272111-5736-6156-bf82-8f394c205f0e}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-f0272111-5736-6156-bf82-8f394c205f0e f0272111-5736-6156-bf82-8f394c205f0e}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.309Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.311Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-54ba4e12-b2e3-2056-c415-bac8d7baa320
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.311Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-54ba4e12-b2e3-2056-c415-bac8d7baa320
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.311Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.311Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-c9ffd001-affd-4795-8b0f-0b693a7c3307
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-c9ffd001-affd-4795-8b0f-0b693a7c3307
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-f1a4d6f2-352f-9423-653f-256aeaea35fc
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-f1a4d6f2-352f-9423-653f-256aeaea35fc
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: added peer, starting replication: peer=server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
testLoggerAdapter verbose: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.313Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
    testing.go:149: server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: 2024-01-08T10:38:11.310Z [INFO]  server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3: aborting pipeline replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.313Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 55bea5a6-dfbf-5309-cf6e-0aa3038778f3}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 55bea5a6-dfbf-5309-cf6e-0aa3038778f3}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.313Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.313Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.312Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.313Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.313Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: pipelining replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.418Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.310855361 +0000 UTC m=+19.462649527 (1 nodes), 2.675904ms from start of poll, 410.894569ms from cluster start. Timeout at 2024-01-08 10:38:11.418615851 +0000 UTC m=+19.570410097, 107.76057ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:11.418Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.310855361 +0000 UTC m=+19.462649527 (1 nodes), 2.675904ms from start of poll, 410.894569ms from cluster start. Timeout at 2024-01-08 10:38:11.418615851 +0000 UTC m=+19.570410097, 107.76057ms after stability
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-f1a4d6f2-352f-9423-653f-256aeaea35fc f1a4d6f2-352f-9423-653f-256aeaea35fc}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-31d75051-ef55-e8ca-8cae-155f2ef7e5ec 31d75051-ef55-e8ca-8cae-155f2ef7e5ec}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-be18fb4a-25ac-1d45-92a6-24eb0b1d5bab be18fb4a-25ac-1d45-92a6-24eb0b1d5bab}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 55bea5a6-dfbf-5309-cf6e-0aa3038778f3}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-55bea5a6-dfbf-5309-cf6e-0aa3038778f3 55bea5a6-dfbf-5309-cf6e-0aa3038778f3}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-54ba4e12-b2e3-2056-c415-bac8d7baa320 54ba4e12-b2e3-2056-c415-bac8d7baa320}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.419Z [INFO]  server-f0272111-5736-6156-bf82-8f394c205f0e: aborting pipeline replication: peer="{Voter server-c9ffd001-affd-4795-8b0f-0b693a7c3307 c9ffd001-affd-4795-8b0f-0b693a7c3307}"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.470Z [ERROR] server-f0272111-5736-6156-bf82-8f394c205f0e: failed to heartbeat to: peer=55bea5a6-dfbf-5309-cf6e-0aa3038778f3 error="command timed out"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.470Z [ERROR] server-f0272111-5736-6156-bf82-8f394c205f0e: failed to heartbeat to: peer=55bea5a6-dfbf-5309-cf6e-0aa3038778f3 error="command timed out"
testLoggerAdapter verbose: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.470Z [ERROR] server-f0272111-5736-6156-bf82-8f394c205f0e: failed to heartbeat to: peer=c9ffd001-affd-4795-8b0f-0b693a7c3307 error="command timed out"
    testing.go:149: server-f0272111-5736-6156-bf82-8f394c205f0e: 2024-01-08T10:38:11.470Z [ERROR] server-f0272111-5736-6156-bf82-8f394c205f0e: failed to heartbeat to: peer=c9ffd001-affd-4795-8b0f-0b693a7c3307 error="command timed out"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.58s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
testLoggerAdapter verbose: server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: 2024-01-08T10:38:11.479Z [INFO]  server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 Address:00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6} {Suffrage:Voter ID:server-f956539a-be76-e801-ae78-e24f373b4565 Address:f956539a-be76-e801-ae78-e24f373b4565} {Suffrage:Voter ID:server-96c78c78-175b-1d06-3742-3441c7520de8 Address:96c78c78-175b-1d06-3742-3441c7520de8}]"
    testing.go:149: server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: 2024-01-08T10:38:11.479Z [INFO]  server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 Address:00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6} {Suffrage:Voter ID:server-f956539a-be76-e801-ae78-e24f373b4565 Address:f956539a-be76-e801-ae78-e24f373b4565} {Suffrage:Voter ID:server-96c78c78-175b-1d06-3742-3441c7520de8 Address:96c78c78-175b-1d06-3742-3441c7520de8}]"
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.480Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 Address:00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6} {Suffrage:Voter ID:server-f956539a-be76-e801-ae78-e24f373b4565 Address:f956539a-be76-e801-ae78-e24f373b4565} {Suffrage:Voter ID:server-96c78c78-175b-1d06-3742-3441c7520de8 Address:96c78c78-175b-1d06-3742-3441c7520de8}]"
testLoggerAdapter verbose: server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: 2024-01-08T10:38:11.480Z [INFO]  server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: entering follower state: follower="Node at 00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 [Follower]" leader=
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.480Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 Address:00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6} {Suffrage:Voter ID:server-f956539a-be76-e801-ae78-e24f373b4565 Address:f956539a-be76-e801-ae78-e24f373b4565} {Suffrage:Voter ID:server-96c78c78-175b-1d06-3742-3441c7520de8 Address:96c78c78-175b-1d06-3742-3441c7520de8}]"
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.482Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: entering follower state: follower="Node at f956539a-be76-e801-ae78-e24f373b4565 [Follower]" leader=
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.482Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: entering follower state: follower="Node at f956539a-be76-e801-ae78-e24f373b4565 [Follower]" leader=
testLoggerAdapter verbose: server-96c78c78-175b-1d06-3742-3441c7520de8: 2024-01-08T10:38:11.482Z [INFO]  server-96c78c78-175b-1d06-3742-3441c7520de8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 Address:00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6} {Suffrage:Voter ID:server-f956539a-be76-e801-ae78-e24f373b4565 Address:f956539a-be76-e801-ae78-e24f373b4565} {Suffrage:Voter ID:server-96c78c78-175b-1d06-3742-3441c7520de8 Address:96c78c78-175b-1d06-3742-3441c7520de8}]"
    testing.go:149: server-96c78c78-175b-1d06-3742-3441c7520de8: 2024-01-08T10:38:11.482Z [INFO]  server-96c78c78-175b-1d06-3742-3441c7520de8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 Address:00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6} {Suffrage:Voter ID:server-f956539a-be76-e801-ae78-e24f373b4565 Address:f956539a-be76-e801-ae78-e24f373b4565} {Suffrage:Voter ID:server-96c78c78-175b-1d06-3742-3441c7520de8 Address:96c78c78-175b-1d06-3742-3441c7520de8}]"
    testing.go:149: server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: 2024-01-08T10:38:11.480Z [INFO]  server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6: entering follower state: follower="Node at 00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 [Follower]" leader=
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.483Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:11.483Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-96c78c78-175b-1d06-3742-3441c7520de8: 2024-01-08T10:38:11.483Z [INFO]  server-96c78c78-175b-1d06-3742-3441c7520de8: entering follower state: follower="Node at 96c78c78-175b-1d06-3742-3441c7520de8 [Follower]" leader=
    testing.go:149: server-96c78c78-175b-1d06-3742-3441c7520de8: 2024-01-08T10:38:11.483Z [INFO]  server-96c78c78-175b-1d06-3742-3441c7520de8: entering follower state: follower="Node at 96c78c78-175b-1d06-3742-3441c7520de8 [Follower]" leader=
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.566Z [WARN]  server-f956539a-be76-e801-ae78-e24f373b4565: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.566Z [WARN]  server-f956539a-be76-e801-ae78-e24f373b4565: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.566Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: entering candidate state: node="Node at f956539a-be76-e801-ae78-e24f373b4565 [Candidate]" term=2
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.566Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: entering candidate state: node="Node at f956539a-be76-e801-ae78-e24f373b4565 [Candidate]" term=2
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.566Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: election won: tally=2
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.566Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: election won: tally=2
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: entering leader state: leader="Node at f956539a-be76-e801-ae78-e24f373b4565 [Leader]"
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: entering leader state: leader="Node at f956539a-be76-e801-ae78-e24f373b4565 [Leader]"
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: added peer, starting replication: peer=server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: added peer, starting replication: peer=server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: added peer, starting replication: peer=server-96c78c78-175b-1d06-3742-3441c7520de8
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: added peer, starting replication: peer=server-96c78c78-175b-1d06-3742-3441c7520de8
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: pipelining replication: peer="{Voter server-96c78c78-175b-1d06-3742-3441c7520de8 96c78c78-175b-1d06-3742-3441c7520de8}"
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.567Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: pipelining replication: peer="{Voter server-96c78c78-175b-1d06-3742-3441c7520de8 96c78c78-175b-1d06-3742-3441c7520de8}"
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.568Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: pipelining replication: peer="{Voter server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6}"
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.568Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: pipelining replication: peer="{Voter server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.674Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.569226533 +0000 UTC m=+19.721020684 (1 nodes), 86.052107ms from start of poll, 90.424886ms from cluster start. Timeout at 2024-01-08 10:38:11.674400533 +0000 UTC m=+19.826194650, 105.173966ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:11.674Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.569226533 +0000 UTC m=+19.721020684 (1 nodes), 86.052107ms from start of poll, 90.424886ms from cluster start. Timeout at 2024-01-08 10:38:11.674400533 +0000 UTC m=+19.826194650, 105.173966ms after stability
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.675Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: aborting pipeline replication: peer="{Voter server-96c78c78-175b-1d06-3742-3441c7520de8 96c78c78-175b-1d06-3742-3441c7520de8}"
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.675Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: aborting pipeline replication: peer="{Voter server-96c78c78-175b-1d06-3742-3441c7520de8 96c78c78-175b-1d06-3742-3441c7520de8}"
testLoggerAdapter verbose: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.675Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: aborting pipeline replication: peer="{Voter server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6}"
    testing.go:149: server-f956539a-be76-e801-ae78-e24f373b4565: 2024-01-08T10:38:11.675Z [INFO]  server-f956539a-be76-e801-ae78-e24f373b4565: aborting pipeline replication: peer="{Voter server-00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6 00fcbb5e-bf4f-8c45-c2d0-7561ca3a5ea6}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.20s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.682Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27fe5bd7-87b4-23d2-7d8e-17513423f730 Address:27fe5bd7-87b4-23d2-7d8e-17513423f730} {Suffrage:Voter ID:server-a8e9af00-a4dd-5523-1c27-53c24057e517 Address:a8e9af00-a4dd-5523-1c27-53c24057e517} {Suffrage:Voter ID:server-6c85be01-ec95-4576-06e8-8de0e31d61e5 Address:6c85be01-ec95-4576-06e8-8de0e31d61e5}]"
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.682Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27fe5bd7-87b4-23d2-7d8e-17513423f730 Address:27fe5bd7-87b4-23d2-7d8e-17513423f730} {Suffrage:Voter ID:server-a8e9af00-a4dd-5523-1c27-53c24057e517 Address:a8e9af00-a4dd-5523-1c27-53c24057e517} {Suffrage:Voter ID:server-6c85be01-ec95-4576-06e8-8de0e31d61e5 Address:6c85be01-ec95-4576-06e8-8de0e31d61e5}]"
testLoggerAdapter verbose: server-a8e9af00-a4dd-5523-1c27-53c24057e517: 2024-01-08T10:38:11.683Z [INFO]  server-a8e9af00-a4dd-5523-1c27-53c24057e517: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27fe5bd7-87b4-23d2-7d8e-17513423f730 Address:27fe5bd7-87b4-23d2-7d8e-17513423f730} {Suffrage:Voter ID:server-a8e9af00-a4dd-5523-1c27-53c24057e517 Address:a8e9af00-a4dd-5523-1c27-53c24057e517} {Suffrage:Voter ID:server-6c85be01-ec95-4576-06e8-8de0e31d61e5 Address:6c85be01-ec95-4576-06e8-8de0e31d61e5}]"
    testing.go:149: server-a8e9af00-a4dd-5523-1c27-53c24057e517: 2024-01-08T10:38:11.683Z [INFO]  server-a8e9af00-a4dd-5523-1c27-53c24057e517: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27fe5bd7-87b4-23d2-7d8e-17513423f730 Address:27fe5bd7-87b4-23d2-7d8e-17513423f730} {Suffrage:Voter ID:server-a8e9af00-a4dd-5523-1c27-53c24057e517 Address:a8e9af00-a4dd-5523-1c27-53c24057e517} {Suffrage:Voter ID:server-6c85be01-ec95-4576-06e8-8de0e31d61e5 Address:6c85be01-ec95-4576-06e8-8de0e31d61e5}]"
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.683Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: entering follower state: follower="Node at 27fe5bd7-87b4-23d2-7d8e-17513423f730 [Follower]" leader=
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.683Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: entering follower state: follower="Node at 27fe5bd7-87b4-23d2-7d8e-17513423f730 [Follower]" leader=
testLoggerAdapter verbose: server-6c85be01-ec95-4576-06e8-8de0e31d61e5: 2024-01-08T10:38:11.684Z [INFO]  server-6c85be01-ec95-4576-06e8-8de0e31d61e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27fe5bd7-87b4-23d2-7d8e-17513423f730 Address:27fe5bd7-87b4-23d2-7d8e-17513423f730} {Suffrage:Voter ID:server-a8e9af00-a4dd-5523-1c27-53c24057e517 Address:a8e9af00-a4dd-5523-1c27-53c24057e517} {Suffrage:Voter ID:server-6c85be01-ec95-4576-06e8-8de0e31d61e5 Address:6c85be01-ec95-4576-06e8-8de0e31d61e5}]"
    testing.go:149: server-6c85be01-ec95-4576-06e8-8de0e31d61e5: 2024-01-08T10:38:11.684Z [INFO]  server-6c85be01-ec95-4576-06e8-8de0e31d61e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27fe5bd7-87b4-23d2-7d8e-17513423f730 Address:27fe5bd7-87b4-23d2-7d8e-17513423f730} {Suffrage:Voter ID:server-a8e9af00-a4dd-5523-1c27-53c24057e517 Address:a8e9af00-a4dd-5523-1c27-53c24057e517} {Suffrage:Voter ID:server-6c85be01-ec95-4576-06e8-8de0e31d61e5 Address:6c85be01-ec95-4576-06e8-8de0e31d61e5}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.684Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:11.684Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-6c85be01-ec95-4576-06e8-8de0e31d61e5: 2024-01-08T10:38:11.684Z [INFO]  server-6c85be01-ec95-4576-06e8-8de0e31d61e5: entering follower state: follower="Node at 6c85be01-ec95-4576-06e8-8de0e31d61e5 [Follower]" leader=
    testing.go:149: server-6c85be01-ec95-4576-06e8-8de0e31d61e5: 2024-01-08T10:38:11.684Z [INFO]  server-6c85be01-ec95-4576-06e8-8de0e31d61e5: entering follower state: follower="Node at 6c85be01-ec95-4576-06e8-8de0e31d61e5 [Follower]" leader=
testLoggerAdapter verbose: server-a8e9af00-a4dd-5523-1c27-53c24057e517: 2024-01-08T10:38:11.684Z [INFO]  server-a8e9af00-a4dd-5523-1c27-53c24057e517: entering follower state: follower="Node at a8e9af00-a4dd-5523-1c27-53c24057e517 [Follower]" leader=
    testing.go:149: server-a8e9af00-a4dd-5523-1c27-53c24057e517: 2024-01-08T10:38:11.684Z [INFO]  server-a8e9af00-a4dd-5523-1c27-53c24057e517: entering follower state: follower="Node at a8e9af00-a4dd-5523-1c27-53c24057e517 [Follower]" leader=
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.754Z [WARN]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.754Z [WARN]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.754Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: entering candidate state: node="Node at 27fe5bd7-87b4-23d2-7d8e-17513423f730 [Candidate]" term=2
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.754Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: entering candidate state: node="Node at 27fe5bd7-87b4-23d2-7d8e-17513423f730 [Candidate]" term=2
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.754Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: election won: tally=2
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.754Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: election won: tally=2
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.755Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: entering leader state: leader="Node at 27fe5bd7-87b4-23d2-7d8e-17513423f730 [Leader]"
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.755Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: entering leader state: leader="Node at 27fe5bd7-87b4-23d2-7d8e-17513423f730 [Leader]"
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.755Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: added peer, starting replication: peer=server-a8e9af00-a4dd-5523-1c27-53c24057e517
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.755Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: added peer, starting replication: peer=server-a8e9af00-a4dd-5523-1c27-53c24057e517
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.755Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: added peer, starting replication: peer=server-6c85be01-ec95-4576-06e8-8de0e31d61e5
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.755Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: added peer, starting replication: peer=server-6c85be01-ec95-4576-06e8-8de0e31d61e5
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.756Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: pipelining replication: peer="{Voter server-6c85be01-ec95-4576-06e8-8de0e31d61e5 6c85be01-ec95-4576-06e8-8de0e31d61e5}"
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.756Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: pipelining replication: peer="{Voter server-6c85be01-ec95-4576-06e8-8de0e31d61e5 6c85be01-ec95-4576-06e8-8de0e31d61e5}"
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.756Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: pipelining replication: peer="{Voter server-a8e9af00-a4dd-5523-1c27-53c24057e517 a8e9af00-a4dd-5523-1c27-53c24057e517}"
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.756Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: pipelining replication: peer="{Voter server-a8e9af00-a4dd-5523-1c27-53c24057e517 a8e9af00-a4dd-5523-1c27-53c24057e517}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.863Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:11.757268034 +0000 UTC m=+19.909062452 (2 nodes), 72.61644ms from start of poll, 76.144918ms from cluster start. Timeout at 2024-01-08 10:38:11.862794649 +0000 UTC m=+20.014588791, 105.526339ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:11.863Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:11.757268034 +0000 UTC m=+19.909062452 (2 nodes), 72.61644ms from start of poll, 76.144918ms from cluster start. Timeout at 2024-01-08 10:38:11.862794649 +0000 UTC m=+20.014588791, 105.526339ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.863Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:11.863Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.969Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.863714456 +0000 UTC m=+20.015508577 (1 nodes), 4.067µs from start of poll, 182.591043ms from cluster start. Timeout at 2024-01-08 10:38:11.969168259 +0000 UTC m=+20.120962377, 105.4538ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:11.969Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:11.863714456 +0000 UTC m=+20.015508577 (1 nodes), 4.067µs from start of poll, 182.591043ms from cluster start. Timeout at 2024-01-08 10:38:11.969168259 +0000 UTC m=+20.120962377, 105.4538ms after stability
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.970Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: aborting pipeline replication: peer="{Voter server-6c85be01-ec95-4576-06e8-8de0e31d61e5 6c85be01-ec95-4576-06e8-8de0e31d61e5}"
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.970Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: aborting pipeline replication: peer="{Voter server-6c85be01-ec95-4576-06e8-8de0e31d61e5 6c85be01-ec95-4576-06e8-8de0e31d61e5}"
testLoggerAdapter verbose: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.970Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: aborting pipeline replication: peer="{Voter server-a8e9af00-a4dd-5523-1c27-53c24057e517 a8e9af00-a4dd-5523-1c27-53c24057e517}"
    testing.go:149: server-27fe5bd7-87b4-23d2-7d8e-17513423f730: 2024-01-08T10:38:11.970Z [INFO]  server-27fe5bd7-87b4-23d2-7d8e-17513423f730: aborting pipeline replication: peer="{Voter server-a8e9af00-a4dd-5523-1c27-53c24057e517 a8e9af00-a4dd-5523-1c27-53c24057e517}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:11.975Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f4247-d6cb-f3e4-0a73-ec02310c5843 Address:873f4247-d6cb-f3e4-0a73-ec02310c5843} {Suffrage:Voter ID:server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd Address:fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd} {Suffrage:Voter ID:server-03b7c3cb-533d-cd23-0206-c024e7a8a12d Address:03b7c3cb-533d-cd23-0206-c024e7a8a12d}]"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:11.975Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f4247-d6cb-f3e4-0a73-ec02310c5843 Address:873f4247-d6cb-f3e4-0a73-ec02310c5843} {Suffrage:Voter ID:server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd Address:fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd} {Suffrage:Voter ID:server-03b7c3cb-533d-cd23-0206-c024e7a8a12d Address:03b7c3cb-533d-cd23-0206-c024e7a8a12d}]"
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:11.976Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: entering follower state: follower="Node at 873f4247-d6cb-f3e4-0a73-ec02310c5843 [Follower]" leader=
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:11.976Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f4247-d6cb-f3e4-0a73-ec02310c5843 Address:873f4247-d6cb-f3e4-0a73-ec02310c5843} {Suffrage:Voter ID:server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd Address:fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd} {Suffrage:Voter ID:server-03b7c3cb-533d-cd23-0206-c024e7a8a12d Address:03b7c3cb-533d-cd23-0206-c024e7a8a12d}]"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:11.976Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: entering follower state: follower="Node at 873f4247-d6cb-f3e4-0a73-ec02310c5843 [Follower]" leader=
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:11.976Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f4247-d6cb-f3e4-0a73-ec02310c5843 Address:873f4247-d6cb-f3e4-0a73-ec02310c5843} {Suffrage:Voter ID:server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd Address:fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd} {Suffrage:Voter ID:server-03b7c3cb-533d-cd23-0206-c024e7a8a12d Address:03b7c3cb-533d-cd23-0206-c024e7a8a12d}]"
testLoggerAdapter verbose: server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: 2024-01-08T10:38:11.977Z [INFO]  server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f4247-d6cb-f3e4-0a73-ec02310c5843 Address:873f4247-d6cb-f3e4-0a73-ec02310c5843} {Suffrage:Voter ID:server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd Address:fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd} {Suffrage:Voter ID:server-03b7c3cb-533d-cd23-0206-c024e7a8a12d Address:03b7c3cb-533d-cd23-0206-c024e7a8a12d}]"
    testing.go:149: server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: 2024-01-08T10:38:11.977Z [INFO]  server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f4247-d6cb-f3e4-0a73-ec02310c5843 Address:873f4247-d6cb-f3e4-0a73-ec02310c5843} {Suffrage:Voter ID:server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd Address:fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd} {Suffrage:Voter ID:server-03b7c3cb-533d-cd23-0206-c024e7a8a12d Address:03b7c3cb-533d-cd23-0206-c024e7a8a12d}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:11.978Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:11.978Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:11.978Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering follower state: follower="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Follower]" leader=
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:11.978Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering follower state: follower="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Follower]" leader=
testLoggerAdapter verbose: server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: 2024-01-08T10:38:11.978Z [INFO]  server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: entering follower state: follower="Node at 03b7c3cb-533d-cd23-0206-c024e7a8a12d [Follower]" leader=
    testing.go:149: server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: 2024-01-08T10:38:11.978Z [INFO]  server-03b7c3cb-533d-cd23-0206-c024e7a8a12d: entering follower state: follower="Node at 03b7c3cb-533d-cd23-0206-c024e7a8a12d [Follower]" leader=
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.040Z [WARN]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.040Z [WARN]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.040Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering candidate state: node="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Candidate]" term=2
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.040Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering candidate state: node="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Candidate]" term=2
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: election won: tally=2
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: election won: tally=2
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering leader state: leader="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Leader]"
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering leader state: leader="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Leader]"
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: added peer, starting replication: peer=server-873f4247-d6cb-f3e4-0a73-ec02310c5843
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: added peer, starting replication: peer=server-873f4247-d6cb-f3e4-0a73-ec02310c5843
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: added peer, starting replication: peer=server-03b7c3cb-533d-cd23-0206-c024e7a8a12d
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.041Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: added peer, starting replication: peer=server-03b7c3cb-533d-cd23-0206-c024e7a8a12d
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.042Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: pipelining replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.042Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: pipelining replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.042Z [WARN]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: rejecting vote request since our last term is greater: candidate=fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd last-term=2 last-candidate-term=1
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.042Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: pipelining replication: peer="{Voter server-873f4247-d6cb-f3e4-0a73-ec02310c5843 873f4247-d6cb-f3e4-0a73-ec02310c5843}"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.042Z [WARN]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: rejecting vote request since our last term is greater: candidate=fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd last-term=2 last-candidate-term=1
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.042Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: pipelining replication: peer="{Voter server-873f4247-d6cb-f3e4-0a73-ec02310c5843 873f4247-d6cb-f3e4-0a73-ec02310c5843}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.149Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.04332978 +0000 UTC m=+20.195123914 (1 nodes), 65.061486ms from start of poll, 68.555178ms from cluster start. Timeout at 2024-01-08 10:38:12.149476659 +0000 UTC m=+20.301270796, 106.146882ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:12.149Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.04332978 +0000 UTC m=+20.195123914 (1 nodes), 65.061486ms from start of poll, 68.555178ms from cluster start. Timeout at 2024-01-08 10:38:12.149476659 +0000 UTC m=+20.301270796, 106.146882ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.150Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:12.150Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.256Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:12.150282268 +0000 UTC m=+20.302076472 (2 nodes), 5.145µs from start of poll, 175.507736ms from cluster start. Timeout at 2024-01-08 10:38:12.256467944 +0000 UTC m=+20.408262073, 106.185601ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:12.256Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:12.150282268 +0000 UTC m=+20.302076472 (2 nodes), 5.145µs from start of poll, 175.507736ms from cluster start. Timeout at 2024-01-08 10:38:12.256467944 +0000 UTC m=+20.408262073, 106.185601ms after stability
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.414Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: entering candidate state: node="Node at 873f4247-d6cb-f3e4-0a73-ec02310c5843 [Candidate]" term=3
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.414Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: entering candidate state: node="Node at 873f4247-d6cb-f3e4-0a73-ec02310c5843 [Candidate]" term=3
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: election won: tally=2
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: election won: tally=2
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: entering leader state: leader="Node at 873f4247-d6cb-f3e4-0a73-ec02310c5843 [Leader]"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: entering leader state: leader="Node at 873f4247-d6cb-f3e4-0a73-ec02310c5843 [Leader]"
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: added peer, starting replication: peer=server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: added peer, starting replication: peer=server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: added peer, starting replication: peer=server-03b7c3cb-533d-cd23-0206-c024e7a8a12d
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.415Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: added peer, starting replication: peer=server-03b7c3cb-533d-cd23-0206-c024e7a8a12d
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.416Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:12.416Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.416Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering follower state: follower="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Follower]" leader=
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.416Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: entering follower state: follower="Node at fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd [Follower]" leader=
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.417Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: aborting pipeline replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.417Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: aborting pipeline replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.417Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: pipelining replication: peer="{Voter server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd}"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.417Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: pipelining replication: peer="{Voter server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd}"
testLoggerAdapter verbose: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.418Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: aborting pipeline replication: peer="{Voter server-873f4247-d6cb-f3e4-0a73-ec02310c5843 873f4247-d6cb-f3e4-0a73-ec02310c5843}"
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.418Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: pipelining replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
    testing.go:149: server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: 2024-01-08T10:38:12.418Z [INFO]  server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd: aborting pipeline replication: peer="{Voter server-873f4247-d6cb-f3e4-0a73-ec02310c5843 873f4247-d6cb-f3e4-0a73-ec02310c5843}"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.418Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: pipelining replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.523Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.418206019 +0000 UTC m=+20.570000189 (1 nodes), 1.628291ms from start of poll, 443.431453ms from cluster start. Timeout at 2024-01-08 10:38:12.523324675 +0000 UTC m=+20.675118825, 105.118636ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:12.523Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.418206019 +0000 UTC m=+20.570000189 (1 nodes), 1.628291ms from start of poll, 443.431453ms from cluster start. Timeout at 2024-01-08 10:38:12.523324675 +0000 UTC m=+20.675118825, 105.118636ms after stability
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.523Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: aborting pipeline replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.523Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: aborting pipeline replication: peer="{Voter server-03b7c3cb-533d-cd23-0206-c024e7a8a12d 03b7c3cb-533d-cd23-0206-c024e7a8a12d}"
testLoggerAdapter verbose: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.524Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: aborting pipeline replication: peer="{Voter server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd}"
    testing.go:149: server-873f4247-d6cb-f3e4-0a73-ec02310c5843: 2024-01-08T10:38:12.524Z [INFO]  server-873f4247-d6cb-f3e4-0a73-ec02310c5843: aborting pipeline replication: peer="{Voter server-fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd fd7d4256-6bf1-f7b7-d36e-c4130a3bf2dd}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.55s)
=== RUN   TestRaft_LeadershipTransferToItself
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.529Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10a609b5-7166-88a6-6479-02df57bea880 Address:10a609b5-7166-88a6-6479-02df57bea880} {Suffrage:Voter ID:server-b09eeb25-94f4-6665-df12-cb2bdb12b653 Address:b09eeb25-94f4-6665-df12-cb2bdb12b653} {Suffrage:Voter ID:server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 Address:3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}]"
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.529Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10a609b5-7166-88a6-6479-02df57bea880 Address:10a609b5-7166-88a6-6479-02df57bea880} {Suffrage:Voter ID:server-b09eeb25-94f4-6665-df12-cb2bdb12b653 Address:b09eeb25-94f4-6665-df12-cb2bdb12b653} {Suffrage:Voter ID:server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 Address:3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}]"
testLoggerAdapter verbose: server-b09eeb25-94f4-6665-df12-cb2bdb12b653: 2024-01-08T10:38:12.530Z [INFO]  server-b09eeb25-94f4-6665-df12-cb2bdb12b653: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10a609b5-7166-88a6-6479-02df57bea880 Address:10a609b5-7166-88a6-6479-02df57bea880} {Suffrage:Voter ID:server-b09eeb25-94f4-6665-df12-cb2bdb12b653 Address:b09eeb25-94f4-6665-df12-cb2bdb12b653} {Suffrage:Voter ID:server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 Address:3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}]"
    testing.go:149: server-b09eeb25-94f4-6665-df12-cb2bdb12b653: 2024-01-08T10:38:12.530Z [INFO]  server-b09eeb25-94f4-6665-df12-cb2bdb12b653: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10a609b5-7166-88a6-6479-02df57bea880 Address:10a609b5-7166-88a6-6479-02df57bea880} {Suffrage:Voter ID:server-b09eeb25-94f4-6665-df12-cb2bdb12b653 Address:b09eeb25-94f4-6665-df12-cb2bdb12b653} {Suffrage:Voter ID:server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 Address:3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}]"
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.531Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: entering follower state: follower="Node at 10a609b5-7166-88a6-6479-02df57bea880 [Follower]" leader=
testLoggerAdapter verbose: server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: 2024-01-08T10:38:12.531Z [INFO]  server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10a609b5-7166-88a6-6479-02df57bea880 Address:10a609b5-7166-88a6-6479-02df57bea880} {Suffrage:Voter ID:server-b09eeb25-94f4-6665-df12-cb2bdb12b653 Address:b09eeb25-94f4-6665-df12-cb2bdb12b653} {Suffrage:Voter ID:server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 Address:3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}]"
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.531Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: entering follower state: follower="Node at 10a609b5-7166-88a6-6479-02df57bea880 [Follower]" leader=
    testing.go:149: server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: 2024-01-08T10:38:12.531Z [INFO]  server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10a609b5-7166-88a6-6479-02df57bea880 Address:10a609b5-7166-88a6-6479-02df57bea880} {Suffrage:Voter ID:server-b09eeb25-94f4-6665-df12-cb2bdb12b653 Address:b09eeb25-94f4-6665-df12-cb2bdb12b653} {Suffrage:Voter ID:server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 Address:3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}]"
testLoggerAdapter verbose: server-b09eeb25-94f4-6665-df12-cb2bdb12b653: 2024-01-08T10:38:12.531Z [INFO]  server-b09eeb25-94f4-6665-df12-cb2bdb12b653: entering follower state: follower="Node at b09eeb25-94f4-6665-df12-cb2bdb12b653 [Follower]" leader=
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.532Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: server-b09eeb25-94f4-6665-df12-cb2bdb12b653: 2024-01-08T10:38:12.531Z [INFO]  server-b09eeb25-94f4-6665-df12-cb2bdb12b653: entering follower state: follower="Node at b09eeb25-94f4-6665-df12-cb2bdb12b653 [Follower]" leader=
    testing.go:149: cluster: 2024-01-08T10:38:12.532Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: 2024-01-08T10:38:12.532Z [INFO]  server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: entering follower state: follower="Node at 3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 [Follower]" leader=
    testing.go:149: server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: 2024-01-08T10:38:12.532Z [INFO]  server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65: entering follower state: follower="Node at 3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 [Follower]" leader=
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.590Z [WARN]  server-10a609b5-7166-88a6-6479-02df57bea880: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.590Z [WARN]  server-10a609b5-7166-88a6-6479-02df57bea880: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.590Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: entering candidate state: node="Node at 10a609b5-7166-88a6-6479-02df57bea880 [Candidate]" term=2
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.590Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: entering candidate state: node="Node at 10a609b5-7166-88a6-6479-02df57bea880 [Candidate]" term=2
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: election won: tally=2
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: election won: tally=2
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: entering leader state: leader="Node at 10a609b5-7166-88a6-6479-02df57bea880 [Leader]"
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: entering leader state: leader="Node at 10a609b5-7166-88a6-6479-02df57bea880 [Leader]"
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: added peer, starting replication: peer=server-b09eeb25-94f4-6665-df12-cb2bdb12b653
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: added peer, starting replication: peer=server-b09eeb25-94f4-6665-df12-cb2bdb12b653
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: added peer, starting replication: peer=server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.591Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: added peer, starting replication: peer=server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.592Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: pipelining replication: peer="{Voter server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}"
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.592Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: pipelining replication: peer="{Voter server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}"
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.592Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: pipelining replication: peer="{Voter server-b09eeb25-94f4-6665-df12-cb2bdb12b653 b09eeb25-94f4-6665-df12-cb2bdb12b653}"
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.592Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: pipelining replication: peer="{Voter server-b09eeb25-94f4-6665-df12-cb2bdb12b653 b09eeb25-94f4-6665-df12-cb2bdb12b653}"
testLoggerAdapter verbose: server-b09eeb25-94f4-6665-df12-cb2bdb12b653: 2024-01-08T10:38:12.593Z [WARN]  server-b09eeb25-94f4-6665-df12-cb2bdb12b653: rejecting vote request since our last term is greater: candidate=10a609b5-7166-88a6-6479-02df57bea880 last-term=2 last-candidate-term=1
    testing.go:149: server-b09eeb25-94f4-6665-df12-cb2bdb12b653: 2024-01-08T10:38:12.593Z [WARN]  server-b09eeb25-94f4-6665-df12-cb2bdb12b653: rejecting vote request since our last term is greater: candidate=10a609b5-7166-88a6-6479-02df57bea880 last-term=2 last-candidate-term=1
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.703Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.59349628 +0000 UTC m=+20.745290433 (1 nodes), 61.135766ms from start of poll, 64.835489ms from cluster start. Timeout at 2024-01-08 10:38:12.70279201 +0000 UTC m=+20.854586145, 109.295712ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:12.703Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.59349628 +0000 UTC m=+20.745290433 (1 nodes), 61.135766ms from start of poll, 64.835489ms from cluster start. Timeout at 2024-01-08 10:38:12.70279201 +0000 UTC m=+20.854586145, 109.295712ms after stability
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.703Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: cannot transfer leadership to itself
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.703Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: cannot transfer leadership to itself
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.703Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: aborting pipeline replication: peer="{Voter server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}"
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.703Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: aborting pipeline replication: peer="{Voter server-3d5aca8c-a51a-d2e5-21dc-1fdff2268e65 3d5aca8c-a51a-d2e5-21dc-1fdff2268e65}"
testLoggerAdapter verbose: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.704Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: aborting pipeline replication: peer="{Voter server-b09eeb25-94f4-6665-df12-cb2bdb12b653 b09eeb25-94f4-6665-df12-cb2bdb12b653}"
    testing.go:149: server-10a609b5-7166-88a6-6479-02df57bea880: 2024-01-08T10:38:12.704Z [INFO]  server-10a609b5-7166-88a6-6479-02df57bea880: aborting pipeline replication: peer="{Voter server-b09eeb25-94f4-6665-df12-cb2bdb12b653 b09eeb25-94f4-6665-df12-cb2bdb12b653}"
--- PASS: TestRaft_LeadershipTransferToItself (0.18s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
    raft_test.go:2064: Disabled unreliable test
--- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
testLoggerAdapter verbose: server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: 2024-01-08T10:38:12.709Z [INFO]  server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 Address:49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9} {Suffrage:Voter ID:server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb Address:d9504d62-ccbf-fe4c-2f67-9e000882c2cb} {Suffrage:Voter ID:server-51a050fc-9c87-fcda-6b33-0e022396b186 Address:51a050fc-9c87-fcda-6b33-0e022396b186}]"
    testing.go:149: server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: 2024-01-08T10:38:12.709Z [INFO]  server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 Address:49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9} {Suffrage:Voter ID:server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb Address:d9504d62-ccbf-fe4c-2f67-9e000882c2cb} {Suffrage:Voter ID:server-51a050fc-9c87-fcda-6b33-0e022396b186 Address:51a050fc-9c87-fcda-6b33-0e022396b186}]"
testLoggerAdapter verbose: server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: 2024-01-08T10:38:12.710Z [INFO]  server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 Address:49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9} {Suffrage:Voter ID:server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb Address:d9504d62-ccbf-fe4c-2f67-9e000882c2cb} {Suffrage:Voter ID:server-51a050fc-9c87-fcda-6b33-0e022396b186 Address:51a050fc-9c87-fcda-6b33-0e022396b186}]"
    testing.go:149: server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: 2024-01-08T10:38:12.710Z [INFO]  server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 Address:49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9} {Suffrage:Voter ID:server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb Address:d9504d62-ccbf-fe4c-2f67-9e000882c2cb} {Suffrage:Voter ID:server-51a050fc-9c87-fcda-6b33-0e022396b186 Address:51a050fc-9c87-fcda-6b33-0e022396b186}]"
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.711Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 Address:49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9} {Suffrage:Voter ID:server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb Address:d9504d62-ccbf-fe4c-2f67-9e000882c2cb} {Suffrage:Voter ID:server-51a050fc-9c87-fcda-6b33-0e022396b186 Address:51a050fc-9c87-fcda-6b33-0e022396b186}]"
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.711Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 Address:49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9} {Suffrage:Voter ID:server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb Address:d9504d62-ccbf-fe4c-2f67-9e000882c2cb} {Suffrage:Voter ID:server-51a050fc-9c87-fcda-6b33-0e022396b186 Address:51a050fc-9c87-fcda-6b33-0e022396b186}]"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.712Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:12.712Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: 2024-01-08T10:38:12.712Z [INFO]  server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: entering follower state: follower="Node at d9504d62-ccbf-fe4c-2f67-9e000882c2cb [Follower]" leader=
    testing.go:149: server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: 2024-01-08T10:38:12.712Z [INFO]  server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb: entering follower state: follower="Node at d9504d62-ccbf-fe4c-2f67-9e000882c2cb [Follower]" leader=
testLoggerAdapter verbose: server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: 2024-01-08T10:38:12.712Z [INFO]  server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: entering follower state: follower="Node at 49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 [Follower]" leader=
    testing.go:149: server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: 2024-01-08T10:38:12.712Z [INFO]  server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9: entering follower state: follower="Node at 49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 [Follower]" leader=
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.712Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: entering follower state: follower="Node at 51a050fc-9c87-fcda-6b33-0e022396b186 [Follower]" leader=
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.712Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: entering follower state: follower="Node at 51a050fc-9c87-fcda-6b33-0e022396b186 [Follower]" leader=
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.775Z [WARN]  server-51a050fc-9c87-fcda-6b33-0e022396b186: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.775Z [WARN]  server-51a050fc-9c87-fcda-6b33-0e022396b186: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.776Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: entering candidate state: node="Node at 51a050fc-9c87-fcda-6b33-0e022396b186 [Candidate]" term=2
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.776Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: entering candidate state: node="Node at 51a050fc-9c87-fcda-6b33-0e022396b186 [Candidate]" term=2
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.776Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: election won: tally=2
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.776Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: election won: tally=2
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.776Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: entering leader state: leader="Node at 51a050fc-9c87-fcda-6b33-0e022396b186 [Leader]"
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.776Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: entering leader state: leader="Node at 51a050fc-9c87-fcda-6b33-0e022396b186 [Leader]"
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.777Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: added peer, starting replication: peer=server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.777Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: added peer, starting replication: peer=server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.777Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: added peer, starting replication: peer=server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.777Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: added peer, starting replication: peer=server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.777Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: pipelining replication: peer="{Voter server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb d9504d62-ccbf-fe4c-2f67-9e000882c2cb}"
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.777Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: pipelining replication: peer="{Voter server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb d9504d62-ccbf-fe4c-2f67-9e000882c2cb}"
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.778Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: pipelining replication: peer="{Voter server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9}"
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:12.778Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: pipelining replication: peer="{Voter server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.886Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.778984673 +0000 UTC m=+20.930778819 (1 nodes), 66.805381ms from start of poll, 70.272808ms from cluster start. Timeout at 2024-01-08 10:38:12.88597709 +0000 UTC m=+21.037771418, 106.992599ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:12.886Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:12.778984673 +0000 UTC m=+20.930778819 (1 nodes), 66.805381ms from start of poll, 70.272808ms from cluster start. Timeout at 2024-01-08 10:38:12.88597709 +0000 UTC m=+21.037771418, 106.992599ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.886Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: cluster: 2024-01-08T10:38:12.886Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:12.992Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:12.886805747 +0000 UTC m=+21.038599857 (2 nodes), 5.7µs from start of poll, 178.093846ms from cluster start. Timeout at 2024-01-08 10:38:12.991935808 +0000 UTC m=+21.143729914, 105.130057ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:12.992Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:12.886805747 +0000 UTC m=+21.038599857 (2 nodes), 5.7µs from start of poll, 178.093846ms from cluster start. Timeout at 2024-01-08 10:38:12.991935808 +0000 UTC m=+21.143729914, 105.130057ms after stability
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:14.909Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: aborting pipeline replication: peer="{Voter server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb d9504d62-ccbf-fe4c-2f67-9e000882c2cb}"
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:14.909Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: aborting pipeline replication: peer="{Voter server-d9504d62-ccbf-fe4c-2f67-9e000882c2cb d9504d62-ccbf-fe4c-2f67-9e000882c2cb}"
testLoggerAdapter verbose: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:14.910Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: aborting pipeline replication: peer="{Voter server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9}"
    testing.go:149: server-51a050fc-9c87-fcda-6b33-0e022396b186: 2024-01-08T10:38:14.910Z [INFO]  server-51a050fc-9c87-fcda-6b33-0e022396b186: aborting pipeline replication: peer="{Voter server-49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9 49446452-d9fb-5ac2-aed8-d4c0b8bbc8b9}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (2.21s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
testLoggerAdapter verbose: server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: 2024-01-08T10:38:14.915Z [INFO]  server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 Address:68c7c05b-d179-a4ab-37b1-d2343ac1d873} {Suffrage:Voter ID:server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a Address:ef0b4f92-bcfa-feaa-de41-4d1a43af682a}]"
    testing.go:149: server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: 2024-01-08T10:38:14.915Z [INFO]  server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 Address:68c7c05b-d179-a4ab-37b1-d2343ac1d873} {Suffrage:Voter ID:server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a Address:ef0b4f92-bcfa-feaa-de41-4d1a43af682a}]"
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:14.916Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 Address:68c7c05b-d179-a4ab-37b1-d2343ac1d873} {Suffrage:Voter ID:server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a Address:ef0b4f92-bcfa-feaa-de41-4d1a43af682a}]"
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:14.916Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 Address:68c7c05b-d179-a4ab-37b1-d2343ac1d873} {Suffrage:Voter ID:server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a Address:ef0b4f92-bcfa-feaa-de41-4d1a43af682a}]"
testLoggerAdapter verbose: server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: 2024-01-08T10:38:14.916Z [INFO]  server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: entering follower state: follower="Node at 68c7c05b-d179-a4ab-37b1-d2343ac1d873 [Follower]" leader=
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:14.916Z [INFO]  starting stability test: raft-state=Follower
    testing.go:149: server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: 2024-01-08T10:38:14.916Z [INFO]  server-68c7c05b-d179-a4ab-37b1-d2343ac1d873: entering follower state: follower="Node at 68c7c05b-d179-a4ab-37b1-d2343ac1d873 [Follower]" leader=
    testing.go:149: cluster: 2024-01-08T10:38:14.916Z [INFO]  starting stability test: raft-state=Follower
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:14.917Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: entering follower state: follower="Node at ef0b4f92-bcfa-feaa-de41-4d1a43af682a [Follower]" leader=
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:14.917Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: entering follower state: follower="Node at ef0b4f92-bcfa-feaa-de41-4d1a43af682a [Follower]" leader=
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.006Z [WARN]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: heartbeat timeout reached, starting election: last-leader=
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.006Z [WARN]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: heartbeat timeout reached, starting election: last-leader=
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.006Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: entering candidate state: node="Node at ef0b4f92-bcfa-feaa-de41-4d1a43af682a [Candidate]" term=2
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.006Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: entering candidate state: node="Node at ef0b4f92-bcfa-feaa-de41-4d1a43af682a [Candidate]" term=2
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.006Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: election won: tally=2
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.006Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: election won: tally=2
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.007Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: entering leader state: leader="Node at ef0b4f92-bcfa-feaa-de41-4d1a43af682a [Leader]"
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.007Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: entering leader state: leader="Node at ef0b4f92-bcfa-feaa-de41-4d1a43af682a [Leader]"
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.007Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: added peer, starting replication: peer=server-68c7c05b-d179-a4ab-37b1-d2343ac1d873
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.007Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: added peer, starting replication: peer=server-68c7c05b-d179-a4ab-37b1-d2343ac1d873
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.007Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: pipelining replication: peer="{Voter server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 68c7c05b-d179-a4ab-37b1-d2343ac1d873}"
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.007Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: pipelining replication: peer="{Voter server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 68c7c05b-d179-a4ab-37b1-d2343ac1d873}"
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:15.113Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:15.008246911 +0000 UTC m=+23.160041073 (1 nodes), 90.914482ms from start of poll, 93.358084ms from cluster start. Timeout at 2024-01-08 10:38:15.113798671 +0000 UTC m=+23.265592523, 105.55145ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:15.113Z [INFO]  stable state for Follower reached at 2024-01-08 10:38:15.008246911 +0000 UTC m=+23.160041073 (1 nodes), 90.914482ms from start of poll, 93.358084ms from cluster start. Timeout at 2024-01-08 10:38:15.113798671 +0000 UTC m=+23.265592523, 105.55145ms after stability
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:15.113Z [INFO]  starting stability test: raft-state=Leader
    testing.go:149: cluster: 2024-01-08T10:38:15.113Z [INFO]  starting stability test: raft-state=Leader
testLoggerAdapter verbose: cluster: 2024-01-08T10:38:15.219Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:15.113994939 +0000 UTC m=+23.265788788 (1 nodes), 2.365µs from start of poll, 199.105799ms from cluster start. Timeout at 2024-01-08 10:38:15.219767244 +0000 UTC m=+23.371561388, 105.7726ms after stability
    testing.go:149: cluster: 2024-01-08T10:38:15.219Z [INFO]  stable state for Leader reached at 2024-01-08 10:38:15.113994939 +0000 UTC m=+23.265788788 (1 nodes), 2.365µs from start of poll, 199.105799ms from cluster start. Timeout at 2024-01-08 10:38:15.219767244 +0000 UTC m=+23.371561388, 105.7726ms after stability
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.232Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 Address:68c7c05b-d179-a4ab-37b1-d2343ac1d873} {Suffrage:Voter ID:server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a Address:ef0b4f92-bcfa-feaa-de41-4d1a43af682a}]"
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.232Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 Address:68c7c05b-d179-a4ab-37b1-d2343ac1d873} {Suffrage:Voter ID:server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a Address:ef0b4f92-bcfa-feaa-de41-4d1a43af682a}]"
testLoggerAdapter verbose: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.233Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: aborting pipeline replication: peer="{Voter server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 68c7c05b-d179-a4ab-37b1-d2343ac1d873}"
    testing.go:149: server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: 2024-01-08T10:38:15.233Z [INFO]  server-ef0b4f92-bcfa-feaa-de41-4d1a43af682a: aborting pipeline replication: peer="{Voter server-68c7c05b-d179-a4ab-37b1-d2343ac1d873 68c7c05b-d179-a4ab-37b1-d2343ac1d873}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.32s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.03s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.04s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
PASS
ok  	github.com/hashicorp/raft	23.527s
?   	github.com/hashicorp/raft/bench	[no test files]
   create-stamp debian/debhelper-build-stamp
 fakeroot debian/rules binary
dh binary --buildsystem=golang --with=golang
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installinit -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_amd64.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/B01_cleanup starting
I: user script /srv/workspace/pbuilder/1225125/tmp/hooks/B01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/1225125 and its subdirectories
I: Current time: Tue Jan  9 00:39:20 +14 2024
I: pbuilder-time-stamp: 1704710360