I: pbuilder: network access will be disabled during build
I: Current time: Sat Jul  5 15:06:28 +14 2025
I: pbuilder-time-stamp: 1751677588
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
gpgv: Signature made Thu Jul 27 15:15:46 2023 
gpgv:                using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29
gpgv:                issuer "dswarbrick@debian.org"
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/D01_modify_environment starting
debug: Running on infom02-amd64.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 Jul  5 01:06 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="21" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
  BASH_VERSION='5.2.21(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=amd64
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 '
  DIRSTACK=()
  DISTRIBUTION=unstable
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=x86_64
  HOST_ARCH=amd64
  IFS=' 	
  '
  INVOCATION_ID=32de06c9a19a453ca549982f46e5de4e
  LANG=C
  LANGUAGE=et_EE:et
  LC_ALL=C
  MACHTYPE=x86_64-pc-linux-gnu
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnu
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=827298
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/pbuilderrc_YNyB --distribution unstable --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID=109
  SUDO_UID=104
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
I: uname -a
  Linux i-capture-the-hostname 6.7.12+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.12-1~bpo12+1 (2024-05-06) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Jun 30 14:05 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: amd64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19719 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.22-go{a} golang-1.22-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 
0 packages upgraded, 102 newly installed, 0 to remove and 0 not upgraded.
Need to get 90.2 MB of archives. After unpacking 627 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian unstable/main amd64 sensible-utils all 0.0.22 [22.4 kB]
Get: 2 http://deb.debian.org/debian unstable/main amd64 libmagic-mgc amd64 1:5.45-3 [314 kB]
Get: 3 http://deb.debian.org/debian unstable/main amd64 libmagic1t64 amd64 1:5.45-3 [105 kB]
Get: 4 http://deb.debian.org/debian unstable/main amd64 file amd64 1:5.45-3 [42.9 kB]
Get: 5 http://deb.debian.org/debian unstable/main amd64 gettext-base amd64 0.21-14+b1 [161 kB]
Get: 6 http://deb.debian.org/debian unstable/main amd64 libuchardet0 amd64 0.0.8-1+b1 [68.8 kB]
Get: 7 http://deb.debian.org/debian unstable/main amd64 groff-base amd64 1.23.0-4 [1180 kB]
Get: 8 http://deb.debian.org/debian unstable/main amd64 bsdextrautils amd64 2.40.1-4 [95.7 kB]
Get: 9 http://deb.debian.org/debian unstable/main amd64 libpipeline1 amd64 1.5.7-2 [38.0 kB]
Get: 10 http://deb.debian.org/debian unstable/main amd64 man-db amd64 2.12.1-1 [1411 kB]
Get: 11 http://deb.debian.org/debian unstable/main amd64 m4 amd64 1.4.19-4 [287 kB]
Get: 12 http://deb.debian.org/debian unstable/main amd64 autoconf all 2.71-3 [332 kB]
Get: 13 http://deb.debian.org/debian unstable/main amd64 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian unstable/main amd64 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian unstable/main amd64 autopoint all 0.21-14 [496 kB]
Get: 16 http://deb.debian.org/debian unstable/main amd64 libdebhelper-perl all 13.15.3 [88.0 kB]
Get: 17 http://deb.debian.org/debian unstable/main amd64 libtool all 2.4.7-7 [517 kB]
Get: 18 http://deb.debian.org/debian unstable/main amd64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian unstable/main amd64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian unstable/main amd64 libfile-stripnondeterminism-perl all 1.14.0-1 [19.5 kB]
Get: 21 http://deb.debian.org/debian unstable/main amd64 dh-strip-nondeterminism all 1.14.0-1 [8448 B]
Get: 22 http://deb.debian.org/debian unstable/main amd64 libelf1t64 amd64 0.191-1+b1 [189 kB]
Get: 23 http://deb.debian.org/debian unstable/main amd64 dwz amd64 0.15-1+b1 [110 kB]
Get: 24 http://deb.debian.org/debian unstable/main amd64 libicu72 amd64 72.1-4+b1 [9395 kB]
Get: 25 http://deb.debian.org/debian unstable/main amd64 libxml2 amd64 2.12.7+dfsg-2 [670 kB]
Get: 26 http://deb.debian.org/debian unstable/main amd64 gettext amd64 0.21-14+b1 [1301 kB]
Get: 27 http://deb.debian.org/debian unstable/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 28 http://deb.debian.org/debian unstable/main amd64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 29 http://deb.debian.org/debian unstable/main amd64 debhelper all 13.15.3 [901 kB]
Get: 30 http://deb.debian.org/debian unstable/main amd64 dh-golang all 1.62 [24.7 kB]
Get: 31 http://deb.debian.org/debian unstable/main amd64 golang-1.22-src all 1.22.3-1 [18.8 MB]
Get: 32 http://deb.debian.org/debian unstable/main amd64 golang-1.22-go amd64 1.22.3-1 [24.7 MB]
Get: 33 http://deb.debian.org/debian unstable/main amd64 golang-src all 2:1.22~3 [5020 B]
Get: 34 http://deb.debian.org/debian unstable/main amd64 golang-go amd64 2:1.22~3 [44.2 kB]
Get: 35 http://deb.debian.org/debian unstable/main amd64 golang-any amd64 2:1.22~3 [5104 B]
Get: 36 http://deb.debian.org/debian unstable/main amd64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B]
Get: 37 http://deb.debian.org/debian unstable/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 38 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 39 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-sys-dev all 0.20.0-1 [430 kB]
Get: 40 http://deb.debian.org/debian unstable/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 41 http://deb.debian.org/debian unstable/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 42 http://deb.debian.org/debian unstable/main amd64 golang-github-fatih-color-dev all 1.16.0-1 [13.8 kB]
Get: 43 http://deb.debian.org/debian unstable/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 44 http://deb.debian.org/debian unstable/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 45 http://deb.debian.org/debian unstable/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 46 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 47 http://deb.debian.org/debian unstable/main amd64 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 48 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 49 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 50 http://deb.debian.org/debian unstable/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 51 http://deb.debian.org/debian unstable/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 52 http://deb.debian.org/debian unstable/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 53 http://deb.debian.org/debian unstable/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 54 http://deb.debian.org/debian unstable/main amd64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 55 http://deb.debian.org/debian unstable/main amd64 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 56 http://deb.debian.org/debian unstable/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 57 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 58 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 59 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 60 http://deb.debian.org/debian unstable/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 61 http://deb.debian.org/debian unstable/main amd64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 62 http://deb.debian.org/debian unstable/main amd64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B]
Get: 63 http://deb.debian.org/debian unstable/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 64 http://deb.debian.org/debian unstable/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 65 http://deb.debian.org/debian unstable/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 66 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 67 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 68 http://deb.debian.org/debian unstable/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 69 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-term-dev all 0.20.0-1 [14.8 kB]
Get: 70 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-text-dev all 0.15.0-1 [4006 kB]
Get: 71 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-crypto-dev all 1:0.23.0-1 [1689 kB]
Get: 72 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-net-dev all 1:0.25.0+dfsg-1 [895 kB]
Get: 73 http://deb.debian.org/debian unstable/main amd64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 74 http://deb.debian.org/debian unstable/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 75 http://deb.debian.org/debian unstable/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 76 http://deb.debian.org/debian unstable/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 77 http://deb.debian.org/debian unstable/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 78 http://deb.debian.org/debian unstable/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 79 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 80 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 81 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-common-dev all 0.53.0-1 [114 kB]
Get: 82 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 83 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 84 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-client-golang-dev all 1.19.0-1 [179 kB]
Get: 85 http://deb.debian.org/debian unstable/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 86 http://deb.debian.org/debian unstable/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 87 http://deb.debian.org/debian unstable/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 88 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 89 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 90 http://deb.debian.org/debian unstable/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-6 [19.5 kB]
Get: 91 http://deb.debian.org/debian unstable/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-6 [56.9 kB]
Get: 92 http://deb.debian.org/debian unstable/main amd64 libssl-dev amd64 3.2.1-3 [2699 kB]
Get: 93 http://deb.debian.org/debian unstable/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-6 [231 kB]
Get: 94 http://deb.debian.org/debian unstable/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 95 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-mod-dev all 0.17.0-1 [103 kB]
Get: 96 http://deb.debian.org/debian unstable/main amd64 golang-github-yuin-goldmark-dev all 1.7.1-1 [113 kB]
Get: 97 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-tools-dev all 1:0.20.0+ds-1 [1571 kB]
Get: 98 http://deb.debian.org/debian unstable/main amd64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 99 http://deb.debian.org/debian unstable/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 100 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 101 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 102 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 90.2 MB in 1s (121 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19719 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.22_all.deb ...
Unpacking sensible-utils (0.0.22) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3_amd64.deb ...
Unpacking libmagic-mgc (1:5.45-3) ...
Selecting previously unselected package libmagic1t64:amd64.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3_amd64.deb ...
Unpacking libmagic1t64:amd64 (1:5.45-3) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3_amd64.deb ...
Unpacking file (1:5.45-3) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.21-14+b1_amd64.deb ...
Unpacking gettext-base (0.21-14+b1) ...
Selecting previously unselected package libuchardet0:amd64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b1_amd64.deb ...
Unpacking libuchardet0:amd64 (0.0.8-1+b1) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-4_amd64.deb ...
Unpacking groff-base (1.23.0-4) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.1-4_amd64.deb ...
Unpacking bsdextrautils (2.40.1-4) ...
Selecting previously unselected package libpipeline1:amd64.
Preparing to unpack .../008-libpipeline1_1.5.7-2_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.7-2) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.12.1-1_amd64.deb ...
Unpacking man-db (2.12.1-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-4_amd64.deb ...
Unpacking m4 (1.4.19-4) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.71-3_all.deb ...
Unpacking autoconf (2.71-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.21-14_all.deb ...
Unpacking autopoint (0.21-14) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.15.3_all.deb ...
Unpacking libdebhelper-perl (13.15.3) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.4.7-7_all.deb ...
Unpacking libtool (2.4.7-7) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.0-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.0-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.0-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.0-1) ...
Selecting previously unselected package libelf1t64:amd64.
Preparing to unpack .../021-libelf1t64_0.191-1+b1_amd64.deb ...
Unpacking libelf1t64:amd64 (0.191-1+b1) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_amd64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libicu72:amd64.
Preparing to unpack .../023-libicu72_72.1-4+b1_amd64.deb ...
Unpacking libicu72:amd64 (72.1-4+b1) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../024-libxml2_2.12.7+dfsg-2_amd64.deb ...
Unpacking libxml2:amd64 (2.12.7+dfsg-2) ...
Selecting previously unselected package gettext.
Preparing to unpack .../025-gettext_0.21-14+b1_amd64.deb ...
Unpacking gettext (0.21-14+b1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../028-debhelper_13.15.3_all.deb ...
Unpacking debhelper (13.15.3) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../029-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.22-src.
Preparing to unpack .../030-golang-1.22-src_1.22.3-1_all.deb ...
Unpacking golang-1.22-src (1.22.3-1) ...
Selecting previously unselected package golang-1.22-go.
Preparing to unpack .../031-golang-1.22-go_1.22.3-1_amd64.deb ...
Unpacking golang-1.22-go (1.22.3-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../032-golang-src_2%3a1.22~3_all.deb ...
Unpacking golang-src (2:1.22~3) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../033-golang-go_2%3a1.22~3_amd64.deb ...
Unpacking golang-go:amd64 (2:1.22~3) ...
Selecting previously unselected package golang-any:amd64.
Preparing to unpack .../034-golang-any_2%3a1.22~3_amd64.deb ...
Unpacking golang-any:amd64 (2:1.22~3) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../038-golang-golang-x-sys-dev_0.20.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.20.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../041-golang-github-fatih-color-dev_1.16.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.16.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../054-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../065-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../066-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../067-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../068-golang-golang-x-term-dev_0.20.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.20.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../069-golang-golang-x-text-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.15.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../070-golang-golang-x-crypto-dev_1%3a0.23.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.23.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../071-golang-golang-x-net-dev_1%3a0.25.0+dfsg-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.25.0+dfsg-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../072-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../073-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../074-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../075-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../076-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../078-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../079-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../080-golang-github-prometheus-common-dev_0.53.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.53.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../081-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../082-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../083-golang-github-prometheus-client-golang-dev_1.19.0-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../084-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../085-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../086-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../087-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../088-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../089-libsasl2-modules-db_2.1.28+dfsg1-6_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../090-libsasl2-2_2.1.28+dfsg1-6_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libssl-dev:amd64.
Preparing to unpack .../091-libssl-dev_3.2.1-3_amd64.deb ...
Unpacking libssl-dev:amd64 (3.2.1-3) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../092-libsasl2-dev_2.1.28+dfsg1-6_amd64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-6) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../093-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../094-golang-golang-x-mod-dev_0.17.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.17.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../095-golang-github-yuin-goldmark-dev_1.7.1-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.1-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../096-golang-golang-x-tools-dev_1%3a0.20.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.20.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../097-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../098-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../099-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../100-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../101-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:amd64 (1.5.7-2) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:amd64 (72.1-4+b1) ...
Setting up bsdextrautils (2.40.1-4) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.15.3) ...
Setting up golang-golang-x-sys-dev (0.20.0-1) ...
Setting up libmagic1t64:amd64 (1:5.45-3) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.21-14+b1) ...
Setting up m4 (1.4.19-4) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.20.0-1) ...
Setting up file (1:5.45-3) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:amd64 (0.191-1+b1) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-6) ...
Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.1-1) ...
Setting up golang-1.22-src (1.22.3-1) ...
Setting up libssl-dev:amd64 (3.2.1-3) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.17.0-1) ...
Setting up autopoint (0.21-14) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-6) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.71-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up sensible-utils (0.0.22) ...
Setting up golang-golang-x-text-dev (0.15.0-1) ...
Setting up libuchardet0:amd64 (0.0.8-1+b1) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.23.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.22~3) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:amd64 (2.12.7+dfsg-2) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.0-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.21-14+b1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.4.7-7) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-1.22-go (1.22.3-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-6) ...
Setting up golang-golang-x-net-dev (1:0.25.0+dfsg-1) ...
Setting up dh-strip-nondeterminism (1.14.0-1) ...
Setting up groff-base (1.23.0-4) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.16.0-1) ...
Setting up golang-go:amd64 (2:1.22~3) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.20.0+ds-1) ...
Setting up man-db (2.12.1-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:amd64 (2:1.22~3) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.15.3) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.53.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.38-11) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for unstable
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/goexperiment
internal/coverage/rtcov
unicode
internal/unsafeheader
internal/goos
internal/godebugs
internal/goarch
internal/cpu
runtime/internal/atomic
sync/atomic
internal/abi
internal/chacha8rand
runtime/internal/math
unicode/utf8
container/list
runtime/internal/syscall
internal/race
runtime/internal/sys
internal/itoa
math/bits
crypto/internal/alias
crypto/subtle
crypto/internal/boring/sig
cmp
encoding
unicode/utf16
log/internal
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
slices
math
internal/bytealg
runtime
internal/reflectlite
sync
internal/bisect
internal/singleflight
internal/testlog
runtime/cgo
errors
sort
io
internal/oserror
internal/godebug
vendor/golang.org/x/net/dns/dnsmessage
github.com/hashicorp/golang-lru/simplelru
strconv
crypto/internal/nistec/fiat
internal/safefilepath
path
syscall
bytes
strings
hash
crypto/internal/randutil
internal/saferio
math/rand
internal/intern
hash/crc32
hash/crc64
crypto
reflect
net/netip
crypto/rc4
net/http/internal/ascii
regexp/syntax
html
bufio
github.com/hashicorp/go-immutable-radix
vendor/golang.org/x/text/transform
internal/syscall/unix
internal/syscall/execenv
time
regexp
internal/poll
io/fs
context
internal/fmtsort
encoding/binary
embed
os
encoding/base64
crypto/internal/edwards25519/field
crypto/internal/nistec
crypto/md5
vendor/golang.org/x/crypto/internal/poly1305
crypto/cipher
golang.org/x/sys/unix
crypto/internal/edwards25519
encoding/pem
crypto/internal/boring
crypto/des
vendor/golang.org/x/crypto/chacha20
crypto/aes
crypto/sha512
crypto/hmac
crypto/sha1
crypto/sha256
vendor/golang.org/x/crypto/hkdf
fmt
path/filepath
vendor/golang.org/x/sys/cpu
net
io/ioutil
os/signal
crypto/ecdh
internal/sysinfo
vendor/golang.org/x/crypto/chacha20poly1305
log
encoding/json
compress/flate
encoding/hex
math/big
net/url
mime
vendor/golang.org/x/net/http2/hpack
mime/quotedprintable
vendor/golang.org/x/text/unicode/norm
net/http/internal
encoding/gob
vendor/golang.org/x/text/unicode/bidi
go/token
text/template/parse
flag
runtime/debug
runtime/trace
vendor/golang.org/x/text/secure/bidirule
compress/gzip
testing
vendor/golang.org/x/net/idna
github.com/mattn/go-isatty
text/template
github.com/mattn/go-colorable
github.com/fatih/color
github.com/hashicorp/go-hclog
crypto/rand
crypto/elliptic
crypto/internal/boring/bbig
crypto/internal/bigmod
encoding/asn1
crypto/dsa
crypto/ed25519
html/template
crypto/rsa
crypto/x509/pkix
vendor/golang.org/x/crypto/cryptobyte
crypto/ecdsa
crypto/x509
vendor/golang.org/x/net/http/httpproxy
net/textproto
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-x86_64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-07-05T01:10:24.767Z [INFO]  creating new snapshot: path=/tmp/raft3159836619/raft1802644306/snapshots/3-10-1751677824767.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.03s)
=== RUN   TestFileSS_CreateSnapshot
2025-07-05T01:10:24.801Z [INFO]  creating new snapshot: path=/tmp/raft2659761758/snapshots/3-10-1751677824801.tmp
2025-07-05T01:10:24.807Z [WARN]  found temporary snapshot: name=3-10-1751677824801.tmp
--- PASS: TestFileSS_CreateSnapshot (0.02s)
=== RUN   TestFileSS_CancelSnapshot
2025-07-05T01:10:24.825Z [INFO]  creating new snapshot: path=/tmp/raft3781357084/snapshots/3-10-1751677824825.tmp
--- PASS: TestFileSS_CancelSnapshot (0.01s)
=== RUN   TestFileSS_Retention
2025-07-05T01:10:24.832Z [INFO]  creating new snapshot: path=/tmp/raft2341103548/snapshots/3-10-1751677824832.tmp
2025-07-05T01:10:24.847Z [INFO]  creating new snapshot: path=/tmp/raft2341103548/snapshots/3-11-1751677824847.tmp
2025-07-05T01:10:24.863Z [INFO]  creating new snapshot: path=/tmp/raft2341103548/snapshots/3-12-1751677824863.tmp
2025-07-05T01:10:24.885Z [INFO]  reaping snapshot: path=/tmp/raft2341103548/snapshots/3-10-1751677824832
2025-07-05T01:10:24.890Z [INFO]  creating new snapshot: path=/tmp/raft2341103548/snapshots/3-13-1751677824890.tmp
2025-07-05T01:10:24.915Z [INFO]  reaping snapshot: path=/tmp/raft2341103548/snapshots/3-11-1751677824847
2025-07-05T01:10:24.915Z [INFO]  creating new snapshot: path=/tmp/raft2341103548/snapshots/3-14-1751677824915.tmp
2025-07-05T01:10:24.943Z [INFO]  reaping snapshot: path=/tmp/raft2341103548/snapshots/3-12-1751677824863
--- PASS: TestFileSS_Retention (0.11s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-07-05T01:10:24.944Z [INFO]  creating new snapshot: path=/tmp/raft2309289625/snapshots/5-130350-1751677824944.tmp
2025-07-05T01:10:24.964Z [INFO]  creating new snapshot: path=/tmp/raft2309289625/snapshots/36-204917-1751677824964.tmp
--- PASS: TestFileSS_Ordering (0.03s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.01s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.01s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2025-07-05T01:10:25.082Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
2025-07-05T01:10:25.133Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.06s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
2025-07-05T01:10:25.185Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.01s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.199Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.204Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.215Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.235Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.276Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.356Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.517Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:25.837Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:26.477Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:27.478Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-05T01:10:28.479Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-07-05T01:10:29.200Z [INFO]  server-d64f90f6-584c-1b68-5e58-8d97b0d332ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d64f90f6-584c-1b68-5e58-8d97b0d332ff Address:d64f90f6-584c-1b68-5e58-8d97b0d332ff}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-07-05T01:10:29.201Z [INFO]  server-29d2cd23-7141-b235-3545-6af839e0d292: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29d2cd23-7141-b235-3545-6af839e0d292 Address:29d2cd23-7141-b235-3545-6af839e0d292}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-07-05T01:10:29.201Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: initial configuration: index=0 servers=[]
2025-07-05T01:10:29.201Z [INFO]  server-30e002a8-aaea-7757-b8c2-5583e55489e1: initial configuration: index=0 servers=[]
2025-07-05T01:10:29.201Z [INFO]  server-9a6aebbd-de6c-e44b-8511-834dceb91265: initial configuration: index=0 servers=[]
2025-07-05T01:10:29.201Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: entering follower state: follower="Node at 3ba32829-4cfc-689d-8bd3-6b7f04154fbc [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.201Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:29.201Z [INFO]  server-30e002a8-aaea-7757-b8c2-5583e55489e1: entering follower state: follower="Node at 30e002a8-aaea-7757-b8c2-5583e55489e1 [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.201Z [INFO]  server-9a6aebbd-de6c-e44b-8511-834dceb91265: entering follower state: follower="Node at 9a6aebbd-de6c-e44b-8511-834dceb91265 [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.264Z [WARN]  server-30e002a8-aaea-7757-b8c2-5583e55489e1: no known peers, aborting election
2025-07-05T01:10:29.270Z [WARN]  server-9a6aebbd-de6c-e44b-8511-834dceb91265: no known peers, aborting election
2025-07-05T01:10:29.297Z [WARN]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:29.297Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: entering candidate state: node="Node at 3ba32829-4cfc-689d-8bd3-6b7f04154fbc [Candidate]" term=2
2025-07-05T01:10:29.297Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: election won: term=2 tally=2
2025-07-05T01:10:29.297Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: entering leader state: leader="Node at 3ba32829-4cfc-689d-8bd3-6b7f04154fbc [Leader]"
2025-07-05T01:10:29.297Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: added peer, starting replication: peer=server-30e002a8-aaea-7757-b8c2-5583e55489e1
2025-07-05T01:10:29.297Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: added peer, starting replication: peer=server-9a6aebbd-de6c-e44b-8511-834dceb91265
2025-07-05T01:10:29.297Z [WARN]  server-9a6aebbd-de6c-e44b-8511-834dceb91265: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-07-05T01:10:29.297Z [WARN]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: appendEntries rejected, sending older logs: peer="{Voter server-9a6aebbd-de6c-e44b-8511-834dceb91265 9a6aebbd-de6c-e44b-8511-834dceb91265}" next=1
2025-07-05T01:10:29.297Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: pipelining replication: peer="{Voter server-9a6aebbd-de6c-e44b-8511-834dceb91265 9a6aebbd-de6c-e44b-8511-834dceb91265}"
2025-07-05T01:10:29.297Z [WARN]  server-30e002a8-aaea-7757-b8c2-5583e55489e1: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-07-05T01:10:29.297Z [WARN]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: appendEntries rejected, sending older logs: peer="{Voter server-30e002a8-aaea-7757-b8c2-5583e55489e1 30e002a8-aaea-7757-b8c2-5583e55489e1}" next=1
2025-07-05T01:10:29.297Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: pipelining replication: peer="{Voter server-30e002a8-aaea-7757-b8c2-5583e55489e1 30e002a8-aaea-7757-b8c2-5583e55489e1}"
2025-07-05T01:10:29.403Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:29.297630791 +0000 UTC m=+4.546367999 (2 nodes), 95.753558ms from start of poll, 95.942472ms from cluster start. Timeout at 2025-07-05 01:10:29.403673571 +0000 UTC m=+4.652410779, 106.04278ms after stability
2025-07-05T01:10:29.403Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:29.509Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:29.403779059 +0000 UTC m=+4.652516267 (1 nodes), 1.643µs from start of poll, 202.09074ms from cluster start. Timeout at 2025-07-05 01:10:29.509197138 +0000 UTC m=+4.757934346, 105.418079ms after stability
2025-07-05T01:10:29.525Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: aborting pipeline replication: peer="{Voter server-9a6aebbd-de6c-e44b-8511-834dceb91265 9a6aebbd-de6c-e44b-8511-834dceb91265}"
2025-07-05T01:10:29.525Z [INFO]  server-3ba32829-4cfc-689d-8bd3-6b7f04154fbc: aborting pipeline replication: peer="{Voter server-30e002a8-aaea-7757-b8c2-5583e55489e1 30e002a8-aaea-7757-b8c2-5583e55489e1}"
--- PASS: TestRaft_LiveBootstrap (0.32s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-07-05T01:10:29.526Z [INFO]  server-cc1c59b0-52e8-7f86-cfa1-93251872d8df: initial configuration: index=0 servers=[]
2025-07-05T01:10:29.526Z [INFO]  server-9e8d91d7-5d9c-0d42-eddc-1254d96f686c: initial configuration: index=0 servers=[]
2025-07-05T01:10:29.526Z [INFO]  server-cc1c59b0-52e8-7f86-cfa1-93251872d8df: entering follower state: follower="Node at cc1c59b0-52e8-7f86-cfa1-93251872d8df [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-07-05T01:10:29.527Z [INFO]  server-c71de794-72ee-2185-392f-fc4669e3b8fd: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-07-05T01:10:29.527Z [INFO]  server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b Address:52e320b3-ce70-b67f-5cd2-a2d61d579c0b} {Suffrage:Voter ID:server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57 Address:25fb08bb-7acc-37f1-0259-4e0cea3f4d57} {Suffrage:Voter ID:server-89fcd069-52d0-2abb-2505-a56fda1de5cc Address:89fcd069-52d0-2abb-2505-a56fda1de5cc}]"
2025-07-05T01:10:29.527Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b Address:52e320b3-ce70-b67f-5cd2-a2d61d579c0b} {Suffrage:Voter ID:server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57 Address:25fb08bb-7acc-37f1-0259-4e0cea3f4d57} {Suffrage:Voter ID:server-89fcd069-52d0-2abb-2505-a56fda1de5cc Address:89fcd069-52d0-2abb-2505-a56fda1de5cc}]"
2025-07-05T01:10:29.527Z [INFO]  server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b: entering follower state: follower="Node at 52e320b3-ce70-b67f-5cd2-a2d61d579c0b [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.527Z [INFO]  server-89fcd069-52d0-2abb-2505-a56fda1de5cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b Address:52e320b3-ce70-b67f-5cd2-a2d61d579c0b} {Suffrage:Voter ID:server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57 Address:25fb08bb-7acc-37f1-0259-4e0cea3f4d57} {Suffrage:Voter ID:server-89fcd069-52d0-2abb-2505-a56fda1de5cc Address:89fcd069-52d0-2abb-2505-a56fda1de5cc}]"
2025-07-05T01:10:29.527Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: entering follower state: follower="Node at 25fb08bb-7acc-37f1-0259-4e0cea3f4d57 [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.527Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:29.527Z [INFO]  server-89fcd069-52d0-2abb-2505-a56fda1de5cc: entering follower state: follower="Node at 89fcd069-52d0-2abb-2505-a56fda1de5cc [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.589Z [WARN]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:29.589Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: entering candidate state: node="Node at 25fb08bb-7acc-37f1-0259-4e0cea3f4d57 [Candidate]" term=2
2025-07-05T01:10:29.589Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: election won: term=2 tally=2
2025-07-05T01:10:29.589Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: entering leader state: leader="Node at 25fb08bb-7acc-37f1-0259-4e0cea3f4d57 [Leader]"
2025-07-05T01:10:29.589Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: added peer, starting replication: peer=server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b
2025-07-05T01:10:29.589Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: added peer, starting replication: peer=server-89fcd069-52d0-2abb-2505-a56fda1de5cc
2025-07-05T01:10:29.589Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: pipelining replication: peer="{Voter server-89fcd069-52d0-2abb-2505-a56fda1de5cc 89fcd069-52d0-2abb-2505-a56fda1de5cc}"
2025-07-05T01:10:29.590Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: pipelining replication: peer="{Voter server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b 52e320b3-ce70-b67f-5cd2-a2d61d579c0b}"
2025-07-05T01:10:29.695Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:29.590092597 +0000 UTC m=+4.838829805 (1 nodes), 62.214542ms from start of poll, 62.504316ms from cluster start. Timeout at 2025-07-05 01:10:29.695482884 +0000 UTC m=+4.944220092, 105.390287ms after stability
2025-07-05T01:10:29.695Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: aborting pipeline replication: peer="{Voter server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b 52e320b3-ce70-b67f-5cd2-a2d61d579c0b}"
2025-07-05T01:10:29.695Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: aborting pipeline replication: peer="{Voter server-89fcd069-52d0-2abb-2505-a56fda1de5cc 89fcd069-52d0-2abb-2505-a56fda1de5cc}"
2025-07-05T01:10:29.695Z [INFO]  creating new snapshot: path=/tmp/raft2534730332/snapshots/2-2-1751677829695.tmp
2025-07-05T01:10:29.696Z [INFO]  server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b: starting restore from snapshot: id=2-2-1751677829695 last-index=2 last-term=2 size-in-bytes=1
2025-07-05T01:10:29.696Z [INFO]  server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b: snapshot restore progress: id=2-2-1751677829695 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-07-05T01:10:29.696Z [INFO]  server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b: restored from snapshot: id=2-2-1751677829695 last-index=2 last-term=2 size-in-bytes=1
2025-07-05T01:10:29.696Z [INFO]  server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b Address:52e320b3-ce70-b67f-5cd2-a2d61d579c0b} {Suffrage:Voter ID:server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57 Address:25fb08bb-7acc-37f1-0259-4e0cea3f4d57} {Suffrage:Voter ID:server-89fcd069-52d0-2abb-2505-a56fda1de5cc Address:89fcd069-52d0-2abb-2505-a56fda1de5cc}]"
2025-07-05T01:10:29.696Z [INFO]  creating new snapshot: path=/tmp/raft984380616/snapshots/2-2-1751677829696.tmp
2025-07-05T01:10:29.696Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: starting restore from snapshot: id=2-2-1751677829696 last-index=2 last-term=2 size-in-bytes=1
2025-07-05T01:10:29.696Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: snapshot restore progress: id=2-2-1751677829696 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-07-05T01:10:29.696Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: restored from snapshot: id=2-2-1751677829696 last-index=2 last-term=2 size-in-bytes=1
2025-07-05T01:10:29.696Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b Address:52e320b3-ce70-b67f-5cd2-a2d61d579c0b} {Suffrage:Voter ID:server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57 Address:25fb08bb-7acc-37f1-0259-4e0cea3f4d57} {Suffrage:Voter ID:server-89fcd069-52d0-2abb-2505-a56fda1de5cc Address:89fcd069-52d0-2abb-2505-a56fda1de5cc}]"
2025-07-05T01:10:29.696Z [INFO]  creating new snapshot: path=/tmp/raft1471222516/snapshots/2-2-1751677829696.tmp
2025-07-05T01:10:29.697Z [INFO]  server-89fcd069-52d0-2abb-2505-a56fda1de5cc: starting restore from snapshot: id=2-2-1751677829696 last-index=2 last-term=2 size-in-bytes=1
2025-07-05T01:10:29.697Z [INFO]  server-89fcd069-52d0-2abb-2505-a56fda1de5cc: snapshot restore progress: id=2-2-1751677829696 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-07-05T01:10:29.697Z [INFO]  server-89fcd069-52d0-2abb-2505-a56fda1de5cc: restored from snapshot: id=2-2-1751677829696 last-index=2 last-term=2 size-in-bytes=1
2025-07-05T01:10:29.697Z [INFO]  server-89fcd069-52d0-2abb-2505-a56fda1de5cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b Address:52e320b3-ce70-b67f-5cd2-a2d61d579c0b} {Suffrage:Voter ID:server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57 Address:25fb08bb-7acc-37f1-0259-4e0cea3f4d57} {Suffrage:Voter ID:server-89fcd069-52d0-2abb-2505-a56fda1de5cc Address:89fcd069-52d0-2abb-2505-a56fda1de5cc}]"
2025-07-05T01:10:29.697Z [INFO]  server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b: entering follower state: follower="Node at 52e320b3-ce70-b67f-5cd2-a2d61d579c0b [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.697Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: entering follower state: follower="Node at 25fb08bb-7acc-37f1-0259-4e0cea3f4d57 [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.697Z [INFO]  server-89fcd069-52d0-2abb-2505-a56fda1de5cc: entering follower state: follower="Node at 89fcd069-52d0-2abb-2505-a56fda1de5cc [Follower]" leader-address= leader-id=
2025-07-05T01:10:29.751Z [WARN]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:29.751Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: entering candidate state: node="Node at 25fb08bb-7acc-37f1-0259-4e0cea3f4d57 [Candidate]" term=3
2025-07-05T01:10:29.751Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: election won: term=3 tally=2
2025-07-05T01:10:29.751Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: entering leader state: leader="Node at 25fb08bb-7acc-37f1-0259-4e0cea3f4d57 [Leader]"
2025-07-05T01:10:29.751Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: added peer, starting replication: peer=server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b
2025-07-05T01:10:29.751Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: added peer, starting replication: peer=server-89fcd069-52d0-2abb-2505-a56fda1de5cc
2025-07-05T01:10:29.751Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: pipelining replication: peer="{Voter server-89fcd069-52d0-2abb-2505-a56fda1de5cc 89fcd069-52d0-2abb-2505-a56fda1de5cc}"
2025-07-05T01:10:29.751Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: pipelining replication: peer="{Voter server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b 52e320b3-ce70-b67f-5cd2-a2d61d579c0b}"
2025-07-05T01:10:30.012Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:30.118Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:30.012934165 +0000 UTC m=+5.261671373 (1 nodes), 13.656µs from start of poll, 485.345884ms from cluster start. Timeout at 2025-07-05 01:10:30.118619525 +0000 UTC m=+5.367356743, 105.68537ms after stability
2025-07-05T01:10:30.118Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: aborting pipeline replication: peer="{Voter server-89fcd069-52d0-2abb-2505-a56fda1de5cc 89fcd069-52d0-2abb-2505-a56fda1de5cc}"
2025-07-05T01:10:30.118Z [INFO]  server-25fb08bb-7acc-37f1-0259-4e0cea3f4d57: aborting pipeline replication: peer="{Voter server-52e320b3-ce70-b67f-5cd2-a2d61d579c0b 52e320b3-ce70-b67f-5cd2-a2d61d579c0b}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-07-05T01:10:30.151Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f78c28-a3c3-dee0-0b2d-dff160114052 Address:c0f78c28-a3c3-dee0-0b2d-dff160114052} {Suffrage:Voter ID:server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 Address:4def10c4-8d7f-25f1-50a4-77e8958e9cd5} {Suffrage:Voter ID:server-dd9efd42-328c-6fb7-6297-15c5c67607fc Address:dd9efd42-328c-6fb7-6297-15c5c67607fc}]"
2025-07-05T01:10:30.151Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f78c28-a3c3-dee0-0b2d-dff160114052 Address:c0f78c28-a3c3-dee0-0b2d-dff160114052} {Suffrage:Voter ID:server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 Address:4def10c4-8d7f-25f1-50a4-77e8958e9cd5} {Suffrage:Voter ID:server-dd9efd42-328c-6fb7-6297-15c5c67607fc Address:dd9efd42-328c-6fb7-6297-15c5c67607fc}]"
2025-07-05T01:10:30.151Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: entering follower state: follower="Node at c0f78c28-a3c3-dee0-0b2d-dff160114052 [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.151Z [INFO]  server-dd9efd42-328c-6fb7-6297-15c5c67607fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f78c28-a3c3-dee0-0b2d-dff160114052 Address:c0f78c28-a3c3-dee0-0b2d-dff160114052} {Suffrage:Voter ID:server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 Address:4def10c4-8d7f-25f1-50a4-77e8958e9cd5} {Suffrage:Voter ID:server-dd9efd42-328c-6fb7-6297-15c5c67607fc Address:dd9efd42-328c-6fb7-6297-15c5c67607fc}]"
2025-07-05T01:10:30.151Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: entering follower state: follower="Node at 4def10c4-8d7f-25f1-50a4-77e8958e9cd5 [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.151Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:30.151Z [INFO]  server-dd9efd42-328c-6fb7-6297-15c5c67607fc: entering follower state: follower="Node at dd9efd42-328c-6fb7-6297-15c5c67607fc [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.209Z [WARN]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:30.209Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: entering candidate state: node="Node at c0f78c28-a3c3-dee0-0b2d-dff160114052 [Candidate]" term=2
2025-07-05T01:10:30.209Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: election won: term=2 tally=2
2025-07-05T01:10:30.209Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: entering leader state: leader="Node at c0f78c28-a3c3-dee0-0b2d-dff160114052 [Leader]"
2025-07-05T01:10:30.209Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: added peer, starting replication: peer=server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5
2025-07-05T01:10:30.209Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: added peer, starting replication: peer=server-dd9efd42-328c-6fb7-6297-15c5c67607fc
2025-07-05T01:10:30.209Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: pipelining replication: peer="{Voter server-dd9efd42-328c-6fb7-6297-15c5c67607fc dd9efd42-328c-6fb7-6297-15c5c67607fc}"
2025-07-05T01:10:30.209Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: pipelining replication: peer="{Voter server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 4def10c4-8d7f-25f1-50a4-77e8958e9cd5}"
2025-07-05T01:10:30.315Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:30.209985035 +0000 UTC m=+5.458722243 (1 nodes), 58.394459ms from start of poll, 58.678402ms from cluster start. Timeout at 2025-07-05 01:10:30.315533859 +0000 UTC m=+5.564271067, 105.548824ms after stability
2025-07-05T01:10:30.316Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: aborting pipeline replication: peer="{Voter server-dd9efd42-328c-6fb7-6297-15c5c67607fc dd9efd42-328c-6fb7-6297-15c5c67607fc}"
2025-07-05T01:10:30.316Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: aborting pipeline replication: peer="{Voter server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 4def10c4-8d7f-25f1-50a4-77e8958e9cd5}"
2025-07-05T01:10:30.316Z [INFO]  creating new snapshot: path=/tmp/raft3318324297/snapshots/2-6-1751677830316.tmp
2025-07-05T01:10:30.316Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: starting restore from snapshot: id=2-6-1751677830316 last-index=6 last-term=2 size-in-bytes=25
2025-07-05T01:10:30.317Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: snapshot restore progress: id=2-6-1751677830316 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-07-05T01:10:30.317Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: restored from snapshot: id=2-6-1751677830316 last-index=6 last-term=2 size-in-bytes=25
2025-07-05T01:10:30.317Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f78c28-a3c3-dee0-0b2d-dff160114052 Address:c0f78c28-a3c3-dee0-0b2d-dff160114052} {Suffrage:Voter ID:server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 Address:4def10c4-8d7f-25f1-50a4-77e8958e9cd5} {Suffrage:Voter ID:server-dd9efd42-328c-6fb7-6297-15c5c67607fc Address:dd9efd42-328c-6fb7-6297-15c5c67607fc}]"
2025-07-05T01:10:30.317Z [INFO]  creating new snapshot: path=/tmp/raft254042853/snapshots/2-6-1751677830317.tmp
2025-07-05T01:10:30.317Z [INFO]  server-c0f78c28-a3c3-dee0-0b2d-dff160114052: entering follower state: follower="Node at c0f78c28-a3c3-dee0-0b2d-dff160114052 [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.317Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: starting restore from snapshot: id=2-6-1751677830317 last-index=6 last-term=2 size-in-bytes=25
2025-07-05T01:10:30.318Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: snapshot restore progress: id=2-6-1751677830317 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-07-05T01:10:30.318Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: restored from snapshot: id=2-6-1751677830317 last-index=6 last-term=2 size-in-bytes=25
2025-07-05T01:10:30.318Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f78c28-a3c3-dee0-0b2d-dff160114052 Address:c0f78c28-a3c3-dee0-0b2d-dff160114052} {Suffrage:Voter ID:server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 Address:4def10c4-8d7f-25f1-50a4-77e8958e9cd5} {Suffrage:Voter ID:server-dd9efd42-328c-6fb7-6297-15c5c67607fc Address:dd9efd42-328c-6fb7-6297-15c5c67607fc}]"
2025-07-05T01:10:30.318Z [INFO]  creating new snapshot: path=/tmp/raft597626920/snapshots/2-6-1751677830318.tmp
2025-07-05T01:10:30.318Z [INFO]  server-dd9efd42-328c-6fb7-6297-15c5c67607fc: starting restore from snapshot: id=2-6-1751677830318 last-index=6 last-term=2 size-in-bytes=25
2025-07-05T01:10:30.318Z [INFO]  server-dd9efd42-328c-6fb7-6297-15c5c67607fc: snapshot restore progress: id=2-6-1751677830318 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-07-05T01:10:30.318Z [INFO]  server-dd9efd42-328c-6fb7-6297-15c5c67607fc: restored from snapshot: id=2-6-1751677830318 last-index=6 last-term=2 size-in-bytes=25
2025-07-05T01:10:30.318Z [INFO]  server-dd9efd42-328c-6fb7-6297-15c5c67607fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f78c28-a3c3-dee0-0b2d-dff160114052 Address:c0f78c28-a3c3-dee0-0b2d-dff160114052} {Suffrage:Voter ID:server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5 Address:4def10c4-8d7f-25f1-50a4-77e8958e9cd5} {Suffrage:Voter ID:server-dd9efd42-328c-6fb7-6297-15c5c67607fc Address:dd9efd42-328c-6fb7-6297-15c5c67607fc}]"
2025-07-05T01:10:30.318Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: entering follower state: follower="Node at 4def10c4-8d7f-25f1-50a4-77e8958e9cd5 [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.318Z [INFO]  server-dd9efd42-328c-6fb7-6297-15c5c67607fc: entering follower state: follower="Node at dd9efd42-328c-6fb7-6297-15c5c67607fc [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.400Z [WARN]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:30.400Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: entering candidate state: node="Node at 4def10c4-8d7f-25f1-50a4-77e8958e9cd5 [Candidate]" term=3
2025-07-05T01:10:30.400Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: election won: term=3 tally=2
2025-07-05T01:10:30.400Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: entering leader state: leader="Node at 4def10c4-8d7f-25f1-50a4-77e8958e9cd5 [Leader]"
2025-07-05T01:10:30.400Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: added peer, starting replication: peer=server-c0f78c28-a3c3-dee0-0b2d-dff160114052
2025-07-05T01:10:30.400Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: added peer, starting replication: peer=server-dd9efd42-328c-6fb7-6297-15c5c67607fc
2025-07-05T01:10:30.400Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: pipelining replication: peer="{Voter server-c0f78c28-a3c3-dee0-0b2d-dff160114052 c0f78c28-a3c3-dee0-0b2d-dff160114052}"
2025-07-05T01:10:30.400Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: pipelining replication: peer="{Voter server-dd9efd42-328c-6fb7-6297-15c5c67607fc dd9efd42-328c-6fb7-6297-15c5c67607fc}"
2025-07-05T01:10:30.634Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:30.739Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:30.634166215 +0000 UTC m=+5.882903423 (1 nodes), 18.575µs from start of poll, 482.859582ms from cluster start. Timeout at 2025-07-05 01:10:30.739390881 +0000 UTC m=+5.988128099, 105.224676ms after stability
2025-07-05T01:10:30.739Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: aborting pipeline replication: peer="{Voter server-dd9efd42-328c-6fb7-6297-15c5c67607fc dd9efd42-328c-6fb7-6297-15c5c67607fc}"
2025-07-05T01:10:30.739Z [INFO]  server-4def10c4-8d7f-25f1-50a4-77e8958e9cd5: aborting pipeline replication: peer="{Voter server-c0f78c28-a3c3-dee0-0b2d-dff160114052 c0f78c28-a3c3-dee0-0b2d-dff160114052}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-07-05T01:10:30.740Z [INFO]  server-79d88d06-41d2-cc09-1224-429daf099252: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79d88d06-41d2-cc09-1224-429daf099252 Address:79d88d06-41d2-cc09-1224-429daf099252} {Suffrage:Voter ID:server-9e5da024-528b-e04f-fb4a-10d009f5bf35 Address:9e5da024-528b-e04f-fb4a-10d009f5bf35} {Suffrage:Voter ID:server-6d026b4d-8ada-9d24-7563-a508c63bcafa Address:6d026b4d-8ada-9d24-7563-a508c63bcafa}]"
2025-07-05T01:10:30.740Z [INFO]  server-9e5da024-528b-e04f-fb4a-10d009f5bf35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79d88d06-41d2-cc09-1224-429daf099252 Address:79d88d06-41d2-cc09-1224-429daf099252} {Suffrage:Voter ID:server-9e5da024-528b-e04f-fb4a-10d009f5bf35 Address:9e5da024-528b-e04f-fb4a-10d009f5bf35} {Suffrage:Voter ID:server-6d026b4d-8ada-9d24-7563-a508c63bcafa Address:6d026b4d-8ada-9d24-7563-a508c63bcafa}]"
2025-07-05T01:10:30.740Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79d88d06-41d2-cc09-1224-429daf099252 Address:79d88d06-41d2-cc09-1224-429daf099252} {Suffrage:Voter ID:server-9e5da024-528b-e04f-fb4a-10d009f5bf35 Address:9e5da024-528b-e04f-fb4a-10d009f5bf35} {Suffrage:Voter ID:server-6d026b4d-8ada-9d24-7563-a508c63bcafa Address:6d026b4d-8ada-9d24-7563-a508c63bcafa}]"
2025-07-05T01:10:30.740Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:30.740Z [INFO]  server-79d88d06-41d2-cc09-1224-429daf099252: entering follower state: follower="Node at 79d88d06-41d2-cc09-1224-429daf099252 [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.741Z [INFO]  server-9e5da024-528b-e04f-fb4a-10d009f5bf35: entering follower state: follower="Node at 9e5da024-528b-e04f-fb4a-10d009f5bf35 [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.741Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: entering follower state: follower="Node at 6d026b4d-8ada-9d24-7563-a508c63bcafa [Follower]" leader-address= leader-id=
2025-07-05T01:10:30.799Z [WARN]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:30.799Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: entering candidate state: node="Node at 6d026b4d-8ada-9d24-7563-a508c63bcafa [Candidate]" term=2
2025-07-05T01:10:30.799Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: election won: term=2 tally=2
2025-07-05T01:10:30.799Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: entering leader state: leader="Node at 6d026b4d-8ada-9d24-7563-a508c63bcafa [Leader]"
2025-07-05T01:10:30.799Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: added peer, starting replication: peer=server-79d88d06-41d2-cc09-1224-429daf099252
2025-07-05T01:10:30.799Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: added peer, starting replication: peer=server-9e5da024-528b-e04f-fb4a-10d009f5bf35
2025-07-05T01:10:30.799Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: pipelining replication: peer="{Voter server-9e5da024-528b-e04f-fb4a-10d009f5bf35 9e5da024-528b-e04f-fb4a-10d009f5bf35}"
2025-07-05T01:10:30.800Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: pipelining replication: peer="{Voter server-79d88d06-41d2-cc09-1224-429daf099252 79d88d06-41d2-cc09-1224-429daf099252}"
2025-07-05T01:10:30.905Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:30.800120699 +0000 UTC m=+6.048857907 (1 nodes), 59.150357ms from start of poll, 59.36572ms from cluster start. Timeout at 2025-07-05 01:10:30.905197919 +0000 UTC m=+6.153935137, 105.07723ms after stability
2025-07-05T01:10:30.906Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: aborting pipeline replication: peer="{Voter server-79d88d06-41d2-cc09-1224-429daf099252 79d88d06-41d2-cc09-1224-429daf099252}"
2025-07-05T01:10:30.910Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: aborting pipeline replication: peer="{Voter server-9e5da024-528b-e04f-fb4a-10d009f5bf35 9e5da024-528b-e04f-fb4a-10d009f5bf35}"
2025-07-05T01:10:31.411Z [ERROR] server-6d026b4d-8ada-9d24-7563-a508c63bcafa: failed to heartbeat to: peer=9e5da024-528b-e04f-fb4a-10d009f5bf35 backoff time=10ms error="command timed out"
2025-07-05T01:10:31.411Z [INFO]  creating new snapshot: path=/tmp/raft909495107/snapshots/2-27-1751677831411.tmp
2025-07-05T01:10:31.412Z [INFO]  server-79d88d06-41d2-cc09-1224-429daf099252: starting restore from snapshot: id=2-27-1751677831411 last-index=27 last-term=2 size-in-bytes=168
2025-07-05T01:10:31.412Z [INFO]  server-79d88d06-41d2-cc09-1224-429daf099252: snapshot restore progress: id=2-27-1751677831411 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-07-05T01:10:31.412Z [INFO]  server-79d88d06-41d2-cc09-1224-429daf099252: restored from snapshot: id=2-27-1751677831411 last-index=27 last-term=2 size-in-bytes=168
2025-07-05T01:10:31.412Z [INFO]  server-79d88d06-41d2-cc09-1224-429daf099252: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79d88d06-41d2-cc09-1224-429daf099252 Address:79d88d06-41d2-cc09-1224-429daf099252} {Suffrage:Voter ID:server-9e5da024-528b-e04f-fb4a-10d009f5bf35 Address:9e5da024-528b-e04f-fb4a-10d009f5bf35} {Suffrage:Voter ID:server-6d026b4d-8ada-9d24-7563-a508c63bcafa Address:6d026b4d-8ada-9d24-7563-a508c63bcafa}]"
2025-07-05T01:10:31.412Z [INFO]  creating new snapshot: path=/tmp/raft3242697690/snapshots/2-27-1751677831412.tmp
2025-07-05T01:10:31.412Z [INFO]  server-79d88d06-41d2-cc09-1224-429daf099252: entering follower state: follower="Node at 79d88d06-41d2-cc09-1224-429daf099252 [Follower]" leader-address= leader-id=
2025-07-05T01:10:31.413Z [INFO]  server-9e5da024-528b-e04f-fb4a-10d009f5bf35: starting restore from snapshot: id=2-27-1751677831412 last-index=27 last-term=2 size-in-bytes=168
2025-07-05T01:10:31.413Z [INFO]  server-9e5da024-528b-e04f-fb4a-10d009f5bf35: snapshot restore progress: id=2-27-1751677831412 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-07-05T01:10:31.413Z [INFO]  server-9e5da024-528b-e04f-fb4a-10d009f5bf35: restored from snapshot: id=2-27-1751677831412 last-index=27 last-term=2 size-in-bytes=168
2025-07-05T01:10:31.413Z [INFO]  server-9e5da024-528b-e04f-fb4a-10d009f5bf35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79d88d06-41d2-cc09-1224-429daf099252 Address:79d88d06-41d2-cc09-1224-429daf099252} {Suffrage:Voter ID:server-9e5da024-528b-e04f-fb4a-10d009f5bf35 Address:9e5da024-528b-e04f-fb4a-10d009f5bf35} {Suffrage:Voter ID:server-6d026b4d-8ada-9d24-7563-a508c63bcafa Address:6d026b4d-8ada-9d24-7563-a508c63bcafa}]"
2025-07-05T01:10:31.413Z [INFO]  creating new snapshot: path=/tmp/raft1444127465/snapshots/2-27-1751677831413.tmp
2025-07-05T01:10:31.413Z [INFO]  server-9e5da024-528b-e04f-fb4a-10d009f5bf35: entering follower state: follower="Node at 9e5da024-528b-e04f-fb4a-10d009f5bf35 [Follower]" leader-address= leader-id=
2025-07-05T01:10:31.413Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: starting restore from snapshot: id=2-27-1751677831413 last-index=27 last-term=2 size-in-bytes=168
2025-07-05T01:10:31.413Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: snapshot restore progress: id=2-27-1751677831413 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-07-05T01:10:31.413Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: restored from snapshot: id=2-27-1751677831413 last-index=27 last-term=2 size-in-bytes=168
2025-07-05T01:10:31.413Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79d88d06-41d2-cc09-1224-429daf099252 Address:79d88d06-41d2-cc09-1224-429daf099252} {Suffrage:Voter ID:server-9e5da024-528b-e04f-fb4a-10d009f5bf35 Address:9e5da024-528b-e04f-fb4a-10d009f5bf35} {Suffrage:Voter ID:server-6d026b4d-8ada-9d24-7563-a508c63bcafa Address:6d026b4d-8ada-9d24-7563-a508c63bcafa}]"
2025-07-05T01:10:31.413Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: entering follower state: follower="Node at 6d026b4d-8ada-9d24-7563-a508c63bcafa [Follower]" leader-address= leader-id=
2025-07-05T01:10:31.468Z [WARN]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:31.468Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: entering candidate state: node="Node at 6d026b4d-8ada-9d24-7563-a508c63bcafa [Candidate]" term=3
2025-07-05T01:10:31.468Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: election won: term=3 tally=2
2025-07-05T01:10:31.468Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: entering leader state: leader="Node at 6d026b4d-8ada-9d24-7563-a508c63bcafa [Leader]"
2025-07-05T01:10:31.468Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: added peer, starting replication: peer=server-79d88d06-41d2-cc09-1224-429daf099252
2025-07-05T01:10:31.468Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: added peer, starting replication: peer=server-9e5da024-528b-e04f-fb4a-10d009f5bf35
2025-07-05T01:10:31.469Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: pipelining replication: peer="{Voter server-79d88d06-41d2-cc09-1224-429daf099252 79d88d06-41d2-cc09-1224-429daf099252}"
2025-07-05T01:10:31.469Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: pipelining replication: peer="{Voter server-9e5da024-528b-e04f-fb4a-10d009f5bf35 9e5da024-528b-e04f-fb4a-10d009f5bf35}"
2025-07-05T01:10:31.729Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:31.834Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:31.729465831 +0000 UTC m=+6.978203039 (1 nodes), 16.892µs from start of poll, 988.710852ms from cluster start. Timeout at 2025-07-05 01:10:31.834684045 +0000 UTC m=+7.083421253, 105.218214ms after stability
2025-07-05T01:10:31.834Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: aborting pipeline replication: peer="{Voter server-9e5da024-528b-e04f-fb4a-10d009f5bf35 9e5da024-528b-e04f-fb4a-10d009f5bf35}"
2025-07-05T01:10:31.834Z [INFO]  server-6d026b4d-8ada-9d24-7563-a508c63bcafa: aborting pipeline replication: peer="{Voter server-79d88d06-41d2-cc09-1224-429daf099252 79d88d06-41d2-cc09-1224-429daf099252}"
--- PASS: TestRaft_RecoverCluster (2.31s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.62s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.59s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (1.10s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-07-05T01:10:31.836Z [INFO]  server-ea0cc2c5-36b8-4678-6581-cb01ca51b55d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea0cc2c5-36b8-4678-6581-cb01ca51b55d Address:ea0cc2c5-36b8-4678-6581-cb01ca51b55d}]"
2025-07-05T01:10:31.836Z [INFO]  server-ea0cc2c5-36b8-4678-6581-cb01ca51b55d: entering follower state: follower="Node at ea0cc2c5-36b8-4678-6581-cb01ca51b55d [Follower]" leader-address= leader-id=
2025-07-05T01:10:31.927Z [WARN]  server-ea0cc2c5-36b8-4678-6581-cb01ca51b55d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:31.927Z [INFO]  server-ea0cc2c5-36b8-4678-6581-cb01ca51b55d: entering candidate state: node="Node at ea0cc2c5-36b8-4678-6581-cb01ca51b55d [Candidate]" term=2
2025-07-05T01:10:31.927Z [INFO]  server-ea0cc2c5-36b8-4678-6581-cb01ca51b55d: election won: term=2 tally=1
2025-07-05T01:10:31.927Z [INFO]  server-ea0cc2c5-36b8-4678-6581-cb01ca51b55d: entering leader state: leader="Node at ea0cc2c5-36b8-4678-6581-cb01ca51b55d [Leader]"
--- PASS: TestRaft_SingleNode (0.09s)
=== RUN   TestRaft_TripleNode
2025-07-05T01:10:31.928Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d58e064-748b-5930-a069-73915c2ed70b Address:8d58e064-748b-5930-a069-73915c2ed70b} {Suffrage:Voter ID:server-44205b31-92db-e0c4-f93e-e196df1ee524 Address:44205b31-92db-e0c4-f93e-e196df1ee524} {Suffrage:Voter ID:server-f32da9d4-97a8-f037-ade4-92efda01d95a Address:f32da9d4-97a8-f037-ade4-92efda01d95a}]"
2025-07-05T01:10:31.928Z [INFO]  server-44205b31-92db-e0c4-f93e-e196df1ee524: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d58e064-748b-5930-a069-73915c2ed70b Address:8d58e064-748b-5930-a069-73915c2ed70b} {Suffrage:Voter ID:server-44205b31-92db-e0c4-f93e-e196df1ee524 Address:44205b31-92db-e0c4-f93e-e196df1ee524} {Suffrage:Voter ID:server-f32da9d4-97a8-f037-ade4-92efda01d95a Address:f32da9d4-97a8-f037-ade4-92efda01d95a}]"
2025-07-05T01:10:31.928Z [INFO]  server-f32da9d4-97a8-f037-ade4-92efda01d95a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d58e064-748b-5930-a069-73915c2ed70b Address:8d58e064-748b-5930-a069-73915c2ed70b} {Suffrage:Voter ID:server-44205b31-92db-e0c4-f93e-e196df1ee524 Address:44205b31-92db-e0c4-f93e-e196df1ee524} {Suffrage:Voter ID:server-f32da9d4-97a8-f037-ade4-92efda01d95a Address:f32da9d4-97a8-f037-ade4-92efda01d95a}]"
2025-07-05T01:10:31.928Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:31.928Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: entering follower state: follower="Node at 8d58e064-748b-5930-a069-73915c2ed70b [Follower]" leader-address= leader-id=
2025-07-05T01:10:31.928Z [INFO]  server-44205b31-92db-e0c4-f93e-e196df1ee524: entering follower state: follower="Node at 44205b31-92db-e0c4-f93e-e196df1ee524 [Follower]" leader-address= leader-id=
2025-07-05T01:10:31.928Z [INFO]  server-f32da9d4-97a8-f037-ade4-92efda01d95a: entering follower state: follower="Node at f32da9d4-97a8-f037-ade4-92efda01d95a [Follower]" leader-address= leader-id=
2025-07-05T01:10:31.982Z [WARN]  server-8d58e064-748b-5930-a069-73915c2ed70b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:31.982Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: entering candidate state: node="Node at 8d58e064-748b-5930-a069-73915c2ed70b [Candidate]" term=2
2025-07-05T01:10:31.982Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: election won: term=2 tally=2
2025-07-05T01:10:31.982Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: entering leader state: leader="Node at 8d58e064-748b-5930-a069-73915c2ed70b [Leader]"
2025-07-05T01:10:31.982Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: added peer, starting replication: peer=server-44205b31-92db-e0c4-f93e-e196df1ee524
2025-07-05T01:10:31.982Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: added peer, starting replication: peer=server-f32da9d4-97a8-f037-ade4-92efda01d95a
2025-07-05T01:10:31.982Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: pipelining replication: peer="{Voter server-f32da9d4-97a8-f037-ade4-92efda01d95a f32da9d4-97a8-f037-ade4-92efda01d95a}"
2025-07-05T01:10:31.982Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: pipelining replication: peer="{Voter server-44205b31-92db-e0c4-f93e-e196df1ee524 44205b31-92db-e0c4-f93e-e196df1ee524}"
2025-07-05T01:10:32.088Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:31.982779344 +0000 UTC m=+7.231516552 (2 nodes), 54.050114ms from start of poll, 54.296897ms from cluster start. Timeout at 2025-07-05 01:10:32.088759336 +0000 UTC m=+7.337496544, 105.979992ms after stability
2025-07-05T01:10:32.088Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:32.194Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:32.088903216 +0000 UTC m=+7.337640424 (1 nodes), 1.543µs from start of poll, 160.420769ms from cluster start. Timeout at 2025-07-05 01:10:32.194634532 +0000 UTC m=+7.443371740, 105.731316ms after stability
2025-07-05T01:10:32.205Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: aborting pipeline replication: peer="{Voter server-f32da9d4-97a8-f037-ade4-92efda01d95a f32da9d4-97a8-f037-ade4-92efda01d95a}"
2025-07-05T01:10:32.205Z [INFO]  server-8d58e064-748b-5930-a069-73915c2ed70b: aborting pipeline replication: peer="{Voter server-44205b31-92db-e0c4-f93e-e196df1ee524 44205b31-92db-e0c4-f93e-e196df1ee524}"
--- PASS: TestRaft_TripleNode (0.28s)
=== RUN   TestRaft_LeaderFail
2025-07-05T01:10:32.211Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba26670e-28cc-40b8-1255-bb79b53db79f Address:ba26670e-28cc-40b8-1255-bb79b53db79f} {Suffrage:Voter ID:server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 Address:be3ae6fc-3436-8f00-090c-1bfaa23d5f50} {Suffrage:Voter ID:server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 Address:2f4c16a9-53ef-4866-1321-72fdf04a43f7}]"
2025-07-05T01:10:32.211Z [INFO]  server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba26670e-28cc-40b8-1255-bb79b53db79f Address:ba26670e-28cc-40b8-1255-bb79b53db79f} {Suffrage:Voter ID:server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 Address:be3ae6fc-3436-8f00-090c-1bfaa23d5f50} {Suffrage:Voter ID:server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 Address:2f4c16a9-53ef-4866-1321-72fdf04a43f7}]"
2025-07-05T01:10:32.211Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba26670e-28cc-40b8-1255-bb79b53db79f Address:ba26670e-28cc-40b8-1255-bb79b53db79f} {Suffrage:Voter ID:server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 Address:be3ae6fc-3436-8f00-090c-1bfaa23d5f50} {Suffrage:Voter ID:server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 Address:2f4c16a9-53ef-4866-1321-72fdf04a43f7}]"
2025-07-05T01:10:32.211Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:32.211Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: entering follower state: follower="Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.211Z [INFO]  server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50: entering follower state: follower="Node at be3ae6fc-3436-8f00-090c-1bfaa23d5f50 [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.211Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: entering follower state: follower="Node at 2f4c16a9-53ef-4866-1321-72fdf04a43f7 [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.272Z [WARN]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:32.272Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: entering candidate state: node="Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Candidate]" term=2
2025-07-05T01:10:32.272Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: election won: term=2 tally=2
2025-07-05T01:10:32.272Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: entering leader state: leader="Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Leader]"
2025-07-05T01:10:32.272Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: added peer, starting replication: peer=server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50
2025-07-05T01:10:32.272Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: added peer, starting replication: peer=server-2f4c16a9-53ef-4866-1321-72fdf04a43f7
2025-07-05T01:10:32.272Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: pipelining replication: peer="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}"
2025-07-05T01:10:32.272Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: pipelining replication: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}"
2025-07-05T01:10:32.378Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:32.27332685 +0000 UTC m=+7.522064068 (2 nodes), 61.929217ms from start of poll, 62.235161ms from cluster start. Timeout at 2025-07-05 01:10:32.378427243 +0000 UTC m=+7.627164462, 105.100394ms after stability
2025-07-05T01:10:32.378Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:32.483Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:32.378496133 +0000 UTC m=+7.627233351 (1 nodes), 1.734µs from start of poll, 167.404444ms from cluster start. Timeout at 2025-07-05 01:10:32.483519321 +0000 UTC m=+7.732256539, 105.023188ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Leader]
2025-07-05T01:10:32.499Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to heartbeat to: peer=be3ae6fc-3436-8f00-090c-1bfaa23d5f50 backoff time=10ms error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50"
2025-07-05T01:10:32.499Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:32.499Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to pipeline appendEntries: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}" error="append pipeline closed"
2025-07-05T01:10:32.499Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: aborting pipeline replication: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}"
2025-07-05T01:10:32.501Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to heartbeat to: peer=2f4c16a9-53ef-4866-1321-72fdf04a43f7 backoff time=10ms error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7"
2025-07-05T01:10:32.502Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to pipeline appendEntries: peer="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}" error="append pipeline closed"
2025-07-05T01:10:32.502Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: aborting pipeline replication: peer="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}"
2025-07-05T01:10:32.505Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to appendEntries to: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}" error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50"
2025-07-05T01:10:32.513Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to appendEntries to: peer="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}" error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7"
2025-07-05T01:10:32.516Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to heartbeat to: peer=be3ae6fc-3436-8f00-090c-1bfaa23d5f50 backoff time=10ms error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50"
2025-07-05T01:10:32.520Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to heartbeat to: peer=2f4c16a9-53ef-4866-1321-72fdf04a43f7 backoff time=10ms error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7"
2025-07-05T01:10:32.522Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to appendEntries to: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}" error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50"
2025-07-05T01:10:32.531Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to appendEntries to: peer="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}" error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7"
2025-07-05T01:10:32.534Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to heartbeat to: peer=be3ae6fc-3436-8f00-090c-1bfaa23d5f50 backoff time=10ms error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50"
2025-07-05T01:10:32.540Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to appendEntries to: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}" error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50"
2025-07-05T01:10:32.542Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to heartbeat to: peer=2f4c16a9-53ef-4866-1321-72fdf04a43f7 backoff time=10ms error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7"
2025-07-05T01:10:32.543Z [WARN]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to contact: server-id=server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 time=51.000245ms
2025-07-05T01:10:32.543Z [WARN]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to contact: server-id=server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 time=50.977102ms
2025-07-05T01:10:32.543Z [WARN]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to contact quorum of nodes, stepping down
2025-07-05T01:10:32.543Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: entering follower state: follower="Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.551Z [WARN]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: heartbeat timeout reached, starting election: last-leader-addr=ba26670e-28cc-40b8-1255-bb79b53db79f last-leader-id=server-ba26670e-28cc-40b8-1255-bb79b53db79f
2025-07-05T01:10:32.551Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: entering candidate state: node="Node at 2f4c16a9-53ef-4866-1321-72fdf04a43f7 [Candidate]" term=3
2025-07-05T01:10:32.551Z [WARN]  server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50: rejecting vote request since we have a leader: from=2f4c16a9-53ef-4866-1321-72fdf04a43f7 leader=ba26670e-28cc-40b8-1255-bb79b53db79f leader-id=server-ba26670e-28cc-40b8-1255-bb79b53db79f
2025-07-05T01:10:32.551Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to appendEntries to: peer="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}" error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7"
2025-07-05T01:10:32.551Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to make requestVote RPC: target="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}" error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f" term=3
2025-07-05T01:10:32.592Z [WARN]  server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50: heartbeat timeout reached, starting election: last-leader-addr=ba26670e-28cc-40b8-1255-bb79b53db79f last-leader-id=server-ba26670e-28cc-40b8-1255-bb79b53db79f
2025-07-05T01:10:32.592Z [INFO]  server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50: entering candidate state: node="Node at be3ae6fc-3436-8f00-090c-1bfaa23d5f50 [Candidate]" term=3
2025-07-05T01:10:32.592Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: duplicate requestVote for same term: term=3
2025-07-05T01:10:32.592Z [ERROR] server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50: failed to make requestVote RPC: target="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}" error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f" term=3
2025-07-05T01:10:32.613Z [WARN]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:32.613Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: entering candidate state: node="Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Candidate]" term=3
2025-07-05T01:10:32.614Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to make requestVote RPC: target="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}" error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7" term=3
2025-07-05T01:10:32.614Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to make requestVote RPC: target="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}" error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50" term=3
2025-07-05T01:10:32.640Z [WARN]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: Election timeout reached, restarting election
2025-07-05T01:10:32.640Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: entering candidate state: node="Node at 2f4c16a9-53ef-4866-1321-72fdf04a43f7 [Candidate]" term=4
2025-07-05T01:10:32.640Z [INFO]  server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50: entering follower state: follower="Node at be3ae6fc-3436-8f00-090c-1bfaa23d5f50 [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.640Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: election won: term=4 tally=2
2025-07-05T01:10:32.640Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: entering leader state: leader="Node at 2f4c16a9-53ef-4866-1321-72fdf04a43f7 [Leader]"
2025-07-05T01:10:32.640Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: added peer, starting replication: peer=server-ba26670e-28cc-40b8-1255-bb79b53db79f
2025-07-05T01:10:32.640Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: added peer, starting replication: peer=server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50
2025-07-05T01:10:32.640Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: pipelining replication: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}"
2025-07-05T01:10:32.640Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to make requestVote RPC: target="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}" error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f" term=4
2025-07-05T01:10:32.640Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to appendEntries to: peer="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}" error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.649Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to heartbeat to: peer=ba26670e-28cc-40b8-1255-bb79b53db79f backoff time=10ms error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.661Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to appendEntries to: peer="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}" error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.670Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to heartbeat to: peer=ba26670e-28cc-40b8-1255-bb79b53db79f backoff time=10ms error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.678Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to appendEntries to: peer="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}" error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.689Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to heartbeat to: peer=ba26670e-28cc-40b8-1255-bb79b53db79f backoff time=10ms error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.690Z [WARN]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to contact: server-id=server-ba26670e-28cc-40b8-1255-bb79b53db79f time=50.288421ms
2025-07-05T01:10:32.694Z [WARN]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: Election timeout reached, restarting election
2025-07-05T01:10:32.694Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: entering candidate state: node="Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Candidate]" term=4
2025-07-05T01:10:32.694Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to make requestVote RPC: target="{Voter server-2f4c16a9-53ef-4866-1321-72fdf04a43f7 2f4c16a9-53ef-4866-1321-72fdf04a43f7}" error="failed to connect to peer: 2f4c16a9-53ef-4866-1321-72fdf04a43f7" term=4
2025-07-05T01:10:32.695Z [ERROR] server-ba26670e-28cc-40b8-1255-bb79b53db79f: failed to make requestVote RPC: target="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}" error="failed to connect to peer: be3ae6fc-3436-8f00-090c-1bfaa23d5f50" term=4
2025-07-05T01:10:32.705Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to appendEntries to: peer="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}" error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.708Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to heartbeat to: peer=ba26670e-28cc-40b8-1255-bb79b53db79f backoff time=20ms error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.735Z [ERROR] server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to heartbeat to: peer=ba26670e-28cc-40b8-1255-bb79b53db79f backoff time=25ms error="failed to connect to peer: ba26670e-28cc-40b8-1255-bb79b53db79f"
2025-07-05T01:10:32.736Z [WARN]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: failed to contact: server-id=server-ba26670e-28cc-40b8-1255-bb79b53db79f time=96.009959ms
2025-07-05T01:10:32.746Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:32.640956325 +0000 UTC m=+7.889693543 (1 nodes), 141.823489ms from start of poll, 429.864636ms from cluster start. Timeout at 2025-07-05 01:10:32.746229703 +0000 UTC m=+7.994966911, 105.273368ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Candidate]
2025-07-05T01:10:32.752Z [INFO]  server-ba26670e-28cc-40b8-1255-bb79b53db79f: entering follower state: follower="Node at ba26670e-28cc-40b8-1255-bb79b53db79f [Follower]" leader-address=2f4c16a9-53ef-4866-1321-72fdf04a43f7 leader-id=server-2f4c16a9-53ef-4866-1321-72fdf04a43f7
2025-07-05T01:10:32.752Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: pipelining replication: peer="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}"
2025-07-05T01:10:32.758Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: aborting pipeline replication: peer="{Voter server-be3ae6fc-3436-8f00-090c-1bfaa23d5f50 be3ae6fc-3436-8f00-090c-1bfaa23d5f50}"
2025-07-05T01:10:32.758Z [INFO]  server-2f4c16a9-53ef-4866-1321-72fdf04a43f7: aborting pipeline replication: peer="{Voter server-ba26670e-28cc-40b8-1255-bb79b53db79f ba26670e-28cc-40b8-1255-bb79b53db79f}"
--- PASS: TestRaft_LeaderFail (0.55s)
=== RUN   TestRaft_BehindFollower
2025-07-05T01:10:32.760Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe Address:e91b9c87-d881-a4af-b2c0-4deaa7b610fe} {Suffrage:Voter ID:server-07097d3f-2883-7843-8df2-7cab68d4b0c0 Address:07097d3f-2883-7843-8df2-7cab68d4b0c0} {Suffrage:Voter ID:server-2a02b826-95b6-ea47-4d8f-5888d63cae44 Address:2a02b826-95b6-ea47-4d8f-5888d63cae44}]"
2025-07-05T01:10:32.760Z [INFO]  server-07097d3f-2883-7843-8df2-7cab68d4b0c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe Address:e91b9c87-d881-a4af-b2c0-4deaa7b610fe} {Suffrage:Voter ID:server-07097d3f-2883-7843-8df2-7cab68d4b0c0 Address:07097d3f-2883-7843-8df2-7cab68d4b0c0} {Suffrage:Voter ID:server-2a02b826-95b6-ea47-4d8f-5888d63cae44 Address:2a02b826-95b6-ea47-4d8f-5888d63cae44}]"
2025-07-05T01:10:32.760Z [INFO]  server-2a02b826-95b6-ea47-4d8f-5888d63cae44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe Address:e91b9c87-d881-a4af-b2c0-4deaa7b610fe} {Suffrage:Voter ID:server-07097d3f-2883-7843-8df2-7cab68d4b0c0 Address:07097d3f-2883-7843-8df2-7cab68d4b0c0} {Suffrage:Voter ID:server-2a02b826-95b6-ea47-4d8f-5888d63cae44 Address:2a02b826-95b6-ea47-4d8f-5888d63cae44}]"
2025-07-05T01:10:32.760Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:32.760Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: entering follower state: follower="Node at e91b9c87-d881-a4af-b2c0-4deaa7b610fe [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.760Z [INFO]  server-07097d3f-2883-7843-8df2-7cab68d4b0c0: entering follower state: follower="Node at 07097d3f-2883-7843-8df2-7cab68d4b0c0 [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.760Z [INFO]  server-2a02b826-95b6-ea47-4d8f-5888d63cae44: entering follower state: follower="Node at 2a02b826-95b6-ea47-4d8f-5888d63cae44 [Follower]" leader-address= leader-id=
2025-07-05T01:10:32.827Z [WARN]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:32.827Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: entering candidate state: node="Node at e91b9c87-d881-a4af-b2c0-4deaa7b610fe [Candidate]" term=2
2025-07-05T01:10:32.827Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: election won: term=2 tally=2
2025-07-05T01:10:32.827Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: entering leader state: leader="Node at e91b9c87-d881-a4af-b2c0-4deaa7b610fe [Leader]"
2025-07-05T01:10:32.827Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: added peer, starting replication: peer=server-07097d3f-2883-7843-8df2-7cab68d4b0c0
2025-07-05T01:10:32.827Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: added peer, starting replication: peer=server-2a02b826-95b6-ea47-4d8f-5888d63cae44
2025-07-05T01:10:32.828Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: pipelining replication: peer="{Voter server-2a02b826-95b6-ea47-4d8f-5888d63cae44 2a02b826-95b6-ea47-4d8f-5888d63cae44}"
2025-07-05T01:10:32.828Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: pipelining replication: peer="{Voter server-07097d3f-2883-7843-8df2-7cab68d4b0c0 07097d3f-2883-7843-8df2-7cab68d4b0c0}"
2025-07-05T01:10:32.933Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:32.828176563 +0000 UTC m=+8.076913771 (1 nodes), 68.028583ms from start of poll, 68.226174ms from cluster start. Timeout at 2025-07-05 01:10:32.93365202 +0000 UTC m=+8.182389228, 105.475457ms after stability
2025-07-05T01:10:32.933Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:33.039Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:32.933782033 +0000 UTC m=+8.182519242 (2 nodes), 1.022µs from start of poll, 173.831645ms from cluster start. Timeout at 2025-07-05 01:10:33.039243834 +0000 UTC m=+8.287981052, 105.46181ms after stability
2025-07-05T01:10:33.039Z [ERROR] server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: failed to pipeline appendEntries: peer="{Voter server-07097d3f-2883-7843-8df2-7cab68d4b0c0 07097d3f-2883-7843-8df2-7cab68d4b0c0}" error="append pipeline closed"
2025-07-05T01:10:33.039Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: aborting pipeline replication: peer="{Voter server-07097d3f-2883-7843-8df2-7cab68d4b0c0 07097d3f-2883-7843-8df2-7cab68d4b0c0}"
2025-07-05T01:10:33.039Z [ERROR] server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: failed to appendEntries to: peer="{Voter server-07097d3f-2883-7843-8df2-7cab68d4b0c0 07097d3f-2883-7843-8df2-7cab68d4b0c0}" error="failed to connect to peer: 07097d3f-2883-7843-8df2-7cab68d4b0c0"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-07-05T01:10:33.049Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: pipelining replication: peer="{Voter server-07097d3f-2883-7843-8df2-7cab68d4b0c0 07097d3f-2883-7843-8df2-7cab68d4b0c0}"
2025-07-05T01:10:33.060Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:33.166Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:33.060673244 +0000 UTC m=+8.309410453 (1 nodes), 902ns from start of poll, 300.722856ms from cluster start. Timeout at 2025-07-05 01:10:33.166022003 +0000 UTC m=+8.414759211, 105.348758ms after stability
2025-07-05T01:10:33.166Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: aborting pipeline replication: peer="{Voter server-2a02b826-95b6-ea47-4d8f-5888d63cae44 2a02b826-95b6-ea47-4d8f-5888d63cae44}"
2025-07-05T01:10:33.166Z [INFO]  server-e91b9c87-d881-a4af-b2c0-4deaa7b610fe: aborting pipeline replication: peer="{Voter server-07097d3f-2883-7843-8df2-7cab68d4b0c0 07097d3f-2883-7843-8df2-7cab68d4b0c0}"
--- PASS: TestRaft_BehindFollower (0.41s)
=== RUN   TestRaft_ApplyNonLeader
2025-07-05T01:10:33.167Z [INFO]  server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9 Address:38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9} {Suffrage:Voter ID:server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd Address:69d97214-425e-dd6b-afc9-e5c5e6b2e7fd} {Suffrage:Voter ID:server-bd719935-7bc1-5295-9f09-7e41650dbf8b Address:bd719935-7bc1-5295-9f09-7e41650dbf8b}]"
2025-07-05T01:10:33.167Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9 Address:38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9} {Suffrage:Voter ID:server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd Address:69d97214-425e-dd6b-afc9-e5c5e6b2e7fd} {Suffrage:Voter ID:server-bd719935-7bc1-5295-9f09-7e41650dbf8b Address:bd719935-7bc1-5295-9f09-7e41650dbf8b}]"
2025-07-05T01:10:33.167Z [INFO]  server-bd719935-7bc1-5295-9f09-7e41650dbf8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9 Address:38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9} {Suffrage:Voter ID:server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd Address:69d97214-425e-dd6b-afc9-e5c5e6b2e7fd} {Suffrage:Voter ID:server-bd719935-7bc1-5295-9f09-7e41650dbf8b Address:bd719935-7bc1-5295-9f09-7e41650dbf8b}]"
2025-07-05T01:10:33.167Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:33.167Z [INFO]  server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9: entering follower state: follower="Node at 38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9 [Follower]" leader-address= leader-id=
2025-07-05T01:10:33.167Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: entering follower state: follower="Node at 69d97214-425e-dd6b-afc9-e5c5e6b2e7fd [Follower]" leader-address= leader-id=
2025-07-05T01:10:33.167Z [INFO]  server-bd719935-7bc1-5295-9f09-7e41650dbf8b: entering follower state: follower="Node at bd719935-7bc1-5295-9f09-7e41650dbf8b [Follower]" leader-address= leader-id=
2025-07-05T01:10:33.243Z [WARN]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:33.243Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: entering candidate state: node="Node at 69d97214-425e-dd6b-afc9-e5c5e6b2e7fd [Candidate]" term=2
2025-07-05T01:10:33.243Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: election won: term=2 tally=2
2025-07-05T01:10:33.243Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: entering leader state: leader="Node at 69d97214-425e-dd6b-afc9-e5c5e6b2e7fd [Leader]"
2025-07-05T01:10:33.243Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: added peer, starting replication: peer=server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9
2025-07-05T01:10:33.243Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: added peer, starting replication: peer=server-bd719935-7bc1-5295-9f09-7e41650dbf8b
2025-07-05T01:10:33.243Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: pipelining replication: peer="{Voter server-bd719935-7bc1-5295-9f09-7e41650dbf8b bd719935-7bc1-5295-9f09-7e41650dbf8b}"
2025-07-05T01:10:33.243Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: pipelining replication: peer="{Voter server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9 38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9}"
2025-07-05T01:10:33.349Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:33.243731118 +0000 UTC m=+8.492468326 (1 nodes), 76.281217ms from start of poll, 76.524243ms from cluster start. Timeout at 2025-07-05 01:10:33.349671967 +0000 UTC m=+8.598409175, 105.940849ms after stability
2025-07-05T01:10:33.349Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:33.455Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:33.349729906 +0000 UTC m=+8.598467114 (2 nodes), 902ns from start of poll, 182.523031ms from cluster start. Timeout at 2025-07-05 01:10:33.455479556 +0000 UTC m=+8.704216764, 105.74965ms after stability
2025-07-05T01:10:33.455Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: aborting pipeline replication: peer="{Voter server-bd719935-7bc1-5295-9f09-7e41650dbf8b bd719935-7bc1-5295-9f09-7e41650dbf8b}"
2025-07-05T01:10:33.455Z [INFO]  server-69d97214-425e-dd6b-afc9-e5c5e6b2e7fd: aborting pipeline replication: peer="{Voter server-38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9 38bc1ef8-1d7a-bca6-b403-b6fcfa3ae8f9}"
--- PASS: TestRaft_ApplyNonLeader (0.29s)
=== RUN   TestRaft_ApplyConcurrent
2025-07-05T01:10:33.457Z [INFO]  server-6155a250-47ad-813b-9638-1cf3895926aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6155a250-47ad-813b-9638-1cf3895926aa Address:6155a250-47ad-813b-9638-1cf3895926aa} {Suffrage:Voter ID:server-48b387c4-6c01-9b6d-aed7-0c59018b41a7 Address:48b387c4-6c01-9b6d-aed7-0c59018b41a7} {Suffrage:Voter ID:server-809efbff-5d69-17c8-30fd-c4cbfc8546d0 Address:809efbff-5d69-17c8-30fd-c4cbfc8546d0}]"
2025-07-05T01:10:33.457Z [INFO]  server-6155a250-47ad-813b-9638-1cf3895926aa: entering follower state: follower="Node at 6155a250-47ad-813b-9638-1cf3895926aa [Follower]" leader-address= leader-id=
2025-07-05T01:10:33.462Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6155a250-47ad-813b-9638-1cf3895926aa Address:6155a250-47ad-813b-9638-1cf3895926aa} {Suffrage:Voter ID:server-48b387c4-6c01-9b6d-aed7-0c59018b41a7 Address:48b387c4-6c01-9b6d-aed7-0c59018b41a7} {Suffrage:Voter ID:server-809efbff-5d69-17c8-30fd-c4cbfc8546d0 Address:809efbff-5d69-17c8-30fd-c4cbfc8546d0}]"
2025-07-05T01:10:33.463Z [INFO]  server-809efbff-5d69-17c8-30fd-c4cbfc8546d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6155a250-47ad-813b-9638-1cf3895926aa Address:6155a250-47ad-813b-9638-1cf3895926aa} {Suffrage:Voter ID:server-48b387c4-6c01-9b6d-aed7-0c59018b41a7 Address:48b387c4-6c01-9b6d-aed7-0c59018b41a7} {Suffrage:Voter ID:server-809efbff-5d69-17c8-30fd-c4cbfc8546d0 Address:809efbff-5d69-17c8-30fd-c4cbfc8546d0}]"
2025-07-05T01:10:33.463Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:33.463Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: entering follower state: follower="Node at 48b387c4-6c01-9b6d-aed7-0c59018b41a7 [Follower]" leader-address= leader-id=
2025-07-05T01:10:33.463Z [INFO]  server-809efbff-5d69-17c8-30fd-c4cbfc8546d0: entering follower state: follower="Node at 809efbff-5d69-17c8-30fd-c4cbfc8546d0 [Follower]" leader-address= leader-id=
2025-07-05T01:10:33.599Z [WARN]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:33.599Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: entering candidate state: node="Node at 48b387c4-6c01-9b6d-aed7-0c59018b41a7 [Candidate]" term=2
2025-07-05T01:10:33.599Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: election won: term=2 tally=2
2025-07-05T01:10:33.599Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: entering leader state: leader="Node at 48b387c4-6c01-9b6d-aed7-0c59018b41a7 [Leader]"
2025-07-05T01:10:33.599Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: added peer, starting replication: peer=server-6155a250-47ad-813b-9638-1cf3895926aa
2025-07-05T01:10:33.599Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: added peer, starting replication: peer=server-809efbff-5d69-17c8-30fd-c4cbfc8546d0
2025-07-05T01:10:33.599Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: pipelining replication: peer="{Voter server-809efbff-5d69-17c8-30fd-c4cbfc8546d0 809efbff-5d69-17c8-30fd-c4cbfc8546d0}"
2025-07-05T01:10:33.599Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: pipelining replication: peer="{Voter server-6155a250-47ad-813b-9638-1cf3895926aa 6155a250-47ad-813b-9638-1cf3895926aa}"
2025-07-05T01:10:33.804Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:33.599343802 +0000 UTC m=+8.848081010 (1 nodes), 136.27697ms from start of poll, 142.251472ms from cluster start. Timeout at 2025-07-05 01:10:33.804915909 +0000 UTC m=+9.053653117, 205.572107ms after stability
2025-07-05T01:10:33.817Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: aborting pipeline replication: peer="{Voter server-809efbff-5d69-17c8-30fd-c4cbfc8546d0 809efbff-5d69-17c8-30fd-c4cbfc8546d0}"
2025-07-05T01:10:33.817Z [INFO]  server-48b387c4-6c01-9b6d-aed7-0c59018b41a7: aborting pipeline replication: peer="{Voter server-6155a250-47ad-813b-9638-1cf3895926aa 6155a250-47ad-813b-9638-1cf3895926aa}"
--- PASS: TestRaft_ApplyConcurrent (0.36s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-07-05T01:10:33.818Z [INFO]  server-cdc1f75c-e3f3-d615-e2d8-ba31f8392d30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdc1f75c-e3f3-d615-e2d8-ba31f8392d30 Address:cdc1f75c-e3f3-d615-e2d8-ba31f8392d30}]"
2025-07-05T01:10:33.818Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:33.818Z [INFO]  server-cdc1f75c-e3f3-d615-e2d8-ba31f8392d30: entering follower state: follower="Node at cdc1f75c-e3f3-d615-e2d8-ba31f8392d30 [Follower]" leader-address= leader-id=
2025-07-05T01:10:33.941Z [WARN]  server-cdc1f75c-e3f3-d615-e2d8-ba31f8392d30: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:33.941Z [INFO]  server-cdc1f75c-e3f3-d615-e2d8-ba31f8392d30: entering candidate state: node="Node at cdc1f75c-e3f3-d615-e2d8-ba31f8392d30 [Candidate]" term=2
2025-07-05T01:10:33.941Z [INFO]  server-cdc1f75c-e3f3-d615-e2d8-ba31f8392d30: election won: term=2 tally=1
2025-07-05T01:10:33.941Z [INFO]  server-cdc1f75c-e3f3-d615-e2d8-ba31f8392d30: entering leader state: leader="Node at cdc1f75c-e3f3-d615-e2d8-ba31f8392d30 [Leader]"
2025-07-05T01:10:34.143Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:33.941686234 +0000 UTC m=+9.190423442 (1 nodes), 123.062683ms from start of poll, 123.171146ms from cluster start. Timeout at 2025-07-05 01:10:34.143247371 +0000 UTC m=+9.391984579, 201.561137ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.33s)
=== RUN   TestRaft_JoinNode
2025-07-05T01:10:34.145Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ff3c9c0-6c87-873a-964c-b47adbec5409 Address:3ff3c9c0-6c87-873a-964c-b47adbec5409} {Suffrage:Voter ID:server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4 Address:f84e00a2-0a2a-a57b-e65e-cb1951dd5de4}]"
2025-07-05T01:10:34.145Z [INFO]  server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ff3c9c0-6c87-873a-964c-b47adbec5409 Address:3ff3c9c0-6c87-873a-964c-b47adbec5409} {Suffrage:Voter ID:server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4 Address:f84e00a2-0a2a-a57b-e65e-cb1951dd5de4}]"
2025-07-05T01:10:34.145Z [INFO]  server-52408a4f-fa03-ab34-8877-51ef0c812ef1: initial configuration: index=0 servers=[]
2025-07-05T01:10:34.145Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:34.145Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: entering follower state: follower="Node at 3ff3c9c0-6c87-873a-964c-b47adbec5409 [Follower]" leader-address= leader-id=
2025-07-05T01:10:34.145Z [INFO]  server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4: entering follower state: follower="Node at f84e00a2-0a2a-a57b-e65e-cb1951dd5de4 [Follower]" leader-address= leader-id=
2025-07-05T01:10:34.145Z [INFO]  server-52408a4f-fa03-ab34-8877-51ef0c812ef1: entering follower state: follower="Node at 52408a4f-fa03-ab34-8877-51ef0c812ef1 [Follower]" leader-address= leader-id=
2025-07-05T01:10:34.203Z [WARN]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:34.203Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: entering candidate state: node="Node at 3ff3c9c0-6c87-873a-964c-b47adbec5409 [Candidate]" term=2
2025-07-05T01:10:34.203Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: election won: term=2 tally=2
2025-07-05T01:10:34.203Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: entering leader state: leader="Node at 3ff3c9c0-6c87-873a-964c-b47adbec5409 [Leader]"
2025-07-05T01:10:34.204Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: added peer, starting replication: peer=server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4
2025-07-05T01:10:34.204Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: pipelining replication: peer="{Voter server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4 f84e00a2-0a2a-a57b-e65e-cb1951dd5de4}"
2025-07-05T01:10:34.221Z [WARN]  server-52408a4f-fa03-ab34-8877-51ef0c812ef1: no known peers, aborting election
2025-07-05T01:10:34.310Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:34.204115498 +0000 UTC m=+9.452852706 (1 nodes), 58.690554ms from start of poll, 59.085465ms from cluster start. Timeout at 2025-07-05 01:10:34.310125958 +0000 UTC m=+9.558863166, 106.01046ms after stability
2025-07-05T01:10:34.310Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: updating configuration: command=AddVoter server-id=server-52408a4f-fa03-ab34-8877-51ef0c812ef1 server-addr=52408a4f-fa03-ab34-8877-51ef0c812ef1 servers="[{Suffrage:Voter ID:server-3ff3c9c0-6c87-873a-964c-b47adbec5409 Address:3ff3c9c0-6c87-873a-964c-b47adbec5409} {Suffrage:Voter ID:server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4 Address:f84e00a2-0a2a-a57b-e65e-cb1951dd5de4} {Suffrage:Voter ID:server-52408a4f-fa03-ab34-8877-51ef0c812ef1 Address:52408a4f-fa03-ab34-8877-51ef0c812ef1}]"
2025-07-05T01:10:34.310Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: added peer, starting replication: peer=server-52408a4f-fa03-ab34-8877-51ef0c812ef1
2025-07-05T01:10:34.310Z [WARN]  server-52408a4f-fa03-ab34-8877-51ef0c812ef1: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-07-05T01:10:34.310Z [WARN]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: appendEntries rejected, sending older logs: peer="{Voter server-52408a4f-fa03-ab34-8877-51ef0c812ef1 52408a4f-fa03-ab34-8877-51ef0c812ef1}" next=1
2025-07-05T01:10:34.310Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: pipelining replication: peer="{Voter server-52408a4f-fa03-ab34-8877-51ef0c812ef1 52408a4f-fa03-ab34-8877-51ef0c812ef1}"
2025-07-05T01:10:34.310Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:34.415Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:34.31042599 +0000 UTC m=+9.559163198 (1 nodes), 410ns from start of poll, 165.395957ms from cluster start. Timeout at 2025-07-05 01:10:34.415821617 +0000 UTC m=+9.664558825, 105.395627ms after stability
2025-07-05T01:10:34.416Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: aborting pipeline replication: peer="{Voter server-f84e00a2-0a2a-a57b-e65e-cb1951dd5de4 f84e00a2-0a2a-a57b-e65e-cb1951dd5de4}"
2025-07-05T01:10:34.416Z [INFO]  server-3ff3c9c0-6c87-873a-964c-b47adbec5409: aborting pipeline replication: peer="{Voter server-52408a4f-fa03-ab34-8877-51ef0c812ef1 52408a4f-fa03-ab34-8877-51ef0c812ef1}"
2025-07-05T01:10:34.917Z [ERROR] server-3ff3c9c0-6c87-873a-964c-b47adbec5409: failed to heartbeat to: peer=f84e00a2-0a2a-a57b-e65e-cb1951dd5de4 backoff time=10ms error="command timed out"
--- PASS: TestRaft_JoinNode (0.77s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-07-05T01:10:34.918Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19a3377a-d67b-0b4d-12ae-85a389e49ccd Address:19a3377a-d67b-0b4d-12ae-85a389e49ccd}]"
2025-07-05T01:10:34.918Z [INFO]  server-6a8402a3-22b6-40b2-e47a-a4591836c640: initial configuration: index=0 servers=[]
2025-07-05T01:10:34.918Z [INFO]  server-2cbfa1af-9492-0662-c110-4fffcbbc84e0: initial configuration: index=0 servers=[]
2025-07-05T01:10:34.918Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:34.919Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: entering follower state: follower="Node at 19a3377a-d67b-0b4d-12ae-85a389e49ccd [Follower]" leader-address= leader-id=
2025-07-05T01:10:34.919Z [INFO]  server-6a8402a3-22b6-40b2-e47a-a4591836c640: entering follower state: follower="Node at 6a8402a3-22b6-40b2-e47a-a4591836c640 [Follower]" leader-address= leader-id=
2025-07-05T01:10:34.919Z [INFO]  server-2cbfa1af-9492-0662-c110-4fffcbbc84e0: entering follower state: follower="Node at 2cbfa1af-9492-0662-c110-4fffcbbc84e0 [Follower]" leader-address= leader-id=
2025-07-05T01:10:34.972Z [WARN]  server-2cbfa1af-9492-0662-c110-4fffcbbc84e0: no known peers, aborting election
2025-07-05T01:10:35.007Z [WARN]  server-6a8402a3-22b6-40b2-e47a-a4591836c640: no known peers, aborting election
2025-07-05T01:10:35.008Z [WARN]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:35.008Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: entering candidate state: node="Node at 19a3377a-d67b-0b4d-12ae-85a389e49ccd [Candidate]" term=2
2025-07-05T01:10:35.008Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: election won: term=2 tally=1
2025-07-05T01:10:35.008Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: entering leader state: leader="Node at 19a3377a-d67b-0b4d-12ae-85a389e49ccd [Leader]"
2025-07-05T01:10:35.114Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:35.008290566 +0000 UTC m=+10.257027774 (1 nodes), 89.292854ms from start of poll, 89.769057ms from cluster start. Timeout at 2025-07-05 01:10:35.114894348 +0000 UTC m=+10.363631566, 106.603792ms after stability
2025-07-05T01:10:35.114Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: updating configuration: command=AddVoter server-id=server-6a8402a3-22b6-40b2-e47a-a4591836c640 server-addr=6a8402a3-22b6-40b2-e47a-a4591836c640 servers="[{Suffrage:Voter ID:server-19a3377a-d67b-0b4d-12ae-85a389e49ccd Address:19a3377a-d67b-0b4d-12ae-85a389e49ccd} {Suffrage:Voter ID:server-6a8402a3-22b6-40b2-e47a-a4591836c640 Address:6a8402a3-22b6-40b2-e47a-a4591836c640}]"
2025-07-05T01:10:35.115Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: added peer, starting replication: peer=server-6a8402a3-22b6-40b2-e47a-a4591836c640
2025-07-05T01:10:35.115Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:35.115Z [WARN]  server-6a8402a3-22b6-40b2-e47a-a4591836c640: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-07-05T01:10:35.115Z [WARN]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: appendEntries rejected, sending older logs: peer="{Voter server-6a8402a3-22b6-40b2-e47a-a4591836c640 6a8402a3-22b6-40b2-e47a-a4591836c640}" next=1
2025-07-05T01:10:35.115Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: pipelining replication: peer="{Voter server-6a8402a3-22b6-40b2-e47a-a4591836c640 6a8402a3-22b6-40b2-e47a-a4591836c640}"
2025-07-05T01:10:35.221Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:35.11511924 +0000 UTC m=+10.363856458 (1 nodes), 832ns from start of poll, 196.597741ms from cluster start. Timeout at 2025-07-05 01:10:35.221193449 +0000 UTC m=+10.469930667, 106.074209ms after stability
2025-07-05T01:10:35.221Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: updating configuration: command=AddVoter server-id=server-2cbfa1af-9492-0662-c110-4fffcbbc84e0 server-addr=2cbfa1af-9492-0662-c110-4fffcbbc84e0 servers="[{Suffrage:Voter ID:server-19a3377a-d67b-0b4d-12ae-85a389e49ccd Address:19a3377a-d67b-0b4d-12ae-85a389e49ccd} {Suffrage:Voter ID:server-6a8402a3-22b6-40b2-e47a-a4591836c640 Address:6a8402a3-22b6-40b2-e47a-a4591836c640} {Suffrage:Voter ID:server-2cbfa1af-9492-0662-c110-4fffcbbc84e0 Address:2cbfa1af-9492-0662-c110-4fffcbbc84e0}]"
2025-07-05T01:10:35.221Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: added peer, starting replication: peer=server-2cbfa1af-9492-0662-c110-4fffcbbc84e0
2025-07-05T01:10:35.221Z [WARN]  server-2cbfa1af-9492-0662-c110-4fffcbbc84e0: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-07-05T01:10:35.221Z [WARN]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: appendEntries rejected, sending older logs: peer="{Voter server-2cbfa1af-9492-0662-c110-4fffcbbc84e0 2cbfa1af-9492-0662-c110-4fffcbbc84e0}" next=1
2025-07-05T01:10:35.221Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: pipelining replication: peer="{Voter server-2cbfa1af-9492-0662-c110-4fffcbbc84e0 2cbfa1af-9492-0662-c110-4fffcbbc84e0}"
2025-07-05T01:10:35.221Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:35.326Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:35.221540961 +0000 UTC m=+10.470278179 (1 nodes), 581ns from start of poll, 303.019462ms from cluster start. Timeout at 2025-07-05 01:10:35.326918003 +0000 UTC m=+10.575655221, 105.377042ms after stability
2025-07-05T01:10:35.327Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: aborting pipeline replication: peer="{Voter server-2cbfa1af-9492-0662-c110-4fffcbbc84e0 2cbfa1af-9492-0662-c110-4fffcbbc84e0}"
2025-07-05T01:10:35.327Z [INFO]  server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: aborting pipeline replication: peer="{Voter server-6a8402a3-22b6-40b2-e47a-a4591836c640 6a8402a3-22b6-40b2-e47a-a4591836c640}"
2025-07-05T01:10:35.827Z [ERROR] server-19a3377a-d67b-0b4d-12ae-85a389e49ccd: failed to heartbeat to: peer=2cbfa1af-9492-0662-c110-4fffcbbc84e0 backoff time=10ms error="command timed out"
--- PASS: TestRaft_JoinNode_ConfigStore (0.91s)
=== RUN   TestRaft_RemoveFollower
2025-07-05T01:10:35.828Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-443ed29a-75b6-1f16-a183-a6a92b488dfe Address:443ed29a-75b6-1f16-a183-a6a92b488dfe} {Suffrage:Voter ID:server-5cb66495-5bda-07eb-bfec-9e0665534625 Address:5cb66495-5bda-07eb-bfec-9e0665534625} {Suffrage:Voter ID:server-c83788b2-8af2-1093-2de0-02ba4f7a9016 Address:c83788b2-8af2-1093-2de0-02ba4f7a9016}]"
2025-07-05T01:10:35.829Z [INFO]  server-5cb66495-5bda-07eb-bfec-9e0665534625: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-443ed29a-75b6-1f16-a183-a6a92b488dfe Address:443ed29a-75b6-1f16-a183-a6a92b488dfe} {Suffrage:Voter ID:server-5cb66495-5bda-07eb-bfec-9e0665534625 Address:5cb66495-5bda-07eb-bfec-9e0665534625} {Suffrage:Voter ID:server-c83788b2-8af2-1093-2de0-02ba4f7a9016 Address:c83788b2-8af2-1093-2de0-02ba4f7a9016}]"
2025-07-05T01:10:35.829Z [INFO]  server-c83788b2-8af2-1093-2de0-02ba4f7a9016: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-443ed29a-75b6-1f16-a183-a6a92b488dfe Address:443ed29a-75b6-1f16-a183-a6a92b488dfe} {Suffrage:Voter ID:server-5cb66495-5bda-07eb-bfec-9e0665534625 Address:5cb66495-5bda-07eb-bfec-9e0665534625} {Suffrage:Voter ID:server-c83788b2-8af2-1093-2de0-02ba4f7a9016 Address:c83788b2-8af2-1093-2de0-02ba4f7a9016}]"
2025-07-05T01:10:35.829Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:35.829Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: entering follower state: follower="Node at 443ed29a-75b6-1f16-a183-a6a92b488dfe [Follower]" leader-address= leader-id=
2025-07-05T01:10:35.829Z [INFO]  server-5cb66495-5bda-07eb-bfec-9e0665534625: entering follower state: follower="Node at 5cb66495-5bda-07eb-bfec-9e0665534625 [Follower]" leader-address= leader-id=
2025-07-05T01:10:35.829Z [INFO]  server-c83788b2-8af2-1093-2de0-02ba4f7a9016: entering follower state: follower="Node at c83788b2-8af2-1093-2de0-02ba4f7a9016 [Follower]" leader-address= leader-id=
2025-07-05T01:10:35.887Z [WARN]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:35.887Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: entering candidate state: node="Node at 443ed29a-75b6-1f16-a183-a6a92b488dfe [Candidate]" term=2
2025-07-05T01:10:35.887Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: election won: term=2 tally=2
2025-07-05T01:10:35.887Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: entering leader state: leader="Node at 443ed29a-75b6-1f16-a183-a6a92b488dfe [Leader]"
2025-07-05T01:10:35.887Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: added peer, starting replication: peer=server-5cb66495-5bda-07eb-bfec-9e0665534625
2025-07-05T01:10:35.887Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: added peer, starting replication: peer=server-c83788b2-8af2-1093-2de0-02ba4f7a9016
2025-07-05T01:10:35.887Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: pipelining replication: peer="{Voter server-c83788b2-8af2-1093-2de0-02ba4f7a9016 c83788b2-8af2-1093-2de0-02ba4f7a9016}"
2025-07-05T01:10:35.887Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: pipelining replication: peer="{Voter server-5cb66495-5bda-07eb-bfec-9e0665534625 5cb66495-5bda-07eb-bfec-9e0665534625}"
2025-07-05T01:10:35.993Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:35.88759423 +0000 UTC m=+11.136331448 (1 nodes), 58.48552ms from start of poll, 58.733775ms from cluster start. Timeout at 2025-07-05 01:10:35.993509231 +0000 UTC m=+11.242246439, 105.914991ms after stability
2025-07-05T01:10:35.998Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:36.104Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:35.998762661 +0000 UTC m=+11.247499879 (2 nodes), 741ns from start of poll, 169.902206ms from cluster start. Timeout at 2025-07-05 01:10:36.104566924 +0000 UTC m=+11.353304132, 105.804253ms after stability
2025-07-05T01:10:36.104Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: updating configuration: command=RemoveServer server-id=server-5cb66495-5bda-07eb-bfec-9e0665534625 server-addr= servers="[{Suffrage:Voter ID:server-443ed29a-75b6-1f16-a183-a6a92b488dfe Address:443ed29a-75b6-1f16-a183-a6a92b488dfe} {Suffrage:Voter ID:server-c83788b2-8af2-1093-2de0-02ba4f7a9016 Address:c83788b2-8af2-1093-2de0-02ba4f7a9016}]"
2025-07-05T01:10:36.104Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: removed peer, stopping replication: peer=server-5cb66495-5bda-07eb-bfec-9e0665534625 last-index=3
2025-07-05T01:10:36.104Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: aborting pipeline replication: peer="{Voter server-5cb66495-5bda-07eb-bfec-9e0665534625 5cb66495-5bda-07eb-bfec-9e0665534625}"
2025-07-05T01:10:36.165Z [WARN]  server-5cb66495-5bda-07eb-bfec-9e0665534625: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-07-05T01:10:36.210Z [INFO]  server-443ed29a-75b6-1f16-a183-a6a92b488dfe: aborting pipeline replication: peer="{Voter server-c83788b2-8af2-1093-2de0-02ba4f7a9016 c83788b2-8af2-1093-2de0-02ba4f7a9016}"
--- PASS: TestRaft_RemoveFollower (0.38s)
=== RUN   TestRaft_RemoveLeader
2025-07-05T01:10:36.211Z [INFO]  server-cf96eb3f-9678-5f33-78d4-568f708baf97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf96eb3f-9678-5f33-78d4-568f708baf97 Address:cf96eb3f-9678-5f33-78d4-568f708baf97} {Suffrage:Voter ID:server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9 Address:dfb6b3d6-655c-d11b-c23f-b6260ab142e9} {Suffrage:Voter ID:server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418 Address:ec866ebc-2b3d-a9de-8aaa-dd72abf32418}]"
2025-07-05T01:10:36.211Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf96eb3f-9678-5f33-78d4-568f708baf97 Address:cf96eb3f-9678-5f33-78d4-568f708baf97} {Suffrage:Voter ID:server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9 Address:dfb6b3d6-655c-d11b-c23f-b6260ab142e9} {Suffrage:Voter ID:server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418 Address:ec866ebc-2b3d-a9de-8aaa-dd72abf32418}]"
2025-07-05T01:10:36.211Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf96eb3f-9678-5f33-78d4-568f708baf97 Address:cf96eb3f-9678-5f33-78d4-568f708baf97} {Suffrage:Voter ID:server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9 Address:dfb6b3d6-655c-d11b-c23f-b6260ab142e9} {Suffrage:Voter ID:server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418 Address:ec866ebc-2b3d-a9de-8aaa-dd72abf32418}]"
2025-07-05T01:10:36.211Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:36.211Z [INFO]  server-cf96eb3f-9678-5f33-78d4-568f708baf97: entering follower state: follower="Node at cf96eb3f-9678-5f33-78d4-568f708baf97 [Follower]" leader-address= leader-id=
2025-07-05T01:10:36.211Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: entering follower state: follower="Node at dfb6b3d6-655c-d11b-c23f-b6260ab142e9 [Follower]" leader-address= leader-id=
2025-07-05T01:10:36.211Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: entering follower state: follower="Node at ec866ebc-2b3d-a9de-8aaa-dd72abf32418 [Follower]" leader-address= leader-id=
2025-07-05T01:10:36.268Z [WARN]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:36.268Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: entering candidate state: node="Node at ec866ebc-2b3d-a9de-8aaa-dd72abf32418 [Candidate]" term=2
2025-07-05T01:10:36.268Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: election won: term=2 tally=2
2025-07-05T01:10:36.268Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: entering leader state: leader="Node at ec866ebc-2b3d-a9de-8aaa-dd72abf32418 [Leader]"
2025-07-05T01:10:36.268Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: added peer, starting replication: peer=server-cf96eb3f-9678-5f33-78d4-568f708baf97
2025-07-05T01:10:36.268Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: added peer, starting replication: peer=server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9
2025-07-05T01:10:36.268Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: pipelining replication: peer="{Voter server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9 dfb6b3d6-655c-d11b-c23f-b6260ab142e9}"
2025-07-05T01:10:36.268Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: pipelining replication: peer="{Voter server-cf96eb3f-9678-5f33-78d4-568f708baf97 cf96eb3f-9678-5f33-78d4-568f708baf97}"
2025-07-05T01:10:36.374Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:36.268691928 +0000 UTC m=+11.517429136 (1 nodes), 57.241568ms from start of poll, 57.457593ms from cluster start. Timeout at 2025-07-05 01:10:36.374361038 +0000 UTC m=+11.623098246, 105.66911ms after stability
2025-07-05T01:10:36.379Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:36.485Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:36.379870518 +0000 UTC m=+11.628607726 (2 nodes), 871ns from start of poll, 168.636183ms from cluster start. Timeout at 2025-07-05 01:10:36.485573621 +0000 UTC m=+11.734310830, 105.703104ms after stability
2025-07-05T01:10:36.485Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: updating configuration: command=RemoveServer server-id=server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418 server-addr= servers="[{Suffrage:Voter ID:server-cf96eb3f-9678-5f33-78d4-568f708baf97 Address:cf96eb3f-9678-5f33-78d4-568f708baf97} {Suffrage:Voter ID:server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9 Address:dfb6b3d6-655c-d11b-c23f-b6260ab142e9}]"
2025-07-05T01:10:36.485Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: removed ourself, shutting down
2025-07-05T01:10:36.485Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: aborting pipeline replication: peer="{Voter server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9 dfb6b3d6-655c-d11b-c23f-b6260ab142e9}"
2025-07-05T01:10:36.485Z [INFO]  server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418: aborting pipeline replication: peer="{Voter server-cf96eb3f-9678-5f33-78d4-568f708baf97 cf96eb3f-9678-5f33-78d4-568f708baf97}"
2025-07-05T01:10:36.558Z [WARN]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: heartbeat timeout reached, starting election: last-leader-addr=ec866ebc-2b3d-a9de-8aaa-dd72abf32418 last-leader-id=server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418
2025-07-05T01:10:36.558Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: entering candidate state: node="Node at dfb6b3d6-655c-d11b-c23f-b6260ab142e9 [Candidate]" term=3
2025-07-05T01:10:36.558Z [WARN]  server-cf96eb3f-9678-5f33-78d4-568f708baf97: rejecting vote request since we have a leader: from=dfb6b3d6-655c-d11b-c23f-b6260ab142e9 leader=ec866ebc-2b3d-a9de-8aaa-dd72abf32418 leader-id=server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418
2025-07-05T01:10:36.579Z [WARN]  server-cf96eb3f-9678-5f33-78d4-568f708baf97: heartbeat timeout reached, starting election: last-leader-addr=ec866ebc-2b3d-a9de-8aaa-dd72abf32418 last-leader-id=server-ec866ebc-2b3d-a9de-8aaa-dd72abf32418
2025-07-05T01:10:36.579Z [INFO]  server-cf96eb3f-9678-5f33-78d4-568f708baf97: entering candidate state: node="Node at cf96eb3f-9678-5f33-78d4-568f708baf97 [Candidate]" term=3
2025-07-05T01:10:36.579Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: duplicate requestVote for same term: term=3
2025-07-05T01:10:36.624Z [WARN]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: Election timeout reached, restarting election
2025-07-05T01:10:36.624Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: entering candidate state: node="Node at dfb6b3d6-655c-d11b-c23f-b6260ab142e9 [Candidate]" term=4
2025-07-05T01:10:36.624Z [INFO]  server-cf96eb3f-9678-5f33-78d4-568f708baf97: entering follower state: follower="Node at cf96eb3f-9678-5f33-78d4-568f708baf97 [Follower]" leader-address= leader-id=
2025-07-05T01:10:36.624Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: election won: term=4 tally=2
2025-07-05T01:10:36.624Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: entering leader state: leader="Node at dfb6b3d6-655c-d11b-c23f-b6260ab142e9 [Leader]"
2025-07-05T01:10:36.624Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: added peer, starting replication: peer=server-cf96eb3f-9678-5f33-78d4-568f708baf97
2025-07-05T01:10:36.625Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: pipelining replication: peer="{Voter server-cf96eb3f-9678-5f33-78d4-568f708baf97 cf96eb3f-9678-5f33-78d4-568f708baf97}"
2025-07-05T01:10:36.697Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:36.803Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:36.69769026 +0000 UTC m=+11.946427468 (1 nodes), 36.017µs from start of poll, 486.455925ms from cluster start. Timeout at 2025-07-05 01:10:36.803250145 +0000 UTC m=+12.051987363, 105.559895ms after stability
2025-07-05T01:10:36.803Z [INFO]  server-dfb6b3d6-655c-d11b-c23f-b6260ab142e9: aborting pipeline replication: peer="{Voter server-cf96eb3f-9678-5f33-78d4-568f708baf97 cf96eb3f-9678-5f33-78d4-568f708baf97}"
--- PASS: TestRaft_RemoveLeader (0.59s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-07-05T01:10:36.804Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c0c755f-368f-0329-8245-a891f1605f87 Address:1c0c755f-368f-0329-8245-a891f1605f87} {Suffrage:Voter ID:server-3e2f84f4-6945-380a-e753-024f0f8e70d8 Address:3e2f84f4-6945-380a-e753-024f0f8e70d8} {Suffrage:Voter ID:server-fb54ac7a-a073-2858-b728-0305f3f91223 Address:fb54ac7a-a073-2858-b728-0305f3f91223}]"
2025-07-05T01:10:36.804Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c0c755f-368f-0329-8245-a891f1605f87 Address:1c0c755f-368f-0329-8245-a891f1605f87} {Suffrage:Voter ID:server-3e2f84f4-6945-380a-e753-024f0f8e70d8 Address:3e2f84f4-6945-380a-e753-024f0f8e70d8} {Suffrage:Voter ID:server-fb54ac7a-a073-2858-b728-0305f3f91223 Address:fb54ac7a-a073-2858-b728-0305f3f91223}]"
2025-07-05T01:10:36.804Z [INFO]  server-fb54ac7a-a073-2858-b728-0305f3f91223: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c0c755f-368f-0329-8245-a891f1605f87 Address:1c0c755f-368f-0329-8245-a891f1605f87} {Suffrage:Voter ID:server-3e2f84f4-6945-380a-e753-024f0f8e70d8 Address:3e2f84f4-6945-380a-e753-024f0f8e70d8} {Suffrage:Voter ID:server-fb54ac7a-a073-2858-b728-0305f3f91223 Address:fb54ac7a-a073-2858-b728-0305f3f91223}]"
2025-07-05T01:10:36.804Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:36.804Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: entering follower state: follower="Node at 1c0c755f-368f-0329-8245-a891f1605f87 [Follower]" leader-address= leader-id=
2025-07-05T01:10:36.804Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: entering follower state: follower="Node at 3e2f84f4-6945-380a-e753-024f0f8e70d8 [Follower]" leader-address= leader-id=
2025-07-05T01:10:36.804Z [INFO]  server-fb54ac7a-a073-2858-b728-0305f3f91223: entering follower state: follower="Node at fb54ac7a-a073-2858-b728-0305f3f91223 [Follower]" leader-address= leader-id=
2025-07-05T01:10:36.858Z [WARN]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:36.858Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: entering candidate state: node="Node at 3e2f84f4-6945-380a-e753-024f0f8e70d8 [Candidate]" term=2
2025-07-05T01:10:36.858Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: election won: term=2 tally=2
2025-07-05T01:10:36.858Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: entering leader state: leader="Node at 3e2f84f4-6945-380a-e753-024f0f8e70d8 [Leader]"
2025-07-05T01:10:36.858Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: added peer, starting replication: peer=server-1c0c755f-368f-0329-8245-a891f1605f87
2025-07-05T01:10:36.858Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: added peer, starting replication: peer=server-fb54ac7a-a073-2858-b728-0305f3f91223
2025-07-05T01:10:36.858Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: pipelining replication: peer="{Voter server-fb54ac7a-a073-2858-b728-0305f3f91223 fb54ac7a-a073-2858-b728-0305f3f91223}"
2025-07-05T01:10:36.858Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: pipelining replication: peer="{Voter server-1c0c755f-368f-0329-8245-a891f1605f87 1c0c755f-368f-0329-8245-a891f1605f87}"
2025-07-05T01:10:36.963Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:36.858529534 +0000 UTC m=+12.107266752 (2 nodes), 53.825834ms from start of poll, 54.064792ms from cluster start. Timeout at 2025-07-05 01:10:36.963537173 +0000 UTC m=+12.212274391, 105.007639ms after stability
2025-07-05T01:10:36.963Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:37.068Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:36.963588269 +0000 UTC m=+12.212325477 (1 nodes), 661ns from start of poll, 159.123517ms from cluster start. Timeout at 2025-07-05 01:10:37.068661902 +0000 UTC m=+12.317399120, 105.073643ms after stability
2025-07-05T01:10:37.070Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: updating configuration: command=RemoveServer server-id=server-3e2f84f4-6945-380a-e753-024f0f8e70d8 server-addr= servers="[{Suffrage:Voter ID:server-1c0c755f-368f-0329-8245-a891f1605f87 Address:1c0c755f-368f-0329-8245-a891f1605f87} {Suffrage:Voter ID:server-fb54ac7a-a073-2858-b728-0305f3f91223 Address:fb54ac7a-a073-2858-b728-0305f3f91223}]"
2025-07-05T01:10:37.070Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: removed ourself, transitioning to follower
2025-07-05T01:10:37.070Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: entering follower state: follower="Node at 3e2f84f4-6945-380a-e753-024f0f8e70d8 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.070Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: aborting pipeline replication: peer="{Voter server-fb54ac7a-a073-2858-b728-0305f3f91223 fb54ac7a-a073-2858-b728-0305f3f91223}"
2025-07-05T01:10:37.070Z [INFO]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: aborting pipeline replication: peer="{Voter server-1c0c755f-368f-0329-8245-a891f1605f87 1c0c755f-368f-0329-8245-a891f1605f87}"
2025-07-05T01:10:37.122Z [WARN]  server-1c0c755f-368f-0329-8245-a891f1605f87: heartbeat timeout reached, starting election: last-leader-addr=3e2f84f4-6945-380a-e753-024f0f8e70d8 last-leader-id=server-3e2f84f4-6945-380a-e753-024f0f8e70d8
2025-07-05T01:10:37.122Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: entering candidate state: node="Node at 1c0c755f-368f-0329-8245-a891f1605f87 [Candidate]" term=3
2025-07-05T01:10:37.122Z [WARN]  server-fb54ac7a-a073-2858-b728-0305f3f91223: rejecting vote request since we have a leader: from=1c0c755f-368f-0329-8245-a891f1605f87 leader=3e2f84f4-6945-380a-e753-024f0f8e70d8 leader-id=server-3e2f84f4-6945-380a-e753-024f0f8e70d8
2025-07-05T01:10:37.138Z [WARN]  server-3e2f84f4-6945-380a-e753-024f0f8e70d8: not part of stable configuration, aborting election
2025-07-05T01:10:37.165Z [WARN]  server-fb54ac7a-a073-2858-b728-0305f3f91223: heartbeat timeout reached, starting election: last-leader-addr=3e2f84f4-6945-380a-e753-024f0f8e70d8 last-leader-id=server-3e2f84f4-6945-380a-e753-024f0f8e70d8
2025-07-05T01:10:37.165Z [INFO]  server-fb54ac7a-a073-2858-b728-0305f3f91223: entering candidate state: node="Node at fb54ac7a-a073-2858-b728-0305f3f91223 [Candidate]" term=3
2025-07-05T01:10:37.165Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: duplicate requestVote for same term: term=3
2025-07-05T01:10:37.175Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:37.217Z [WARN]  server-1c0c755f-368f-0329-8245-a891f1605f87: Election timeout reached, restarting election
2025-07-05T01:10:37.217Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: entering candidate state: node="Node at 1c0c755f-368f-0329-8245-a891f1605f87 [Candidate]" term=4
2025-07-05T01:10:37.217Z [INFO]  server-fb54ac7a-a073-2858-b728-0305f3f91223: entering follower state: follower="Node at fb54ac7a-a073-2858-b728-0305f3f91223 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.217Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: election won: term=4 tally=2
2025-07-05T01:10:37.217Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: entering leader state: leader="Node at 1c0c755f-368f-0329-8245-a891f1605f87 [Leader]"
2025-07-05T01:10:37.217Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: added peer, starting replication: peer=server-fb54ac7a-a073-2858-b728-0305f3f91223
2025-07-05T01:10:37.218Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: pipelining replication: peer="{Voter server-fb54ac7a-a073-2858-b728-0305f3f91223 fb54ac7a-a073-2858-b728-0305f3f91223}"
2025-07-05T01:10:37.323Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:37.217958723 +0000 UTC m=+12.466695941 (1 nodes), 42.117109ms from start of poll, 413.493981ms from cluster start. Timeout at 2025-07-05 01:10:37.323486027 +0000 UTC m=+12.572223235, 105.527294ms after stability
2025-07-05T01:10:37.429Z [INFO]  server-1c0c755f-368f-0329-8245-a891f1605f87: aborting pipeline replication: peer="{Voter server-fb54ac7a-a073-2858-b728-0305f3f91223 fb54ac7a-a073-2858-b728-0305f3f91223}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-07-05T01:10:37.430Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8 Address:2b448f7d-f2d1-6fbf-317d-8db3806e91a8} {Suffrage:Voter ID:server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f Address:6af6876a-d09a-2f2b-fefb-3b3c4307d92f} {Suffrage:Voter ID:server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 Address:18c21124-d1fc-d2b3-c7f3-301f19d04a91} {Suffrage:Voter ID:server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 Address:cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9}]"
2025-07-05T01:10:37.430Z [INFO]  server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8 Address:2b448f7d-f2d1-6fbf-317d-8db3806e91a8} {Suffrage:Voter ID:server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f Address:6af6876a-d09a-2f2b-fefb-3b3c4307d92f} {Suffrage:Voter ID:server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 Address:18c21124-d1fc-d2b3-c7f3-301f19d04a91} {Suffrage:Voter ID:server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 Address:cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9}]"
2025-07-05T01:10:37.430Z [INFO]  server-18c21124-d1fc-d2b3-c7f3-301f19d04a91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8 Address:2b448f7d-f2d1-6fbf-317d-8db3806e91a8} {Suffrage:Voter ID:server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f Address:6af6876a-d09a-2f2b-fefb-3b3c4307d92f} {Suffrage:Voter ID:server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 Address:18c21124-d1fc-d2b3-c7f3-301f19d04a91} {Suffrage:Voter ID:server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 Address:cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9}]"
2025-07-05T01:10:37.430Z [INFO]  server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8 Address:2b448f7d-f2d1-6fbf-317d-8db3806e91a8} {Suffrage:Voter ID:server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f Address:6af6876a-d09a-2f2b-fefb-3b3c4307d92f} {Suffrage:Voter ID:server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 Address:18c21124-d1fc-d2b3-c7f3-301f19d04a91} {Suffrage:Voter ID:server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 Address:cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9}]"
2025-07-05T01:10:37.430Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:37.430Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: entering follower state: follower="Node at 2b448f7d-f2d1-6fbf-317d-8db3806e91a8 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.431Z [INFO]  server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f: entering follower state: follower="Node at 6af6876a-d09a-2f2b-fefb-3b3c4307d92f [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.431Z [INFO]  server-18c21124-d1fc-d2b3-c7f3-301f19d04a91: entering follower state: follower="Node at 18c21124-d1fc-d2b3-c7f3-301f19d04a91 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.431Z [INFO]  server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9: entering follower state: follower="Node at cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.505Z [WARN]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: entering candidate state: node="Node at 2b448f7d-f2d1-6fbf-317d-8db3806e91a8 [Candidate]" term=2
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: election won: term=2 tally=3
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: entering leader state: leader="Node at 2b448f7d-f2d1-6fbf-317d-8db3806e91a8 [Leader]"
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: added peer, starting replication: peer=server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: added peer, starting replication: peer=server-18c21124-d1fc-d2b3-c7f3-301f19d04a91
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: added peer, starting replication: peer=server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: pipelining replication: peer="{Voter server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9}"
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: pipelining replication: peer="{Voter server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f 6af6876a-d09a-2f2b-fefb-3b3c4307d92f}"
2025-07-05T01:10:37.505Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: pipelining replication: peer="{Voter server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 18c21124-d1fc-d2b3-c7f3-301f19d04a91}"
2025-07-05T01:10:37.611Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:37.50565825 +0000 UTC m=+12.754395468 (1 nodes), 74.686006ms from start of poll, 75.026986ms from cluster start. Timeout at 2025-07-05 01:10:37.611513629 +0000 UTC m=+12.860250837, 105.855369ms after stability
2025-07-05T01:10:37.716Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:37.821Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:37.716605055 +0000 UTC m=+12.965342273 (3 nodes), 1.413µs from start of poll, 285.973791ms from cluster start. Timeout at 2025-07-05 01:10:37.821640867 +0000 UTC m=+13.070378085, 105.035812ms after stability
2025-07-05T01:10:37.821Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: updating configuration: command=RemoveServer server-id=server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 server-addr= servers="[{Suffrage:Voter ID:server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8 Address:2b448f7d-f2d1-6fbf-317d-8db3806e91a8} {Suffrage:Voter ID:server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f Address:6af6876a-d09a-2f2b-fefb-3b3c4307d92f} {Suffrage:Voter ID:server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 Address:18c21124-d1fc-d2b3-c7f3-301f19d04a91}]"
2025-07-05T01:10:37.821Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: removed peer, stopping replication: peer=server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 last-index=3
2025-07-05T01:10:37.821Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: aborting pipeline replication: peer="{Voter server-cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9 cd2d3d7b-3dc4-2b24-e373-3c6662dac4b9}"
2025-07-05T01:10:37.821Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to pipeline appendEntries: peer="{Voter server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f 6af6876a-d09a-2f2b-fefb-3b3c4307d92f}" error="append pipeline closed"
2025-07-05T01:10:37.821Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: aborting pipeline replication: peer="{Voter server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f 6af6876a-d09a-2f2b-fefb-3b3c4307d92f}"
2025-07-05T01:10:37.821Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to pipeline appendEntries: peer="{Voter server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 18c21124-d1fc-d2b3-c7f3-301f19d04a91}" error="append pipeline closed"
2025-07-05T01:10:37.821Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: aborting pipeline replication: peer="{Voter server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 18c21124-d1fc-d2b3-c7f3-301f19d04a91}"
2025-07-05T01:10:37.822Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to heartbeat to: peer=18c21124-d1fc-d2b3-c7f3-301f19d04a91 backoff time=10ms error="failed to connect to peer: 18c21124-d1fc-d2b3-c7f3-301f19d04a91"
2025-07-05T01:10:37.824Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to heartbeat to: peer=6af6876a-d09a-2f2b-fefb-3b3c4307d92f backoff time=10ms error="failed to connect to peer: 6af6876a-d09a-2f2b-fefb-3b3c4307d92f"
2025-07-05T01:10:37.827Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to appendEntries to: peer="{Voter server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 18c21124-d1fc-d2b3-c7f3-301f19d04a91}" error="failed to connect to peer: 18c21124-d1fc-d2b3-c7f3-301f19d04a91"
2025-07-05T01:10:37.832Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to appendEntries to: peer="{Voter server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f 6af6876a-d09a-2f2b-fefb-3b3c4307d92f}" error="failed to connect to peer: 6af6876a-d09a-2f2b-fefb-3b3c4307d92f"
2025-07-05T01:10:37.842Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to heartbeat to: peer=18c21124-d1fc-d2b3-c7f3-301f19d04a91 backoff time=10ms error="failed to connect to peer: 18c21124-d1fc-d2b3-c7f3-301f19d04a91"
2025-07-05T01:10:37.845Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to heartbeat to: peer=6af6876a-d09a-2f2b-fefb-3b3c4307d92f backoff time=10ms error="failed to connect to peer: 6af6876a-d09a-2f2b-fefb-3b3c4307d92f"
2025-07-05T01:10:37.845Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to appendEntries to: peer="{Voter server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 18c21124-d1fc-d2b3-c7f3-301f19d04a91}" error="failed to connect to peer: 18c21124-d1fc-d2b3-c7f3-301f19d04a91"
2025-07-05T01:10:37.851Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to appendEntries to: peer="{Voter server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f 6af6876a-d09a-2f2b-fefb-3b3c4307d92f}" error="failed to connect to peer: 6af6876a-d09a-2f2b-fefb-3b3c4307d92f"
2025-07-05T01:10:37.862Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to heartbeat to: peer=18c21124-d1fc-d2b3-c7f3-301f19d04a91 backoff time=10ms error="failed to connect to peer: 18c21124-d1fc-d2b3-c7f3-301f19d04a91"
2025-07-05T01:10:37.863Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to appendEntries to: peer="{Voter server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 18c21124-d1fc-d2b3-c7f3-301f19d04a91}" error="failed to connect to peer: 18c21124-d1fc-d2b3-c7f3-301f19d04a91"
2025-07-05T01:10:37.863Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to heartbeat to: peer=6af6876a-d09a-2f2b-fefb-3b3c4307d92f backoff time=10ms error="failed to connect to peer: 6af6876a-d09a-2f2b-fefb-3b3c4307d92f"
2025-07-05T01:10:37.867Z [WARN]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to contact: server-id=server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f time=50.116378ms
2025-07-05T01:10:37.873Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to appendEntries to: peer="{Voter server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f 6af6876a-d09a-2f2b-fefb-3b3c4307d92f}" error="failed to connect to peer: 6af6876a-d09a-2f2b-fefb-3b3c4307d92f"
2025-07-05T01:10:37.877Z [WARN]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to contact: server-id=server-6af6876a-d09a-2f2b-fefb-3b3c4307d92f time=60.763762ms
2025-07-05T01:10:37.877Z [WARN]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to contact: server-id=server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 time=59.623845ms
2025-07-05T01:10:37.877Z [WARN]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to contact quorum of nodes, stepping down
2025-07-05T01:10:37.878Z [INFO]  server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: entering follower state: follower="Node at 2b448f7d-f2d1-6fbf-317d-8db3806e91a8 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.878Z [ERROR] server-2b448f7d-f2d1-6fbf-317d-8db3806e91a8: failed to appendEntries to: peer="{Voter server-18c21124-d1fc-d2b3-c7f3-301f19d04a91 18c21124-d1fc-d2b3-c7f3-301f19d04a91}" error="failed to connect to peer: 18c21124-d1fc-d2b3-c7f3-301f19d04a91"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s)
=== RUN   TestRaft_AddKnownPeer
2025-07-05T01:10:37.879Z [INFO]  server-9f61b967-b64f-073a-023e-ad23d5ee41f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f61b967-b64f-073a-023e-ad23d5ee41f2 Address:9f61b967-b64f-073a-023e-ad23d5ee41f2} {Suffrage:Voter ID:server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0 Address:60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0} {Suffrage:Voter ID:server-727a0b7f-1565-5a3a-8867-3d831fb2fe54 Address:727a0b7f-1565-5a3a-8867-3d831fb2fe54}]"
2025-07-05T01:10:37.879Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f61b967-b64f-073a-023e-ad23d5ee41f2 Address:9f61b967-b64f-073a-023e-ad23d5ee41f2} {Suffrage:Voter ID:server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0 Address:60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0} {Suffrage:Voter ID:server-727a0b7f-1565-5a3a-8867-3d831fb2fe54 Address:727a0b7f-1565-5a3a-8867-3d831fb2fe54}]"
2025-07-05T01:10:37.879Z [INFO]  server-727a0b7f-1565-5a3a-8867-3d831fb2fe54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f61b967-b64f-073a-023e-ad23d5ee41f2 Address:9f61b967-b64f-073a-023e-ad23d5ee41f2} {Suffrage:Voter ID:server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0 Address:60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0} {Suffrage:Voter ID:server-727a0b7f-1565-5a3a-8867-3d831fb2fe54 Address:727a0b7f-1565-5a3a-8867-3d831fb2fe54}]"
2025-07-05T01:10:37.879Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:37.879Z [INFO]  server-9f61b967-b64f-073a-023e-ad23d5ee41f2: entering follower state: follower="Node at 9f61b967-b64f-073a-023e-ad23d5ee41f2 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.879Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: entering follower state: follower="Node at 60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.879Z [INFO]  server-727a0b7f-1565-5a3a-8867-3d831fb2fe54: entering follower state: follower="Node at 727a0b7f-1565-5a3a-8867-3d831fb2fe54 [Follower]" leader-address= leader-id=
2025-07-05T01:10:37.942Z [WARN]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:37.942Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: entering candidate state: node="Node at 60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0 [Candidate]" term=2
2025-07-05T01:10:37.942Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: election won: term=2 tally=2
2025-07-05T01:10:37.942Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: entering leader state: leader="Node at 60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0 [Leader]"
2025-07-05T01:10:37.942Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: added peer, starting replication: peer=server-9f61b967-b64f-073a-023e-ad23d5ee41f2
2025-07-05T01:10:37.942Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: added peer, starting replication: peer=server-727a0b7f-1565-5a3a-8867-3d831fb2fe54
2025-07-05T01:10:37.942Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: pipelining replication: peer="{Voter server-727a0b7f-1565-5a3a-8867-3d831fb2fe54 727a0b7f-1565-5a3a-8867-3d831fb2fe54}"
2025-07-05T01:10:37.942Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: pipelining replication: peer="{Voter server-9f61b967-b64f-073a-023e-ad23d5ee41f2 9f61b967-b64f-073a-023e-ad23d5ee41f2}"
2025-07-05T01:10:38.047Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:37.942348764 +0000 UTC m=+13.191085982 (1 nodes), 62.790903ms from start of poll, 63.02966ms from cluster start. Timeout at 2025-07-05 01:10:38.047356013 +0000 UTC m=+13.296093231, 105.007249ms after stability
2025-07-05T01:10:38.047Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:38.152Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:38.047409163 +0000 UTC m=+13.296146371 (2 nodes), 1.734µs from start of poll, 168.090049ms from cluster start. Timeout at 2025-07-05 01:10:38.152947567 +0000 UTC m=+13.401684785, 105.538414ms after stability
2025-07-05T01:10:38.153Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: updating configuration: command=AddVoter server-id=server-9f61b967-b64f-073a-023e-ad23d5ee41f2 server-addr=9f61b967-b64f-073a-023e-ad23d5ee41f2 servers="[{Suffrage:Voter ID:server-9f61b967-b64f-073a-023e-ad23d5ee41f2 Address:9f61b967-b64f-073a-023e-ad23d5ee41f2} {Suffrage:Voter ID:server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0 Address:60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0} {Suffrage:Voter ID:server-727a0b7f-1565-5a3a-8867-3d831fb2fe54 Address:727a0b7f-1565-5a3a-8867-3d831fb2fe54}]"
2025-07-05T01:10:38.153Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: aborting pipeline replication: peer="{Voter server-727a0b7f-1565-5a3a-8867-3d831fb2fe54 727a0b7f-1565-5a3a-8867-3d831fb2fe54}"
2025-07-05T01:10:38.153Z [INFO]  server-60a69b81-98c3-3df1-fcc8-2aeb10ffa1d0: aborting pipeline replication: peer="{Voter server-9f61b967-b64f-073a-023e-ad23d5ee41f2 9f61b967-b64f-073a-023e-ad23d5ee41f2}"
--- PASS: TestRaft_AddKnownPeer (0.27s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-07-05T01:10:38.154Z [INFO]  server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0 Address:d19f0848-5c96-3a68-3db5-2ac4ab9e80b0} {Suffrage:Voter ID:server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993 Address:f2ff7252-4cde-c1c8-c5a5-3fab00bd5993} {Suffrage:Voter ID:server-5e433507-cbe6-e28e-dc54-4ad5060db305 Address:5e433507-cbe6-e28e-dc54-4ad5060db305}]"
2025-07-05T01:10:38.155Z [INFO]  server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0: entering follower state: follower="Node at d19f0848-5c96-3a68-3db5-2ac4ab9e80b0 [Follower]" leader-address= leader-id=
2025-07-05T01:10:38.155Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0 Address:d19f0848-5c96-3a68-3db5-2ac4ab9e80b0} {Suffrage:Voter ID:server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993 Address:f2ff7252-4cde-c1c8-c5a5-3fab00bd5993} {Suffrage:Voter ID:server-5e433507-cbe6-e28e-dc54-4ad5060db305 Address:5e433507-cbe6-e28e-dc54-4ad5060db305}]"
2025-07-05T01:10:38.155Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: entering follower state: follower="Node at f2ff7252-4cde-c1c8-c5a5-3fab00bd5993 [Follower]" leader-address= leader-id=
2025-07-05T01:10:38.155Z [INFO]  server-5e433507-cbe6-e28e-dc54-4ad5060db305: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0 Address:d19f0848-5c96-3a68-3db5-2ac4ab9e80b0} {Suffrage:Voter ID:server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993 Address:f2ff7252-4cde-c1c8-c5a5-3fab00bd5993} {Suffrage:Voter ID:server-5e433507-cbe6-e28e-dc54-4ad5060db305 Address:5e433507-cbe6-e28e-dc54-4ad5060db305}]"
2025-07-05T01:10:38.155Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:38.155Z [INFO]  server-5e433507-cbe6-e28e-dc54-4ad5060db305: entering follower state: follower="Node at 5e433507-cbe6-e28e-dc54-4ad5060db305 [Follower]" leader-address= leader-id=
2025-07-05T01:10:38.210Z [WARN]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:38.210Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: entering candidate state: node="Node at f2ff7252-4cde-c1c8-c5a5-3fab00bd5993 [Candidate]" term=2
2025-07-05T01:10:38.211Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: election won: term=2 tally=2
2025-07-05T01:10:38.211Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: entering leader state: leader="Node at f2ff7252-4cde-c1c8-c5a5-3fab00bd5993 [Leader]"
2025-07-05T01:10:38.211Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: added peer, starting replication: peer=server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0
2025-07-05T01:10:38.211Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: added peer, starting replication: peer=server-5e433507-cbe6-e28e-dc54-4ad5060db305
2025-07-05T01:10:38.211Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: pipelining replication: peer="{Voter server-5e433507-cbe6-e28e-dc54-4ad5060db305 5e433507-cbe6-e28e-dc54-4ad5060db305}"
2025-07-05T01:10:38.211Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: pipelining replication: peer="{Voter server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0 d19f0848-5c96-3a68-3db5-2ac4ab9e80b0}"
2025-07-05T01:10:38.316Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:38.211215139 +0000 UTC m=+13.459952357 (1 nodes), 55.553132ms from start of poll, 56.793889ms from cluster start. Timeout at 2025-07-05 01:10:38.316631995 +0000 UTC m=+13.565369223, 105.416866ms after stability
2025-07-05T01:10:38.316Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: updating configuration: command=RemoveServer server-id=46e7f49c-5843-93fc-23bb-73fd4278eeba server-addr= servers="[{Suffrage:Voter ID:server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0 Address:d19f0848-5c96-3a68-3db5-2ac4ab9e80b0} {Suffrage:Voter ID:server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993 Address:f2ff7252-4cde-c1c8-c5a5-3fab00bd5993} {Suffrage:Voter ID:server-5e433507-cbe6-e28e-dc54-4ad5060db305 Address:5e433507-cbe6-e28e-dc54-4ad5060db305}]"
2025-07-05T01:10:38.316Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: aborting pipeline replication: peer="{Voter server-5e433507-cbe6-e28e-dc54-4ad5060db305 5e433507-cbe6-e28e-dc54-4ad5060db305}"
2025-07-05T01:10:38.316Z [INFO]  server-f2ff7252-4cde-c1c8-c5a5-3fab00bd5993: aborting pipeline replication: peer="{Voter server-d19f0848-5c96-3a68-3db5-2ac4ab9e80b0 d19f0848-5c96-3a68-3db5-2ac4ab9e80b0}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2025-07-05T01:10:38.319Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1941fb82-7afd-6fc8-a95c-c7492a52d427 Address:1941fb82-7afd-6fc8-a95c-c7492a52d427}]"
2025-07-05T01:10:38.319Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:38.319Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: entering follower state: follower="Node at 1941fb82-7afd-6fc8-a95c-c7492a52d427 [Follower]" leader-address= leader-id=
2025-07-05T01:10:38.396Z [WARN]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:38.396Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: entering candidate state: node="Node at 1941fb82-7afd-6fc8-a95c-c7492a52d427 [Candidate]" term=2
2025-07-05T01:10:38.396Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: election won: term=2 tally=1
2025-07-05T01:10:38.396Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: entering leader state: leader="Node at 1941fb82-7afd-6fc8-a95c-c7492a52d427 [Leader]"
2025-07-05T01:10:38.501Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:38.39688503 +0000 UTC m=+13.645622248 (1 nodes), 77.039079ms from start of poll, 77.229496ms from cluster start. Timeout at 2025-07-05 01:10:38.50191438 +0000 UTC m=+13.750651598, 105.02935ms after stability
2025-07-05T01:10:38.502Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: starting snapshot up to: index=102
2025-07-05T01:10:38.502Z [INFO]  creating new snapshot: path=/tmp/raft1361701960/snapshots/2-102-1751677838502.tmp
2025-07-05T01:10:38.503Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: compacting logs: from=1 to=92
2025-07-05T01:10:38.503Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: snapshot complete up to: index=102
2025-07-05T01:10:38.503Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: starting restore from snapshot: id=2-102-1751677838502 last-index=102 last-term=2 size-in-bytes=693
2025-07-05T01:10:38.503Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: snapshot restore progress: id=2-102-1751677838502 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-05T01:10:38.503Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: restored from snapshot: id=2-102-1751677838502 last-index=102 last-term=2 size-in-bytes=693
2025-07-05T01:10:38.503Z [INFO]  server-1941fb82-7afd-6fc8-a95c-c7492a52d427: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1941fb82-7afd-6fc8-a95c-c7492a52d427 Address:1941fb82-7afd-6fc8-a95c-c7492a52d427}]"
--- PASS: TestRaft_SnapshotRestore (0.19s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-07-05T01:10:38.505Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f9b1681-a6a7-ca47-3ade-43442c678af5 Address:3f9b1681-a6a7-ca47-3ade-43442c678af5}]"
2025-07-05T01:10:38.505Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:38.505Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: entering follower state: follower="Node at 3f9b1681-a6a7-ca47-3ade-43442c678af5 [Follower]" leader-address= leader-id=
2025-07-05T01:10:38.594Z [WARN]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:38.594Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: entering candidate state: node="Node at 3f9b1681-a6a7-ca47-3ade-43442c678af5 [Candidate]" term=2
2025-07-05T01:10:38.594Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: election won: term=2 tally=1
2025-07-05T01:10:38.594Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: entering leader state: leader="Node at 3f9b1681-a6a7-ca47-3ade-43442c678af5 [Leader]"
2025-07-05T01:10:38.699Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:38.594338024 +0000 UTC m=+13.843075242 (1 nodes), 88.795071ms from start of poll, 88.973786ms from cluster start. Timeout at 2025-07-05 01:10:38.699853095 +0000 UTC m=+13.948590313, 105.515071ms after stability
2025-07-05T01:10:38.700Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: starting snapshot up to: index=102
2025-07-05T01:10:38.700Z [INFO]  creating new snapshot: path=/tmp/raft475702696/snapshots/2-102-1751677838700.tmp
2025-07-05T01:10:38.701Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: compacting logs: from=1 to=92
2025-07-05T01:10:38.701Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: snapshot complete up to: index=102
2025-07-05T01:10:38.701Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: starting restore from snapshot: id=2-102-1751677838700 last-index=102 last-term=2 size-in-bytes=693
2025-07-05T01:10:38.701Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: snapshot restore progress: id=2-102-1751677838700 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-05T01:10:38.701Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: restored from snapshot: id=2-102-1751677838700 last-index=102 last-term=2 size-in-bytes=693
2025-07-05T01:10:38.701Z [INFO]  server-3f9b1681-a6a7-ca47-3ade-43442c678af5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f9b1681-a6a7-ca47-3ade-43442c678af5 Address:3f9b1681-a6a7-ca47-3ade-43442c678af5}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-07-05T01:10:38.703Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2 Address:fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2}]"
2025-07-05T01:10:38.703Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: entering follower state: follower="Node at fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2 [Follower]" leader-address= leader-id=
2025-07-05T01:10:38.703Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:38.767Z [WARN]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:38.767Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: entering candidate state: node="Node at fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2 [Candidate]" term=2
2025-07-05T01:10:38.767Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: election won: term=2 tally=1
2025-07-05T01:10:38.767Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: entering leader state: leader="Node at fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2 [Leader]"
2025-07-05T01:10:38.873Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:38.767690279 +0000 UTC m=+14.016427497 (1 nodes), 64.374271ms from start of poll, 64.646762ms from cluster start. Timeout at 2025-07-05 01:10:38.873550998 +0000 UTC m=+14.122288216, 105.860719ms after stability
2025-07-05T01:10:38.874Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: starting snapshot up to: index=102
2025-07-05T01:10:38.874Z [INFO]  creating new snapshot: path=/tmp/raft3008026436/snapshots/2-102-1751677838874.tmp
2025-07-05T01:10:38.875Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: compacting logs: from=1 to=92
2025-07-05T01:10:38.875Z [INFO]  server-fd55e1b0-f1d6-8064-bdc8-1f90cd1f63d2: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.18s)
=== RUN   TestRaft_NoRestoreOnStart
2025-07-05T01:10:38.883Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-643c1ca6-6043-e1c6-ba58-b47879be11e5 Address:643c1ca6-6043-e1c6-ba58-b47879be11e5}]"
2025-07-05T01:10:38.883Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:38.884Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: entering follower state: follower="Node at 643c1ca6-6043-e1c6-ba58-b47879be11e5 [Follower]" leader-address= leader-id=
2025-07-05T01:10:38.965Z [WARN]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:38.965Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: entering candidate state: node="Node at 643c1ca6-6043-e1c6-ba58-b47879be11e5 [Candidate]" term=2
2025-07-05T01:10:38.965Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: election won: term=2 tally=1
2025-07-05T01:10:38.965Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: entering leader state: leader="Node at 643c1ca6-6043-e1c6-ba58-b47879be11e5 [Leader]"
2025-07-05T01:10:39.071Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:38.965865147 +0000 UTC m=+14.214602355 (1 nodes), 82.196578ms from start of poll, 82.418585ms from cluster start. Timeout at 2025-07-05 01:10:39.07195729 +0000 UTC m=+14.320694518, 106.092163ms after stability
2025-07-05T01:10:39.072Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: starting snapshot up to: index=102
2025-07-05T01:10:39.072Z [INFO]  creating new snapshot: path=/tmp/raft2629126442/snapshots/2-102-1751677839072.tmp
2025-07-05T01:10:39.073Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: compacting logs: from=1 to=92
2025-07-05T01:10:39.073Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: snapshot complete up to: index=102
2025-07-05T01:10:39.073Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-643c1ca6-6043-e1c6-ba58-b47879be11e5 Address:643c1ca6-6043-e1c6-ba58-b47879be11e5}]"
--- PASS: TestRaft_NoRestoreOnStart (0.19s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-07-05T01:10:39.073Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a469c3e8-2d49-137c-0821-ac119b39bd14 Address:a469c3e8-2d49-137c-0821-ac119b39bd14}]"
2025-07-05T01:10:39.073Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:39.073Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: entering follower state: follower="Node at 643c1ca6-6043-e1c6-ba58-b47879be11e5 [Follower]" leader-address= leader-id=
2025-07-05T01:10:39.073Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: entering follower state: follower="Node at a469c3e8-2d49-137c-0821-ac119b39bd14 [Follower]" leader-address= leader-id=
2025-07-05T01:10:39.159Z [WARN]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:39.159Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: entering candidate state: node="Node at 643c1ca6-6043-e1c6-ba58-b47879be11e5 [Candidate]" term=3
2025-07-05T01:10:39.159Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: election won: term=3 tally=1
2025-07-05T01:10:39.159Z [INFO]  server-643c1ca6-6043-e1c6-ba58-b47879be11e5: entering leader state: leader="Node at 643c1ca6-6043-e1c6-ba58-b47879be11e5 [Leader]"
2025-07-05T01:10:39.160Z [WARN]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:39.160Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: entering candidate state: node="Node at a469c3e8-2d49-137c-0821-ac119b39bd14 [Candidate]" term=2
2025-07-05T01:10:39.160Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: election won: term=2 tally=1
2025-07-05T01:10:39.160Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: entering leader state: leader="Node at a469c3e8-2d49-137c-0821-ac119b39bd14 [Leader]"
2025-07-05T01:10:39.266Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:39.160652213 +0000 UTC m=+14.409389421 (1 nodes), 86.968446ms from start of poll, 87.049839ms from cluster start. Timeout at 2025-07-05 01:10:39.266352351 +0000 UTC m=+14.515089569, 105.700148ms after stability
2025-07-05T01:10:39.274Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: starting snapshot up to: index=102
2025-07-05T01:10:39.274Z [INFO]  creating new snapshot: path=/tmp/raft4046282437/snapshots/2-102-1751677839274.tmp
2025-07-05T01:10:39.274Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: compacting logs: from=1 to=92
2025-07-05T01:10:39.274Z [INFO]  server-a469c3e8-2d49-137c-0821-ac119b39bd14: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.30s)
=== RUN   TestRaft_UserSnapshot
2025-07-05T01:10:39.373Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7acff6f-db2e-4064-a72e-4f6947fdea4a Address:e7acff6f-db2e-4064-a72e-4f6947fdea4a}]"
2025-07-05T01:10:39.374Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:39.374Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: entering follower state: follower="Node at e7acff6f-db2e-4064-a72e-4f6947fdea4a [Follower]" leader-address= leader-id=
2025-07-05T01:10:39.438Z [WARN]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:39.438Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: entering candidate state: node="Node at e7acff6f-db2e-4064-a72e-4f6947fdea4a [Candidate]" term=2
2025-07-05T01:10:39.438Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: election won: term=2 tally=1
2025-07-05T01:10:39.438Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: entering leader state: leader="Node at e7acff6f-db2e-4064-a72e-4f6947fdea4a [Leader]"
2025-07-05T01:10:39.544Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:39.438468108 +0000 UTC m=+14.687205316 (1 nodes), 64.450595ms from start of poll, 64.58088ms from cluster start. Timeout at 2025-07-05 01:10:39.544185738 +0000 UTC m=+14.792922956, 105.71764ms after stability
2025-07-05T01:10:39.544Z [ERROR] server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: failed to take snapshot: error="nothing new to snapshot"
2025-07-05T01:10:39.544Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: starting snapshot up to: index=12
2025-07-05T01:10:39.544Z [INFO]  creating new snapshot: path=/tmp/raft3061055092/snapshots/2-12-1751677839544.tmp
2025-07-05T01:10:39.544Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: compacting logs: from=1 to=2
2025-07-05T01:10:39.544Z [INFO]  server-e7acff6f-db2e-4064-a72e-4f6947fdea4a: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.18s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-07-05T01:10:39.551Z [INFO]  server-375929fc-ade7-0b98-9c2d-890aed33a767: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-375929fc-ade7-0b98-9c2d-890aed33a767 Address:375929fc-ade7-0b98-9c2d-890aed33a767} {Suffrage:Voter ID:server-0a69a901-9026-bcfe-b092-2f7ca640536d Address:0a69a901-9026-bcfe-b092-2f7ca640536d} {Suffrage:Voter ID:server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc Address:ec3007b7-07ba-5266-c2eb-11fa83e8c5cc}]"
2025-07-05T01:10:39.551Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-375929fc-ade7-0b98-9c2d-890aed33a767 Address:375929fc-ade7-0b98-9c2d-890aed33a767} {Suffrage:Voter ID:server-0a69a901-9026-bcfe-b092-2f7ca640536d Address:0a69a901-9026-bcfe-b092-2f7ca640536d} {Suffrage:Voter ID:server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc Address:ec3007b7-07ba-5266-c2eb-11fa83e8c5cc}]"
2025-07-05T01:10:39.552Z [INFO]  server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-375929fc-ade7-0b98-9c2d-890aed33a767 Address:375929fc-ade7-0b98-9c2d-890aed33a767} {Suffrage:Voter ID:server-0a69a901-9026-bcfe-b092-2f7ca640536d Address:0a69a901-9026-bcfe-b092-2f7ca640536d} {Suffrage:Voter ID:server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc Address:ec3007b7-07ba-5266-c2eb-11fa83e8c5cc}]"
2025-07-05T01:10:39.552Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:39.552Z [INFO]  server-375929fc-ade7-0b98-9c2d-890aed33a767: entering follower state: follower="Node at 375929fc-ade7-0b98-9c2d-890aed33a767 [Follower]" leader-address= leader-id=
2025-07-05T01:10:39.552Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: entering follower state: follower="Node at 0a69a901-9026-bcfe-b092-2f7ca640536d [Follower]" leader-address= leader-id=
2025-07-05T01:10:39.552Z [INFO]  server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc: entering follower state: follower="Node at ec3007b7-07ba-5266-c2eb-11fa83e8c5cc [Follower]" leader-address= leader-id=
2025-07-05T01:10:40.380Z [WARN]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:40.380Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: entering candidate state: node="Node at 0a69a901-9026-bcfe-b092-2f7ca640536d [Candidate]" term=2
2025-07-05T01:10:40.380Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: election won: term=2 tally=2
2025-07-05T01:10:40.380Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: entering leader state: leader="Node at 0a69a901-9026-bcfe-b092-2f7ca640536d [Leader]"
2025-07-05T01:10:40.380Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: added peer, starting replication: peer=server-375929fc-ade7-0b98-9c2d-890aed33a767
2025-07-05T01:10:40.380Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: added peer, starting replication: peer=server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc
2025-07-05T01:10:40.380Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: pipelining replication: peer="{Voter server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc ec3007b7-07ba-5266-c2eb-11fa83e8c5cc}"
2025-07-05T01:10:40.380Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: pipelining replication: peer="{Voter server-375929fc-ade7-0b98-9c2d-890aed33a767 375929fc-ade7-0b98-9c2d-890aed33a767}"
2025-07-05T01:10:41.386Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:40.380524783 +0000 UTC m=+15.629261992 (1 nodes), 828.428894ms from start of poll, 828.880852ms from cluster start. Timeout at 2025-07-05 01:10:41.386019766 +0000 UTC m=+16.634756974, 1.005494982s after stability
2025-07-05T01:10:41.386Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: starting snapshot up to: index=12
2025-07-05T01:10:41.386Z [INFO]  creating new snapshot: path=/tmp/raft3720456206/snapshots/2-12-1751677841386.tmp
2025-07-05T01:10:41.386Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: snapshot complete up to: index=12
2025-07-05T01:10:41.387Z [INFO]  creating new snapshot: path=/tmp/raft3720456206/snapshots/2-23-1751677841387.tmp
2025-07-05T01:10:41.387Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: copied to local snapshot: bytes=71
2025-07-05T01:10:41.387Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: snapshot restore progress: id=2-23-1751677841387 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:41.387Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: restored user snapshot: index=23
2025-07-05T01:10:41.387Z [ERROR] server-0a69a901-9026-bcfe-b092-2f7ca640536d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:41.387Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: aborting pipeline replication: peer="{Voter server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc ec3007b7-07ba-5266-c2eb-11fa83e8c5cc}"
2025-07-05T01:10:41.387Z [ERROR] server-0a69a901-9026-bcfe-b092-2f7ca640536d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:41.387Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: aborting pipeline replication: peer="{Voter server-375929fc-ade7-0b98-9c2d-890aed33a767 375929fc-ade7-0b98-9c2d-890aed33a767}"
2025-07-05T01:10:41.395Z [ERROR] server-0a69a901-9026-bcfe-b092-2f7ca640536d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:41.395Z [INFO]  creating new snapshot: path=/tmp/raft470402632/snapshots/2-23-1751677841395.tmp
2025-07-05T01:10:41.395Z [INFO]  server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:41.395Z [INFO]  server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc: copied to local snapshot: bytes=71
2025-07-05T01:10:41.396Z [INFO]  server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc: snapshot restore progress: id=2-23-1751677841395 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:41.396Z [INFO]  server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc: Installed remote snapshot
2025-07-05T01:10:41.396Z [WARN]  server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc: skipping application of old log: index=22
2025-07-05T01:10:41.396Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: pipelining replication: peer="{Voter server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc ec3007b7-07ba-5266-c2eb-11fa83e8c5cc}"
2025-07-05T01:10:41.396Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:41.396Z [ERROR] server-0a69a901-9026-bcfe-b092-2f7ca640536d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:41.396Z [INFO]  creating new snapshot: path=/tmp/raft1582551592/snapshots/2-23-1751677841396.tmp
2025-07-05T01:10:41.396Z [INFO]  server-375929fc-ade7-0b98-9c2d-890aed33a767: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:41.396Z [INFO]  server-375929fc-ade7-0b98-9c2d-890aed33a767: copied to local snapshot: bytes=71
2025-07-05T01:10:41.396Z [INFO]  server-375929fc-ade7-0b98-9c2d-890aed33a767: snapshot restore progress: id=2-23-1751677841396 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:41.396Z [INFO]  server-375929fc-ade7-0b98-9c2d-890aed33a767: Installed remote snapshot
2025-07-05T01:10:41.396Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: pipelining replication: peer="{Voter server-375929fc-ade7-0b98-9c2d-890aed33a767 375929fc-ade7-0b98-9c2d-890aed33a767}"
2025-07-05T01:10:42.402Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:41.396194494 +0000 UTC m=+16.644931712 (2 nodes), 471ns from start of poll, 1.844550572s from cluster start. Timeout at 2025-07-05 01:10:42.401981334 +0000 UTC m=+17.650718552, 1.00578684s after stability
2025-07-05T01:10:42.413Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: aborting pipeline replication: peer="{Voter server-ec3007b7-07ba-5266-c2eb-11fa83e8c5cc ec3007b7-07ba-5266-c2eb-11fa83e8c5cc}"
2025-07-05T01:10:42.413Z [INFO]  server-0a69a901-9026-bcfe-b092-2f7ca640536d: aborting pipeline replication: peer="{Voter server-375929fc-ade7-0b98-9c2d-890aed33a767 375929fc-ade7-0b98-9c2d-890aed33a767}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-07-05T01:10:42.415Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a Address:bfb1c9a9-27ea-78d7-cbad-08e442e3420a} {Suffrage:Voter ID:server-09fd34be-f783-b802-4c70-c4f116219446 Address:09fd34be-f783-b802-4c70-c4f116219446} {Suffrage:Voter ID:server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef Address:cdd16c5d-901d-b142-5fba-cb4af1bf29ef}]"
2025-07-05T01:10:42.415Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a Address:bfb1c9a9-27ea-78d7-cbad-08e442e3420a} {Suffrage:Voter ID:server-09fd34be-f783-b802-4c70-c4f116219446 Address:09fd34be-f783-b802-4c70-c4f116219446} {Suffrage:Voter ID:server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef Address:cdd16c5d-901d-b142-5fba-cb4af1bf29ef}]"
2025-07-05T01:10:42.415Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a Address:bfb1c9a9-27ea-78d7-cbad-08e442e3420a} {Suffrage:Voter ID:server-09fd34be-f783-b802-4c70-c4f116219446 Address:09fd34be-f783-b802-4c70-c4f116219446} {Suffrage:Voter ID:server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef Address:cdd16c5d-901d-b142-5fba-cb4af1bf29ef}]"
2025-07-05T01:10:42.415Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:42.415Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: entering follower state: follower="Node at bfb1c9a9-27ea-78d7-cbad-08e442e3420a [Follower]" leader-address= leader-id=
2025-07-05T01:10:42.415Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: entering follower state: follower="Node at 09fd34be-f783-b802-4c70-c4f116219446 [Follower]" leader-address= leader-id=
2025-07-05T01:10:42.415Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: entering follower state: follower="Node at cdd16c5d-901d-b142-5fba-cb4af1bf29ef [Follower]" leader-address= leader-id=
2025-07-05T01:10:42.991Z [WARN]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:42.991Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: entering candidate state: node="Node at cdd16c5d-901d-b142-5fba-cb4af1bf29ef [Candidate]" term=2
2025-07-05T01:10:42.991Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: election won: term=2 tally=2
2025-07-05T01:10:42.991Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: entering leader state: leader="Node at cdd16c5d-901d-b142-5fba-cb4af1bf29ef [Leader]"
2025-07-05T01:10:42.991Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: added peer, starting replication: peer=server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a
2025-07-05T01:10:42.991Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: added peer, starting replication: peer=server-09fd34be-f783-b802-4c70-c4f116219446
2025-07-05T01:10:42.991Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: pipelining replication: peer="{Voter server-09fd34be-f783-b802-4c70-c4f116219446 09fd34be-f783-b802-4c70-c4f116219446}"
2025-07-05T01:10:42.991Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: pipelining replication: peer="{Voter server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a bfb1c9a9-27ea-78d7-cbad-08e442e3420a}"
2025-07-05T01:10:43.997Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:42.991705306 +0000 UTC m=+18.240442524 (1 nodes), 576.31482ms from start of poll, 576.554349ms from cluster start. Timeout at 2025-07-05 01:10:43.997189599 +0000 UTC m=+19.245926807, 1.005484283s after stability
2025-07-05T01:10:43.997Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: starting snapshot up to: index=12
2025-07-05T01:10:43.997Z [INFO]  creating new snapshot: path=/tmp/raft1193014887/snapshots/2-12-1751677843997.tmp
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: snapshot complete up to: index=12
2025-07-05T01:10:43.998Z [INFO]  creating new snapshot: path=/tmp/raft1193014887/snapshots/2-23-1751677843998.tmp
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: copied to local snapshot: bytes=71
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: snapshot restore progress: id=2-23-1751677843998 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: removing all old logs from log store
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: compacting logs: from=1 to=22
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: restored user snapshot: index=23
2025-07-05T01:10:43.998Z [ERROR] server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: failed to get log: index=22 error="log not found"
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: aborting pipeline replication: peer="{Voter server-09fd34be-f783-b802-4c70-c4f116219446 09fd34be-f783-b802-4c70-c4f116219446}"
2025-07-05T01:10:43.998Z [ERROR] server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: failed to get log: index=22 error="log not found"
2025-07-05T01:10:43.998Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: aborting pipeline replication: peer="{Voter server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a bfb1c9a9-27ea-78d7-cbad-08e442e3420a}"
2025-07-05T01:10:44.006Z [ERROR] server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: failed to get log: index=22 error="log not found"
2025-07-05T01:10:44.007Z [INFO]  creating new snapshot: path=/tmp/raft2410160766/snapshots/2-23-1751677844007.tmp
2025-07-05T01:10:44.007Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:44.007Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: copied to local snapshot: bytes=71
2025-07-05T01:10:44.007Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: snapshot restore progress: id=2-23-1751677844007 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:44.007Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: removing all old logs from log store
2025-07-05T01:10:44.007Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: compacting logs: from=1 to=22
2025-07-05T01:10:44.007Z [INFO]  server-09fd34be-f783-b802-4c70-c4f116219446: Installed remote snapshot
2025-07-05T01:10:44.007Z [WARN]  server-09fd34be-f783-b802-4c70-c4f116219446: skipping application of old log: index=22
2025-07-05T01:10:44.007Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: pipelining replication: peer="{Voter server-09fd34be-f783-b802-4c70-c4f116219446 09fd34be-f783-b802-4c70-c4f116219446}"
2025-07-05T01:10:44.007Z [ERROR] server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: failed to get log: index=22 error="log not found"
2025-07-05T01:10:44.007Z [INFO]  creating new snapshot: path=/tmp/raft4219740805/snapshots/2-23-1751677844007.tmp
2025-07-05T01:10:44.007Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:44.008Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: copied to local snapshot: bytes=71
2025-07-05T01:10:44.008Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: snapshot restore progress: id=2-23-1751677844007 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:44.008Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: removing all old logs from log store
2025-07-05T01:10:44.008Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: compacting logs: from=1 to=22
2025-07-05T01:10:44.008Z [INFO]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: Installed remote snapshot
2025-07-05T01:10:44.008Z [WARN]  server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a: skipping application of old log: index=22
2025-07-05T01:10:44.008Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: pipelining replication: peer="{Voter server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a bfb1c9a9-27ea-78d7-cbad-08e442e3420a}"
2025-07-05T01:10:44.008Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:45.014Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:44.008454931 +0000 UTC m=+19.257192139 (2 nodes), 921ns from start of poll, 1.593303964s from cluster start. Timeout at 2025-07-05 01:10:45.014229668 +0000 UTC m=+20.262966877, 1.005774738s after stability
2025-07-05T01:10:45.025Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: aborting pipeline replication: peer="{Voter server-09fd34be-f783-b802-4c70-c4f116219446 09fd34be-f783-b802-4c70-c4f116219446}"
2025-07-05T01:10:45.025Z [INFO]  server-cdd16c5d-901d-b142-5fba-cb4af1bf29ef: aborting pipeline replication: peer="{Voter server-bfb1c9a9-27ea-78d7-cbad-08e442e3420a bfb1c9a9-27ea-78d7-cbad-08e442e3420a}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-07-05T01:10:45.026Z [INFO]  server-b6609031-2f14-075f-d8dd-bc9468bd8e72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6609031-2f14-075f-d8dd-bc9468bd8e72 Address:b6609031-2f14-075f-d8dd-bc9468bd8e72} {Suffrage:Voter ID:server-64624349-8719-5e8f-43af-9a7f2fd17100 Address:64624349-8719-5e8f-43af-9a7f2fd17100} {Suffrage:Voter ID:server-7571f033-69f3-71f2-2606-98b2730be9a0 Address:7571f033-69f3-71f2-2606-98b2730be9a0}]"
2025-07-05T01:10:45.027Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6609031-2f14-075f-d8dd-bc9468bd8e72 Address:b6609031-2f14-075f-d8dd-bc9468bd8e72} {Suffrage:Voter ID:server-64624349-8719-5e8f-43af-9a7f2fd17100 Address:64624349-8719-5e8f-43af-9a7f2fd17100} {Suffrage:Voter ID:server-7571f033-69f3-71f2-2606-98b2730be9a0 Address:7571f033-69f3-71f2-2606-98b2730be9a0}]"
2025-07-05T01:10:45.027Z [INFO]  server-7571f033-69f3-71f2-2606-98b2730be9a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6609031-2f14-075f-d8dd-bc9468bd8e72 Address:b6609031-2f14-075f-d8dd-bc9468bd8e72} {Suffrage:Voter ID:server-64624349-8719-5e8f-43af-9a7f2fd17100 Address:64624349-8719-5e8f-43af-9a7f2fd17100} {Suffrage:Voter ID:server-7571f033-69f3-71f2-2606-98b2730be9a0 Address:7571f033-69f3-71f2-2606-98b2730be9a0}]"
2025-07-05T01:10:45.027Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:45.027Z [INFO]  server-b6609031-2f14-075f-d8dd-bc9468bd8e72: entering follower state: follower="Node at b6609031-2f14-075f-d8dd-bc9468bd8e72 [Follower]" leader-address= leader-id=
2025-07-05T01:10:45.027Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: entering follower state: follower="Node at 64624349-8719-5e8f-43af-9a7f2fd17100 [Follower]" leader-address= leader-id=
2025-07-05T01:10:45.027Z [INFO]  server-7571f033-69f3-71f2-2606-98b2730be9a0: entering follower state: follower="Node at 7571f033-69f3-71f2-2606-98b2730be9a0 [Follower]" leader-address= leader-id=
2025-07-05T01:10:45.653Z [WARN]  server-64624349-8719-5e8f-43af-9a7f2fd17100: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:45.653Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: entering candidate state: node="Node at 64624349-8719-5e8f-43af-9a7f2fd17100 [Candidate]" term=2
2025-07-05T01:10:45.653Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: election won: term=2 tally=2
2025-07-05T01:10:45.653Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: entering leader state: leader="Node at 64624349-8719-5e8f-43af-9a7f2fd17100 [Leader]"
2025-07-05T01:10:45.653Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: added peer, starting replication: peer=server-b6609031-2f14-075f-d8dd-bc9468bd8e72
2025-07-05T01:10:45.653Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: added peer, starting replication: peer=server-7571f033-69f3-71f2-2606-98b2730be9a0
2025-07-05T01:10:45.653Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: pipelining replication: peer="{Voter server-7571f033-69f3-71f2-2606-98b2730be9a0 7571f033-69f3-71f2-2606-98b2730be9a0}"
2025-07-05T01:10:45.653Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: pipelining replication: peer="{Voter server-b6609031-2f14-075f-d8dd-bc9468bd8e72 b6609031-2f14-075f-d8dd-bc9468bd8e72}"
2025-07-05T01:10:46.658Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:45.653685829 +0000 UTC m=+20.902423047 (1 nodes), 626.58657ms from start of poll, 626.806973ms from cluster start. Timeout at 2025-07-05 01:10:46.658917087 +0000 UTC m=+21.907654295, 1.005231248s after stability
2025-07-05T01:10:46.659Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: starting snapshot up to: index=12
2025-07-05T01:10:46.659Z [INFO]  creating new snapshot: path=/tmp/raft637972548/snapshots/2-12-1751677846659.tmp
2025-07-05T01:10:46.659Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: snapshot complete up to: index=12
2025-07-05T01:10:46.660Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c Address:12ef3e68-ca8f-d1c5-ce59-24a525e9d41c} {Suffrage:Voter ID:server-f1bec6b6-feee-8298-3561-b0a5b2624fd3 Address:f1bec6b6-feee-8298-3561-b0a5b2624fd3} {Suffrage:Voter ID:server-63a28faa-02f4-790d-6857-3dcaff8ba173 Address:63a28faa-02f4-790d-6857-3dcaff8ba173}]"
2025-07-05T01:10:46.660Z [INFO]  server-f1bec6b6-feee-8298-3561-b0a5b2624fd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c Address:12ef3e68-ca8f-d1c5-ce59-24a525e9d41c} {Suffrage:Voter ID:server-f1bec6b6-feee-8298-3561-b0a5b2624fd3 Address:f1bec6b6-feee-8298-3561-b0a5b2624fd3} {Suffrage:Voter ID:server-63a28faa-02f4-790d-6857-3dcaff8ba173 Address:63a28faa-02f4-790d-6857-3dcaff8ba173}]"
2025-07-05T01:10:46.660Z [INFO]  server-63a28faa-02f4-790d-6857-3dcaff8ba173: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c Address:12ef3e68-ca8f-d1c5-ce59-24a525e9d41c} {Suffrage:Voter ID:server-f1bec6b6-feee-8298-3561-b0a5b2624fd3 Address:f1bec6b6-feee-8298-3561-b0a5b2624fd3} {Suffrage:Voter ID:server-63a28faa-02f4-790d-6857-3dcaff8ba173 Address:63a28faa-02f4-790d-6857-3dcaff8ba173}]"
2025-07-05T01:10:46.660Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:46.660Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: entering follower state: follower="Node at 12ef3e68-ca8f-d1c5-ce59-24a525e9d41c [Follower]" leader-address= leader-id=
2025-07-05T01:10:46.660Z [INFO]  server-f1bec6b6-feee-8298-3561-b0a5b2624fd3: entering follower state: follower="Node at f1bec6b6-feee-8298-3561-b0a5b2624fd3 [Follower]" leader-address= leader-id=
2025-07-05T01:10:46.660Z [INFO]  server-63a28faa-02f4-790d-6857-3dcaff8ba173: entering follower state: follower="Node at 63a28faa-02f4-790d-6857-3dcaff8ba173 [Follower]" leader-address= leader-id=
2025-07-05T01:10:47.421Z [WARN]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:47.421Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: entering candidate state: node="Node at 12ef3e68-ca8f-d1c5-ce59-24a525e9d41c [Candidate]" term=2
2025-07-05T01:10:47.421Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: election won: term=2 tally=2
2025-07-05T01:10:47.421Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: entering leader state: leader="Node at 12ef3e68-ca8f-d1c5-ce59-24a525e9d41c [Leader]"
2025-07-05T01:10:47.421Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: added peer, starting replication: peer=server-f1bec6b6-feee-8298-3561-b0a5b2624fd3
2025-07-05T01:10:47.421Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: added peer, starting replication: peer=server-63a28faa-02f4-790d-6857-3dcaff8ba173
2025-07-05T01:10:47.421Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: pipelining replication: peer="{Voter server-63a28faa-02f4-790d-6857-3dcaff8ba173 63a28faa-02f4-790d-6857-3dcaff8ba173}"
2025-07-05T01:10:47.422Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: pipelining replication: peer="{Voter server-f1bec6b6-feee-8298-3561-b0a5b2624fd3 f1bec6b6-feee-8298-3561-b0a5b2624fd3}"
2025-07-05T01:10:48.427Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:47.422112478 +0000 UTC m=+22.670849686 (1 nodes), 761.384265ms from start of poll, 761.570454ms from cluster start. Timeout at 2025-07-05 01:10:48.427588615 +0000 UTC m=+23.676325823, 1.005476137s after stability
2025-07-05T01:10:48.427Z [INFO]  creating new snapshot: path=/tmp/raft3408806487/snapshots/2-13-1751677848427.tmp
2025-07-05T01:10:48.428Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: copied to local snapshot: bytes=71
2025-07-05T01:10:48.428Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: snapshot restore progress: id=2-13-1751677848427 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:48.428Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: restored user snapshot: index=13
2025-07-05T01:10:48.428Z [ERROR] server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: failed to get log: index=3 error="log not found"
2025-07-05T01:10:48.428Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: aborting pipeline replication: peer="{Voter server-63a28faa-02f4-790d-6857-3dcaff8ba173 63a28faa-02f4-790d-6857-3dcaff8ba173}"
2025-07-05T01:10:48.428Z [ERROR] server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: failed to get log: index=3 error="log not found"
2025-07-05T01:10:48.429Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: aborting pipeline replication: peer="{Voter server-f1bec6b6-feee-8298-3561-b0a5b2624fd3 f1bec6b6-feee-8298-3561-b0a5b2624fd3}"
2025-07-05T01:10:48.437Z [ERROR] server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: failed to get log: index=3 error="log not found"
2025-07-05T01:10:48.438Z [INFO]  creating new snapshot: path=/tmp/raft733721385/snapshots/2-13-1751677848438.tmp
2025-07-05T01:10:48.438Z [ERROR] server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: failed to get log: index=3 error="log not found"
2025-07-05T01:10:48.439Z [INFO]  creating new snapshot: path=/tmp/raft3066359115/snapshots/2-13-1751677848439.tmp
2025-07-05T01:10:48.439Z [INFO]  server-f1bec6b6-feee-8298-3561-b0a5b2624fd3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:48.439Z [INFO]  server-63a28faa-02f4-790d-6857-3dcaff8ba173: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:48.442Z [INFO]  server-63a28faa-02f4-790d-6857-3dcaff8ba173: copied to local snapshot: bytes=71
2025-07-05T01:10:48.442Z [INFO]  server-f1bec6b6-feee-8298-3561-b0a5b2624fd3: copied to local snapshot: bytes=71
2025-07-05T01:10:48.442Z [INFO]  server-63a28faa-02f4-790d-6857-3dcaff8ba173: snapshot restore progress: id=2-13-1751677848439 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:48.442Z [INFO]  server-63a28faa-02f4-790d-6857-3dcaff8ba173: Installed remote snapshot
2025-07-05T01:10:48.442Z [INFO]  server-f1bec6b6-feee-8298-3561-b0a5b2624fd3: snapshot restore progress: id=2-13-1751677848438 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:48.442Z [INFO]  server-f1bec6b6-feee-8298-3561-b0a5b2624fd3: Installed remote snapshot
2025-07-05T01:10:48.442Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: pipelining replication: peer="{Voter server-63a28faa-02f4-790d-6857-3dcaff8ba173 63a28faa-02f4-790d-6857-3dcaff8ba173}"
2025-07-05T01:10:48.442Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:48.443Z [INFO]  server-12ef3e68-ca8f-d1c5-ce59-24a525e9d41c: pipelining replication: peer="{Voter server-f1bec6b6-feee-8298-3561-b0a5b2624fd3 f1bec6b6-feee-8298-3561-b0a5b2624fd3}"
2025-07-05T01:10:49.448Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:48.442976938 +0000 UTC m=+23.691714156 (2 nodes), 841ns from start of poll, 1.782434924s from cluster start. Timeout at 2025-07-05 01:10:49.448530451 +0000 UTC m=+24.697267669, 1.005553513s after stability
2025-07-05T01:10:49.460Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: aborting pipeline replication: peer="{Voter server-7571f033-69f3-71f2-2606-98b2730be9a0 7571f033-69f3-71f2-2606-98b2730be9a0}"
2025-07-05T01:10:49.460Z [INFO]  server-64624349-8719-5e8f-43af-9a7f2fd17100: aborting pipeline replication: peer="{Voter server-b6609031-2f14-075f-d8dd-bc9468bd8e72 b6609031-2f14-075f-d8dd-bc9468bd8e72}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-07-05T01:10:49.462Z [INFO]  server-942094bb-86c1-e0c4-5829-eb719cf94fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-942094bb-86c1-e0c4-5829-eb719cf94fb2 Address:942094bb-86c1-e0c4-5829-eb719cf94fb2} {Suffrage:Voter ID:server-9caf0429-dba9-99e1-0398-8976a22ca0dd Address:9caf0429-dba9-99e1-0398-8976a22ca0dd} {Suffrage:Voter ID:server-4d1412b4-99e2-6838-109f-c2c39f072914 Address:4d1412b4-99e2-6838-109f-c2c39f072914}]"
2025-07-05T01:10:49.462Z [INFO]  server-9caf0429-dba9-99e1-0398-8976a22ca0dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-942094bb-86c1-e0c4-5829-eb719cf94fb2 Address:942094bb-86c1-e0c4-5829-eb719cf94fb2} {Suffrage:Voter ID:server-9caf0429-dba9-99e1-0398-8976a22ca0dd Address:9caf0429-dba9-99e1-0398-8976a22ca0dd} {Suffrage:Voter ID:server-4d1412b4-99e2-6838-109f-c2c39f072914 Address:4d1412b4-99e2-6838-109f-c2c39f072914}]"
2025-07-05T01:10:49.462Z [INFO]  server-942094bb-86c1-e0c4-5829-eb719cf94fb2: entering follower state: follower="Node at 942094bb-86c1-e0c4-5829-eb719cf94fb2 [Follower]" leader-address= leader-id=
2025-07-05T01:10:49.462Z [INFO]  server-9caf0429-dba9-99e1-0398-8976a22ca0dd: entering follower state: follower="Node at 9caf0429-dba9-99e1-0398-8976a22ca0dd [Follower]" leader-address= leader-id=
2025-07-05T01:10:49.462Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-942094bb-86c1-e0c4-5829-eb719cf94fb2 Address:942094bb-86c1-e0c4-5829-eb719cf94fb2} {Suffrage:Voter ID:server-9caf0429-dba9-99e1-0398-8976a22ca0dd Address:9caf0429-dba9-99e1-0398-8976a22ca0dd} {Suffrage:Voter ID:server-4d1412b4-99e2-6838-109f-c2c39f072914 Address:4d1412b4-99e2-6838-109f-c2c39f072914}]"
2025-07-05T01:10:49.462Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:49.462Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: entering follower state: follower="Node at 4d1412b4-99e2-6838-109f-c2c39f072914 [Follower]" leader-address= leader-id=
2025-07-05T01:10:49.982Z [WARN]  server-4d1412b4-99e2-6838-109f-c2c39f072914: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:49.982Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: entering candidate state: node="Node at 4d1412b4-99e2-6838-109f-c2c39f072914 [Candidate]" term=2
2025-07-05T01:10:49.982Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: election won: term=2 tally=2
2025-07-05T01:10:49.982Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: entering leader state: leader="Node at 4d1412b4-99e2-6838-109f-c2c39f072914 [Leader]"
2025-07-05T01:10:49.982Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: added peer, starting replication: peer=server-942094bb-86c1-e0c4-5829-eb719cf94fb2
2025-07-05T01:10:49.982Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: added peer, starting replication: peer=server-9caf0429-dba9-99e1-0398-8976a22ca0dd
2025-07-05T01:10:49.982Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: pipelining replication: peer="{Voter server-942094bb-86c1-e0c4-5829-eb719cf94fb2 942094bb-86c1-e0c4-5829-eb719cf94fb2}"
2025-07-05T01:10:49.983Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: pipelining replication: peer="{Voter server-9caf0429-dba9-99e1-0398-8976a22ca0dd 9caf0429-dba9-99e1-0398-8976a22ca0dd}"
2025-07-05T01:10:50.988Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:49.983032682 +0000 UTC m=+25.231769900 (1 nodes), 520.74674ms from start of poll, 521.089452ms from cluster start. Timeout at 2025-07-05 01:10:50.988539417 +0000 UTC m=+26.237276635, 1.005506735s after stability
2025-07-05T01:10:50.989Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: starting snapshot up to: index=12
2025-07-05T01:10:50.989Z [INFO]  creating new snapshot: path=/tmp/raft2249480079/snapshots/2-12-1751677850989.tmp
2025-07-05T01:10:51.082Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: snapshot complete up to: index=12
2025-07-05T01:10:51.083Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7 Address:4a99381c-0dfc-d896-93c9-2ea9c94ed6b7} {Suffrage:Voter ID:server-d80eb242-fc6d-ea06-8c15-178b01ef8db4 Address:d80eb242-fc6d-ea06-8c15-178b01ef8db4} {Suffrage:Voter ID:server-08777f3b-9d30-8002-8dce-4682bde50e37 Address:08777f3b-9d30-8002-8dce-4682bde50e37}]"
2025-07-05T01:10:51.083Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: entering follower state: follower="Node at 4a99381c-0dfc-d896-93c9-2ea9c94ed6b7 [Follower]" leader-address= leader-id=
2025-07-05T01:10:51.083Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7 Address:4a99381c-0dfc-d896-93c9-2ea9c94ed6b7} {Suffrage:Voter ID:server-d80eb242-fc6d-ea06-8c15-178b01ef8db4 Address:d80eb242-fc6d-ea06-8c15-178b01ef8db4} {Suffrage:Voter ID:server-08777f3b-9d30-8002-8dce-4682bde50e37 Address:08777f3b-9d30-8002-8dce-4682bde50e37}]"
2025-07-05T01:10:51.084Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7 Address:4a99381c-0dfc-d896-93c9-2ea9c94ed6b7} {Suffrage:Voter ID:server-d80eb242-fc6d-ea06-8c15-178b01ef8db4 Address:d80eb242-fc6d-ea06-8c15-178b01ef8db4} {Suffrage:Voter ID:server-08777f3b-9d30-8002-8dce-4682bde50e37 Address:08777f3b-9d30-8002-8dce-4682bde50e37}]"
2025-07-05T01:10:51.084Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: entering follower state: follower="Node at d80eb242-fc6d-ea06-8c15-178b01ef8db4 [Follower]" leader-address= leader-id=
2025-07-05T01:10:51.084Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: entering follower state: follower="Node at 08777f3b-9d30-8002-8dce-4682bde50e37 [Follower]" leader-address= leader-id=
2025-07-05T01:10:51.084Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:51.819Z [WARN]  server-08777f3b-9d30-8002-8dce-4682bde50e37: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:51.819Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: entering candidate state: node="Node at 08777f3b-9d30-8002-8dce-4682bde50e37 [Candidate]" term=2
2025-07-05T01:10:51.819Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: election won: term=2 tally=2
2025-07-05T01:10:51.819Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: entering leader state: leader="Node at 08777f3b-9d30-8002-8dce-4682bde50e37 [Leader]"
2025-07-05T01:10:51.819Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: added peer, starting replication: peer=server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7
2025-07-05T01:10:51.819Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: added peer, starting replication: peer=server-d80eb242-fc6d-ea06-8c15-178b01ef8db4
2025-07-05T01:10:51.819Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: pipelining replication: peer="{Voter server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7 4a99381c-0dfc-d896-93c9-2ea9c94ed6b7}"
2025-07-05T01:10:51.819Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: pipelining replication: peer="{Voter server-d80eb242-fc6d-ea06-8c15-178b01ef8db4 d80eb242-fc6d-ea06-8c15-178b01ef8db4}"
2025-07-05T01:10:52.825Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:51.819581651 +0000 UTC m=+27.068318869 (1 nodes), 735.303183ms from start of poll, 736.113362ms from cluster start. Timeout at 2025-07-05 01:10:52.825398497 +0000 UTC m=+28.074135715, 1.005816846s after stability
2025-07-05T01:10:52.825Z [INFO]  creating new snapshot: path=/tmp/raft1507067614/snapshots/2-13-1751677852825.tmp
2025-07-05T01:10:52.979Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: copied to local snapshot: bytes=71
2025-07-05T01:10:52.979Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: snapshot restore progress: id=2-13-1751677852825 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:52.979Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: removing all old logs from log store
2025-07-05T01:10:52.979Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: compacting logs: from=1 to=2
2025-07-05T01:10:52.979Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: restored user snapshot: index=13
2025-07-05T01:10:52.979Z [ERROR] server-08777f3b-9d30-8002-8dce-4682bde50e37: failed to get log: index=2 error="log not found"
2025-07-05T01:10:52.979Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: aborting pipeline replication: peer="{Voter server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7 4a99381c-0dfc-d896-93c9-2ea9c94ed6b7}"
2025-07-05T01:10:52.979Z [ERROR] server-08777f3b-9d30-8002-8dce-4682bde50e37: failed to get log: index=2 error="log not found"
2025-07-05T01:10:52.979Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: aborting pipeline replication: peer="{Voter server-d80eb242-fc6d-ea06-8c15-178b01ef8db4 d80eb242-fc6d-ea06-8c15-178b01ef8db4}"
2025-07-05T01:10:52.988Z [ERROR] server-08777f3b-9d30-8002-8dce-4682bde50e37: failed to get log: index=2 error="log not found"
2025-07-05T01:10:52.988Z [ERROR] server-08777f3b-9d30-8002-8dce-4682bde50e37: failed to get log: index=2 error="log not found"
2025-07-05T01:10:52.988Z [INFO]  creating new snapshot: path=/tmp/raft3581698588/snapshots/2-13-1751677852988.tmp
2025-07-05T01:10:52.988Z [INFO]  creating new snapshot: path=/tmp/raft615154107/snapshots/2-13-1751677852988.tmp
2025-07-05T01:10:52.988Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:52.989Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:53.115Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: copied to local snapshot: bytes=71
2025-07-05T01:10:53.115Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: copied to local snapshot: bytes=71
2025-07-05T01:10:53.116Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: snapshot restore progress: id=2-13-1751677852988 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:53.116Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: removing all old logs from log store
2025-07-05T01:10:53.116Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: compacting logs: from=1 to=2
2025-07-05T01:10:53.116Z [INFO]  server-d80eb242-fc6d-ea06-8c15-178b01ef8db4: Installed remote snapshot
2025-07-05T01:10:53.116Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: snapshot restore progress: id=2-13-1751677852988 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:53.116Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: removing all old logs from log store
2025-07-05T01:10:53.116Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: compacting logs: from=1 to=2
2025-07-05T01:10:53.116Z [INFO]  server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7: Installed remote snapshot
2025-07-05T01:10:53.116Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: pipelining replication: peer="{Voter server-4a99381c-0dfc-d896-93c9-2ea9c94ed6b7 4a99381c-0dfc-d896-93c9-2ea9c94ed6b7}"
2025-07-05T01:10:53.116Z [INFO]  server-08777f3b-9d30-8002-8dce-4682bde50e37: pipelining replication: peer="{Voter server-d80eb242-fc6d-ea06-8c15-178b01ef8db4 d80eb242-fc6d-ea06-8c15-178b01ef8db4}"
2025-07-05T01:10:53.116Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:54.123Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:53.116526372 +0000 UTC m=+28.365263590 (2 nodes), 1.573µs from start of poll, 2.033058083s from cluster start. Timeout at 2025-07-05 01:10:54.123067105 +0000 UTC m=+29.371804334, 1.006540744s after stability
2025-07-05T01:10:54.134Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: aborting pipeline replication: peer="{Voter server-9caf0429-dba9-99e1-0398-8976a22ca0dd 9caf0429-dba9-99e1-0398-8976a22ca0dd}"
2025-07-05T01:10:54.135Z [INFO]  server-4d1412b4-99e2-6838-109f-c2c39f072914: aborting pipeline replication: peer="{Voter server-942094bb-86c1-e0c4-5829-eb719cf94fb2 942094bb-86c1-e0c4-5829-eb719cf94fb2}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-07-05T01:10:54.136Z [INFO]  server-9032945a-d4bc-9d0f-4727-835f4df7b455: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9032945a-d4bc-9d0f-4727-835f4df7b455 Address:9032945a-d4bc-9d0f-4727-835f4df7b455} {Suffrage:Voter ID:server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79 Address:c7f847d1-7d5b-3805-4cab-d0d58a4dec79} {Suffrage:Voter ID:server-36b6d695-cbe2-bb97-b818-1d012bc52f6d Address:36b6d695-cbe2-bb97-b818-1d012bc52f6d}]"
2025-07-05T01:10:54.136Z [INFO]  server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9032945a-d4bc-9d0f-4727-835f4df7b455 Address:9032945a-d4bc-9d0f-4727-835f4df7b455} {Suffrage:Voter ID:server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79 Address:c7f847d1-7d5b-3805-4cab-d0d58a4dec79} {Suffrage:Voter ID:server-36b6d695-cbe2-bb97-b818-1d012bc52f6d Address:36b6d695-cbe2-bb97-b818-1d012bc52f6d}]"
2025-07-05T01:10:54.136Z [INFO]  server-9032945a-d4bc-9d0f-4727-835f4df7b455: entering follower state: follower="Node at 9032945a-d4bc-9d0f-4727-835f4df7b455 [Follower]" leader-address= leader-id=
2025-07-05T01:10:54.136Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9032945a-d4bc-9d0f-4727-835f4df7b455 Address:9032945a-d4bc-9d0f-4727-835f4df7b455} {Suffrage:Voter ID:server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79 Address:c7f847d1-7d5b-3805-4cab-d0d58a4dec79} {Suffrage:Voter ID:server-36b6d695-cbe2-bb97-b818-1d012bc52f6d Address:36b6d695-cbe2-bb97-b818-1d012bc52f6d}]"
2025-07-05T01:10:54.136Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:54.136Z [INFO]  server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79: entering follower state: follower="Node at c7f847d1-7d5b-3805-4cab-d0d58a4dec79 [Follower]" leader-address= leader-id=
2025-07-05T01:10:54.136Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: entering follower state: follower="Node at 36b6d695-cbe2-bb97-b818-1d012bc52f6d [Follower]" leader-address= leader-id=
2025-07-05T01:10:54.932Z [WARN]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:54.932Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: entering candidate state: node="Node at 36b6d695-cbe2-bb97-b818-1d012bc52f6d [Candidate]" term=2
2025-07-05T01:10:54.932Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: election won: term=2 tally=2
2025-07-05T01:10:54.932Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: entering leader state: leader="Node at 36b6d695-cbe2-bb97-b818-1d012bc52f6d [Leader]"
2025-07-05T01:10:54.932Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: added peer, starting replication: peer=server-9032945a-d4bc-9d0f-4727-835f4df7b455
2025-07-05T01:10:54.932Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: added peer, starting replication: peer=server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79
2025-07-05T01:10:54.932Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: pipelining replication: peer="{Voter server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79 c7f847d1-7d5b-3805-4cab-d0d58a4dec79}"
2025-07-05T01:10:54.932Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: pipelining replication: peer="{Voter server-9032945a-d4bc-9d0f-4727-835f4df7b455 9032945a-d4bc-9d0f-4727-835f4df7b455}"
2025-07-05T01:10:55.938Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:54.932537242 +0000 UTC m=+30.181274460 (1 nodes), 795.809443ms from start of poll, 796.113573ms from cluster start. Timeout at 2025-07-05 01:10:55.937617597 +0000 UTC m=+31.186354815, 1.005080355s after stability
2025-07-05T01:10:55.939Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: starting snapshot up to: index=12
2025-07-05T01:10:55.939Z [INFO]  creating new snapshot: path=/tmp/raft1236751761/snapshots/2-12-1751677855939.tmp
2025-07-05T01:10:56.039Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: snapshot complete up to: index=12
2025-07-05T01:10:56.040Z [INFO]  creating new snapshot: path=/tmp/raft1236751761/snapshots/2-23-1751677856040.tmp
2025-07-05T01:10:56.041Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: copied to local snapshot: bytes=71
2025-07-05T01:10:56.041Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: snapshot restore progress: id=2-23-1751677856040 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:56.041Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: restored user snapshot: index=23
2025-07-05T01:10:56.041Z [ERROR] server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:56.041Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: aborting pipeline replication: peer="{Voter server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79 c7f847d1-7d5b-3805-4cab-d0d58a4dec79}"
2025-07-05T01:10:56.041Z [ERROR] server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:56.041Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: aborting pipeline replication: peer="{Voter server-9032945a-d4bc-9d0f-4727-835f4df7b455 9032945a-d4bc-9d0f-4727-835f4df7b455}"
2025-07-05T01:10:56.048Z [ERROR] server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:56.048Z [INFO]  creating new snapshot: path=/tmp/raft3089706266/snapshots/2-23-1751677856048.tmp
2025-07-05T01:10:56.048Z [INFO]  server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:56.049Z [INFO]  server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79: copied to local snapshot: bytes=71
2025-07-05T01:10:56.049Z [ERROR] server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: failed to get log: index=23 error="log not found"
2025-07-05T01:10:56.049Z [INFO]  server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79: snapshot restore progress: id=2-23-1751677856048 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:56.049Z [INFO]  server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79: Installed remote snapshot
2025-07-05T01:10:56.049Z [WARN]  server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79: skipping application of old log: index=22
2025-07-05T01:10:56.049Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: pipelining replication: peer="{Voter server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79 c7f847d1-7d5b-3805-4cab-d0d58a4dec79}"
2025-07-05T01:10:56.049Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:56.049Z [INFO]  creating new snapshot: path=/tmp/raft1022200464/snapshots/2-23-1751677856049.tmp
2025-07-05T01:10:56.049Z [INFO]  server-9032945a-d4bc-9d0f-4727-835f4df7b455: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:56.050Z [INFO]  server-9032945a-d4bc-9d0f-4727-835f4df7b455: copied to local snapshot: bytes=71
2025-07-05T01:10:56.050Z [INFO]  server-9032945a-d4bc-9d0f-4727-835f4df7b455: snapshot restore progress: id=2-23-1751677856049 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:56.050Z [INFO]  server-9032945a-d4bc-9d0f-4727-835f4df7b455: Installed remote snapshot
2025-07-05T01:10:56.050Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: pipelining replication: peer="{Voter server-9032945a-d4bc-9d0f-4727-835f4df7b455 9032945a-d4bc-9d0f-4727-835f4df7b455}"
2025-07-05T01:10:57.055Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:56.049532337 +0000 UTC m=+31.298269555 (2 nodes), 1.082µs from start of poll, 1.913108668s from cluster start. Timeout at 2025-07-05 01:10:57.055455152 +0000 UTC m=+32.304192370, 1.005922815s after stability
2025-07-05T01:10:57.068Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: aborting pipeline replication: peer="{Voter server-c7f847d1-7d5b-3805-4cab-d0d58a4dec79 c7f847d1-7d5b-3805-4cab-d0d58a4dec79}"
2025-07-05T01:10:57.068Z [INFO]  server-36b6d695-cbe2-bb97-b818-1d012bc52f6d: aborting pipeline replication: peer="{Voter server-9032945a-d4bc-9d0f-4727-835f4df7b455 9032945a-d4bc-9d0f-4727-835f4df7b455}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-07-05T01:10:57.071Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-945147f5-3c72-32d9-d0cd-e80d4acb1853 Address:945147f5-3c72-32d9-d0cd-e80d4acb1853} {Suffrage:Voter ID:server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856 Address:23c6ac6e-ce18-61c3-f0e8-d3aa276b7856} {Suffrage:Voter ID:server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092 Address:6e8d7517-9c7d-efb4-c2b0-9b6a05317092}]"
2025-07-05T01:10:57.074Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: entering follower state: follower="Node at 945147f5-3c72-32d9-d0cd-e80d4acb1853 [Follower]" leader-address= leader-id=
2025-07-05T01:10:57.075Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-945147f5-3c72-32d9-d0cd-e80d4acb1853 Address:945147f5-3c72-32d9-d0cd-e80d4acb1853} {Suffrage:Voter ID:server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856 Address:23c6ac6e-ce18-61c3-f0e8-d3aa276b7856} {Suffrage:Voter ID:server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092 Address:6e8d7517-9c7d-efb4-c2b0-9b6a05317092}]"
2025-07-05T01:10:57.075Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: entering follower state: follower="Node at 23c6ac6e-ce18-61c3-f0e8-d3aa276b7856 [Follower]" leader-address= leader-id=
2025-07-05T01:10:57.075Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-945147f5-3c72-32d9-d0cd-e80d4acb1853 Address:945147f5-3c72-32d9-d0cd-e80d4acb1853} {Suffrage:Voter ID:server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856 Address:23c6ac6e-ce18-61c3-f0e8-d3aa276b7856} {Suffrage:Voter ID:server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092 Address:6e8d7517-9c7d-efb4-c2b0-9b6a05317092}]"
2025-07-05T01:10:57.075Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:57.075Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: entering follower state: follower="Node at 6e8d7517-9c7d-efb4-c2b0-9b6a05317092 [Follower]" leader-address= leader-id=
2025-07-05T01:10:57.623Z [WARN]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:10:57.623Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: entering candidate state: node="Node at 23c6ac6e-ce18-61c3-f0e8-d3aa276b7856 [Candidate]" term=2
2025-07-05T01:10:57.624Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: election won: term=2 tally=2
2025-07-05T01:10:57.624Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: entering leader state: leader="Node at 23c6ac6e-ce18-61c3-f0e8-d3aa276b7856 [Leader]"
2025-07-05T01:10:57.624Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: added peer, starting replication: peer=server-945147f5-3c72-32d9-d0cd-e80d4acb1853
2025-07-05T01:10:57.624Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: added peer, starting replication: peer=server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092
2025-07-05T01:10:57.624Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: pipelining replication: peer="{Voter server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092 6e8d7517-9c7d-efb4-c2b0-9b6a05317092}"
2025-07-05T01:10:57.624Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: pipelining replication: peer="{Voter server-945147f5-3c72-32d9-d0cd-e80d4acb1853 945147f5-3c72-32d9-d0cd-e80d4acb1853}"
2025-07-05T01:10:58.629Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:10:57.624294385 +0000 UTC m=+32.873031603 (1 nodes), 548.990637ms from start of poll, 552.930374ms from cluster start. Timeout at 2025-07-05 01:10:58.629636932 +0000 UTC m=+33.878374150, 1.005342547s after stability
2025-07-05T01:10:58.630Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: starting snapshot up to: index=12
2025-07-05T01:10:58.630Z [INFO]  creating new snapshot: path=/tmp/raft3612431454/snapshots/2-12-1751677858630.tmp
2025-07-05T01:10:58.636Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: snapshot complete up to: index=12
2025-07-05T01:10:58.637Z [INFO]  creating new snapshot: path=/tmp/raft3612431454/snapshots/2-23-1751677858637.tmp
2025-07-05T01:10:58.637Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: copied to local snapshot: bytes=71
2025-07-05T01:10:58.638Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: snapshot restore progress: id=2-23-1751677858637 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:58.638Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: removing all old logs from log store
2025-07-05T01:10:58.638Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: compacting logs: from=1 to=22
2025-07-05T01:10:58.638Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: restored user snapshot: index=23
2025-07-05T01:10:58.638Z [ERROR] server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: failed to get log: index=22 error="log not found"
2025-07-05T01:10:58.638Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: aborting pipeline replication: peer="{Voter server-945147f5-3c72-32d9-d0cd-e80d4acb1853 945147f5-3c72-32d9-d0cd-e80d4acb1853}"
2025-07-05T01:10:58.638Z [ERROR] server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: failed to get log: index=16 error="log not found"
2025-07-05T01:10:58.638Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: aborting pipeline replication: peer="{Voter server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092 6e8d7517-9c7d-efb4-c2b0-9b6a05317092}"
2025-07-05T01:10:58.644Z [ERROR] server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: failed to get log: index=16 error="log not found"
2025-07-05T01:10:58.644Z [INFO]  creating new snapshot: path=/tmp/raft1252275933/snapshots/2-23-1751677858644.tmp
2025-07-05T01:10:58.644Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:58.644Z [ERROR] server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: failed to get log: index=22 error="log not found"
2025-07-05T01:10:58.645Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: copied to local snapshot: bytes=71
2025-07-05T01:10:58.645Z [INFO]  creating new snapshot: path=/tmp/raft2652409735/snapshots/2-23-1751677858645.tmp
2025-07-05T01:10:58.645Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: snapshot restore progress: id=2-23-1751677858644 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:58.645Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: removing all old logs from log store
2025-07-05T01:10:58.645Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: compacting logs: from=1 to=16
2025-07-05T01:10:58.645Z [INFO]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: Installed remote snapshot
2025-07-05T01:10:58.645Z [WARN]  server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092: skipping application of old log: index=22
2025-07-05T01:10:58.645Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: pipelining replication: peer="{Voter server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092 6e8d7517-9c7d-efb4-c2b0-9b6a05317092}"
2025-07-05T01:10:58.645Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:10:58.645Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:58.645Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: copied to local snapshot: bytes=71
2025-07-05T01:10:58.645Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: snapshot restore progress: id=2-23-1751677858645 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:10:58.645Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: removing all old logs from log store
2025-07-05T01:10:58.645Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: compacting logs: from=1 to=22
2025-07-05T01:10:58.645Z [INFO]  server-945147f5-3c72-32d9-d0cd-e80d4acb1853: Installed remote snapshot
2025-07-05T01:10:58.645Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: pipelining replication: peer="{Voter server-945147f5-3c72-32d9-d0cd-e80d4acb1853 945147f5-3c72-32d9-d0cd-e80d4acb1853}"
2025-07-05T01:10:59.650Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:10:58.645308737 +0000 UTC m=+33.894045945 (2 nodes), 641ns from start of poll, 1.573944716s from cluster start. Timeout at 2025-07-05 01:10:59.650342214 +0000 UTC m=+34.899079422, 1.005033477s after stability
2025-07-05T01:10:59.661Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: aborting pipeline replication: peer="{Voter server-6e8d7517-9c7d-efb4-c2b0-9b6a05317092 6e8d7517-9c7d-efb4-c2b0-9b6a05317092}"
2025-07-05T01:10:59.661Z [INFO]  server-23c6ac6e-ce18-61c3-f0e8-d3aa276b7856: aborting pipeline replication: peer="{Voter server-945147f5-3c72-32d9-d0cd-e80d4acb1853 945147f5-3c72-32d9-d0cd-e80d4acb1853}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-07-05T01:10:59.662Z [INFO]  server-4996c51a-5512-2212-d63d-6a566bed1b65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4996c51a-5512-2212-d63d-6a566bed1b65 Address:4996c51a-5512-2212-d63d-6a566bed1b65} {Suffrage:Voter ID:server-283b049b-596c-ca49-fa0a-be0c0bb2ce17 Address:283b049b-596c-ca49-fa0a-be0c0bb2ce17} {Suffrage:Voter ID:server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913 Address:e20c03f0-73ba-661b-2ee8-b0df6ce7e913}]"
2025-07-05T01:10:59.663Z [INFO]  server-283b049b-596c-ca49-fa0a-be0c0bb2ce17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4996c51a-5512-2212-d63d-6a566bed1b65 Address:4996c51a-5512-2212-d63d-6a566bed1b65} {Suffrage:Voter ID:server-283b049b-596c-ca49-fa0a-be0c0bb2ce17 Address:283b049b-596c-ca49-fa0a-be0c0bb2ce17} {Suffrage:Voter ID:server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913 Address:e20c03f0-73ba-661b-2ee8-b0df6ce7e913}]"
2025-07-05T01:10:59.663Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4996c51a-5512-2212-d63d-6a566bed1b65 Address:4996c51a-5512-2212-d63d-6a566bed1b65} {Suffrage:Voter ID:server-283b049b-596c-ca49-fa0a-be0c0bb2ce17 Address:283b049b-596c-ca49-fa0a-be0c0bb2ce17} {Suffrage:Voter ID:server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913 Address:e20c03f0-73ba-661b-2ee8-b0df6ce7e913}]"
2025-07-05T01:10:59.663Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:10:59.663Z [INFO]  server-4996c51a-5512-2212-d63d-6a566bed1b65: entering follower state: follower="Node at 4996c51a-5512-2212-d63d-6a566bed1b65 [Follower]" leader-address= leader-id=
2025-07-05T01:10:59.663Z [INFO]  server-283b049b-596c-ca49-fa0a-be0c0bb2ce17: entering follower state: follower="Node at 283b049b-596c-ca49-fa0a-be0c0bb2ce17 [Follower]" leader-address= leader-id=
2025-07-05T01:10:59.663Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: entering follower state: follower="Node at e20c03f0-73ba-661b-2ee8-b0df6ce7e913 [Follower]" leader-address= leader-id=
2025-07-05T01:11:00.364Z [WARN]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:00.364Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: entering candidate state: node="Node at e20c03f0-73ba-661b-2ee8-b0df6ce7e913 [Candidate]" term=2
2025-07-05T01:11:00.364Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: election won: term=2 tally=2
2025-07-05T01:11:00.364Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: entering leader state: leader="Node at e20c03f0-73ba-661b-2ee8-b0df6ce7e913 [Leader]"
2025-07-05T01:11:00.364Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: added peer, starting replication: peer=server-4996c51a-5512-2212-d63d-6a566bed1b65
2025-07-05T01:11:00.364Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: added peer, starting replication: peer=server-283b049b-596c-ca49-fa0a-be0c0bb2ce17
2025-07-05T01:11:00.364Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: pipelining replication: peer="{Voter server-283b049b-596c-ca49-fa0a-be0c0bb2ce17 283b049b-596c-ca49-fa0a-be0c0bb2ce17}"
2025-07-05T01:11:00.364Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: pipelining replication: peer="{Voter server-4996c51a-5512-2212-d63d-6a566bed1b65 4996c51a-5512-2212-d63d-6a566bed1b65}"
2025-07-05T01:11:01.371Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:00.366021102 +0000 UTC m=+35.614758310 (1 nodes), 702.92275ms from start of poll, 703.183809ms from cluster start. Timeout at 2025-07-05 01:11:01.371153805 +0000 UTC m=+36.619891013, 1.005132703s after stability
2025-07-05T01:11:01.371Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: starting snapshot up to: index=12
2025-07-05T01:11:01.371Z [INFO]  creating new snapshot: path=/tmp/raft4100388863/snapshots/2-12-1751677861371.tmp
2025-07-05T01:11:01.372Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: snapshot complete up to: index=12
2025-07-05T01:11:01.372Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b707cbe1-c124-c19f-e824-e9b104b84555 Address:b707cbe1-c124-c19f-e824-e9b104b84555} {Suffrage:Voter ID:server-3e3cd9eb-af83-4b41-4ea4-536240751d0e Address:3e3cd9eb-af83-4b41-4ea4-536240751d0e} {Suffrage:Voter ID:server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876 Address:0eb5a66e-a4ab-1fc7-62a5-532fbe38e876}]"
2025-07-05T01:11:01.372Z [INFO]  server-3e3cd9eb-af83-4b41-4ea4-536240751d0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b707cbe1-c124-c19f-e824-e9b104b84555 Address:b707cbe1-c124-c19f-e824-e9b104b84555} {Suffrage:Voter ID:server-3e3cd9eb-af83-4b41-4ea4-536240751d0e Address:3e3cd9eb-af83-4b41-4ea4-536240751d0e} {Suffrage:Voter ID:server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876 Address:0eb5a66e-a4ab-1fc7-62a5-532fbe38e876}]"
2025-07-05T01:11:01.372Z [INFO]  server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b707cbe1-c124-c19f-e824-e9b104b84555 Address:b707cbe1-c124-c19f-e824-e9b104b84555} {Suffrage:Voter ID:server-3e3cd9eb-af83-4b41-4ea4-536240751d0e Address:3e3cd9eb-af83-4b41-4ea4-536240751d0e} {Suffrage:Voter ID:server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876 Address:0eb5a66e-a4ab-1fc7-62a5-532fbe38e876}]"
2025-07-05T01:11:01.372Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:01.373Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: entering follower state: follower="Node at b707cbe1-c124-c19f-e824-e9b104b84555 [Follower]" leader-address= leader-id=
2025-07-05T01:11:01.373Z [INFO]  server-3e3cd9eb-af83-4b41-4ea4-536240751d0e: entering follower state: follower="Node at 3e3cd9eb-af83-4b41-4ea4-536240751d0e [Follower]" leader-address= leader-id=
2025-07-05T01:11:01.373Z [INFO]  server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876: entering follower state: follower="Node at 0eb5a66e-a4ab-1fc7-62a5-532fbe38e876 [Follower]" leader-address= leader-id=
2025-07-05T01:11:02.227Z [WARN]  server-b707cbe1-c124-c19f-e824-e9b104b84555: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:02.227Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: entering candidate state: node="Node at b707cbe1-c124-c19f-e824-e9b104b84555 [Candidate]" term=2
2025-07-05T01:11:02.227Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: election won: term=2 tally=2
2025-07-05T01:11:02.227Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: entering leader state: leader="Node at b707cbe1-c124-c19f-e824-e9b104b84555 [Leader]"
2025-07-05T01:11:02.227Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: added peer, starting replication: peer=server-3e3cd9eb-af83-4b41-4ea4-536240751d0e
2025-07-05T01:11:02.227Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: added peer, starting replication: peer=server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876
2025-07-05T01:11:02.227Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: pipelining replication: peer="{Voter server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876 0eb5a66e-a4ab-1fc7-62a5-532fbe38e876}"
2025-07-05T01:11:02.227Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: pipelining replication: peer="{Voter server-3e3cd9eb-af83-4b41-4ea4-536240751d0e 3e3cd9eb-af83-4b41-4ea4-536240751d0e}"
2025-07-05T01:11:03.232Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:02.227275443 +0000 UTC m=+37.476012651 (1 nodes), 854.27243ms from start of poll, 854.481152ms from cluster start. Timeout at 2025-07-05 01:11:03.232473169 +0000 UTC m=+38.481210387, 1.005197736s after stability
2025-07-05T01:11:03.232Z [INFO]  creating new snapshot: path=/tmp/raft2884844946/snapshots/2-14-1751677863232.tmp
2025-07-05T01:11:03.233Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: copied to local snapshot: bytes=71
2025-07-05T01:11:03.233Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: snapshot restore progress: id=2-14-1751677863232 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:03.233Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: restored user snapshot: index=14
2025-07-05T01:11:03.233Z [ERROR] server-b707cbe1-c124-c19f-e824-e9b104b84555: failed to get log: index=3 error="log not found"
2025-07-05T01:11:03.233Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: aborting pipeline replication: peer="{Voter server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876 0eb5a66e-a4ab-1fc7-62a5-532fbe38e876}"
2025-07-05T01:11:03.233Z [ERROR] server-b707cbe1-c124-c19f-e824-e9b104b84555: failed to get log: index=3 error="log not found"
2025-07-05T01:11:03.233Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: aborting pipeline replication: peer="{Voter server-3e3cd9eb-af83-4b41-4ea4-536240751d0e 3e3cd9eb-af83-4b41-4ea4-536240751d0e}"
2025-07-05T01:11:03.238Z [ERROR] server-b707cbe1-c124-c19f-e824-e9b104b84555: failed to get log: index=3 error="log not found"
2025-07-05T01:11:03.239Z [INFO]  creating new snapshot: path=/tmp/raft2009290974/snapshots/2-14-1751677863239.tmp
2025-07-05T01:11:03.239Z [INFO]  server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:03.239Z [INFO]  server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876: copied to local snapshot: bytes=71
2025-07-05T01:11:03.239Z [INFO]  server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876: snapshot restore progress: id=2-14-1751677863239 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:03.239Z [INFO]  server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876: Installed remote snapshot
2025-07-05T01:11:03.240Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: pipelining replication: peer="{Voter server-0eb5a66e-a4ab-1fc7-62a5-532fbe38e876 0eb5a66e-a4ab-1fc7-62a5-532fbe38e876}"
2025-07-05T01:11:03.240Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:03.241Z [ERROR] server-b707cbe1-c124-c19f-e824-e9b104b84555: failed to get log: index=3 error="log not found"
2025-07-05T01:11:03.241Z [INFO]  creating new snapshot: path=/tmp/raft3656893544/snapshots/2-14-1751677863241.tmp
2025-07-05T01:11:03.241Z [INFO]  server-3e3cd9eb-af83-4b41-4ea4-536240751d0e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:03.241Z [INFO]  server-3e3cd9eb-af83-4b41-4ea4-536240751d0e: copied to local snapshot: bytes=71
2025-07-05T01:11:03.242Z [INFO]  server-3e3cd9eb-af83-4b41-4ea4-536240751d0e: snapshot restore progress: id=2-14-1751677863241 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:03.242Z [INFO]  server-3e3cd9eb-af83-4b41-4ea4-536240751d0e: Installed remote snapshot
2025-07-05T01:11:03.242Z [INFO]  server-b707cbe1-c124-c19f-e824-e9b104b84555: pipelining replication: peer="{Voter server-3e3cd9eb-af83-4b41-4ea4-536240751d0e 3e3cd9eb-af83-4b41-4ea4-536240751d0e}"
2025-07-05T01:11:04.245Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:03.240165762 +0000 UTC m=+38.488902980 (2 nodes), 601ns from start of poll, 1.867371481s from cluster start. Timeout at 2025-07-05 01:11:04.245233323 +0000 UTC m=+39.493970542, 1.005067562s after stability
2025-07-05T01:11:04.256Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: aborting pipeline replication: peer="{Voter server-283b049b-596c-ca49-fa0a-be0c0bb2ce17 283b049b-596c-ca49-fa0a-be0c0bb2ce17}"
2025-07-05T01:11:04.256Z [INFO]  server-e20c03f0-73ba-661b-2ee8-b0df6ce7e913: aborting pipeline replication: peer="{Voter server-4996c51a-5512-2212-d63d-6a566bed1b65 4996c51a-5512-2212-d63d-6a566bed1b65}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-07-05T01:11:04.258Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bfd954f-df0a-ecac-c685-787ee3ca4d36 Address:6bfd954f-df0a-ecac-c685-787ee3ca4d36} {Suffrage:Voter ID:server-66900aed-785e-de67-1b0a-3ac2d2de7aac Address:66900aed-785e-de67-1b0a-3ac2d2de7aac} {Suffrage:Voter ID:server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6 Address:67c5578f-9ebd-6c07-f4f6-d1e86aff46d6}]"
2025-07-05T01:11:04.258Z [INFO]  server-66900aed-785e-de67-1b0a-3ac2d2de7aac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bfd954f-df0a-ecac-c685-787ee3ca4d36 Address:6bfd954f-df0a-ecac-c685-787ee3ca4d36} {Suffrage:Voter ID:server-66900aed-785e-de67-1b0a-3ac2d2de7aac Address:66900aed-785e-de67-1b0a-3ac2d2de7aac} {Suffrage:Voter ID:server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6 Address:67c5578f-9ebd-6c07-f4f6-d1e86aff46d6}]"
2025-07-05T01:11:04.258Z [INFO]  server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bfd954f-df0a-ecac-c685-787ee3ca4d36 Address:6bfd954f-df0a-ecac-c685-787ee3ca4d36} {Suffrage:Voter ID:server-66900aed-785e-de67-1b0a-3ac2d2de7aac Address:66900aed-785e-de67-1b0a-3ac2d2de7aac} {Suffrage:Voter ID:server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6 Address:67c5578f-9ebd-6c07-f4f6-d1e86aff46d6}]"
2025-07-05T01:11:04.258Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:04.258Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: entering follower state: follower="Node at 6bfd954f-df0a-ecac-c685-787ee3ca4d36 [Follower]" leader-address= leader-id=
2025-07-05T01:11:04.258Z [INFO]  server-66900aed-785e-de67-1b0a-3ac2d2de7aac: entering follower state: follower="Node at 66900aed-785e-de67-1b0a-3ac2d2de7aac [Follower]" leader-address= leader-id=
2025-07-05T01:11:04.258Z [INFO]  server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6: entering follower state: follower="Node at 67c5578f-9ebd-6c07-f4f6-d1e86aff46d6 [Follower]" leader-address= leader-id=
2025-07-05T01:11:04.999Z [WARN]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:04.999Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: entering candidate state: node="Node at 6bfd954f-df0a-ecac-c685-787ee3ca4d36 [Candidate]" term=2
2025-07-05T01:11:04.999Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: election won: term=2 tally=2
2025-07-05T01:11:04.999Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: entering leader state: leader="Node at 6bfd954f-df0a-ecac-c685-787ee3ca4d36 [Leader]"
2025-07-05T01:11:04.999Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: added peer, starting replication: peer=server-66900aed-785e-de67-1b0a-3ac2d2de7aac
2025-07-05T01:11:04.999Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: added peer, starting replication: peer=server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6
2025-07-05T01:11:04.999Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: pipelining replication: peer="{Voter server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6 67c5578f-9ebd-6c07-f4f6-d1e86aff46d6}"
2025-07-05T01:11:04.999Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: pipelining replication: peer="{Voter server-66900aed-785e-de67-1b0a-3ac2d2de7aac 66900aed-785e-de67-1b0a-3ac2d2de7aac}"
2025-07-05T01:11:06.005Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:04.999373526 +0000 UTC m=+40.248110744 (1 nodes), 741.17374ms from start of poll, 741.408661ms from cluster start. Timeout at 2025-07-05 01:11:06.005213275 +0000 UTC m=+41.253950493, 1.005839749s after stability
2025-07-05T01:11:06.005Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: starting snapshot up to: index=12
2025-07-05T01:11:06.005Z [INFO]  creating new snapshot: path=/tmp/raft3448196433/snapshots/2-12-1751677866005.tmp
2025-07-05T01:11:06.065Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: snapshot complete up to: index=12
2025-07-05T01:11:06.067Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37 Address:c8bb149a-4d43-7fa2-0d4c-37c21fee3e37} {Suffrage:Voter ID:server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519 Address:4a4d631e-e3cc-ecfc-a398-d6fc2af48519} {Suffrage:Voter ID:server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660 Address:f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660}]"
2025-07-05T01:11:06.068Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37 Address:c8bb149a-4d43-7fa2-0d4c-37c21fee3e37} {Suffrage:Voter ID:server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519 Address:4a4d631e-e3cc-ecfc-a398-d6fc2af48519} {Suffrage:Voter ID:server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660 Address:f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660}]"
2025-07-05T01:11:06.068Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: entering follower state: follower="Node at c8bb149a-4d43-7fa2-0d4c-37c21fee3e37 [Follower]" leader-address= leader-id=
2025-07-05T01:11:06.068Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: entering follower state: follower="Node at 4a4d631e-e3cc-ecfc-a398-d6fc2af48519 [Follower]" leader-address= leader-id=
2025-07-05T01:11:06.068Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37 Address:c8bb149a-4d43-7fa2-0d4c-37c21fee3e37} {Suffrage:Voter ID:server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519 Address:4a4d631e-e3cc-ecfc-a398-d6fc2af48519} {Suffrage:Voter ID:server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660 Address:f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660}]"
2025-07-05T01:11:06.068Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:06.068Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: entering follower state: follower="Node at f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660 [Follower]" leader-address= leader-id=
2025-07-05T01:11:06.803Z [WARN]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:06.803Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: entering candidate state: node="Node at f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660 [Candidate]" term=2
2025-07-05T01:11:06.803Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: election won: term=2 tally=2
2025-07-05T01:11:06.803Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: entering leader state: leader="Node at f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660 [Leader]"
2025-07-05T01:11:06.803Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: added peer, starting replication: peer=server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37
2025-07-05T01:11:06.803Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: added peer, starting replication: peer=server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519
2025-07-05T01:11:06.803Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: pipelining replication: peer="{Voter server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519 4a4d631e-e3cc-ecfc-a398-d6fc2af48519}"
2025-07-05T01:11:06.803Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: pipelining replication: peer="{Voter server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37 c8bb149a-4d43-7fa2-0d4c-37c21fee3e37}"
2025-07-05T01:11:07.809Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:06.803475336 +0000 UTC m=+42.052212544 (1 nodes), 735.3602ms from start of poll, 735.729131ms from cluster start. Timeout at 2025-07-05 01:11:07.80948814 +0000 UTC m=+43.058225358, 1.006012814s after stability
2025-07-05T01:11:07.809Z [INFO]  creating new snapshot: path=/tmp/raft3306697774/snapshots/2-14-1751677867809.tmp
2025-07-05T01:11:07.811Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: copied to local snapshot: bytes=71
2025-07-05T01:11:07.811Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: snapshot restore progress: id=2-14-1751677867809 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:07.811Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: removing all old logs from log store
2025-07-05T01:11:07.811Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: compacting logs: from=1 to=2
2025-07-05T01:11:07.811Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: restored user snapshot: index=14
2025-07-05T01:11:07.811Z [ERROR] server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: failed to get log: index=2 error="log not found"
2025-07-05T01:11:07.811Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: aborting pipeline replication: peer="{Voter server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519 4a4d631e-e3cc-ecfc-a398-d6fc2af48519}"
2025-07-05T01:11:07.811Z [ERROR] server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: failed to get log: index=2 error="log not found"
2025-07-05T01:11:07.811Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: aborting pipeline replication: peer="{Voter server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37 c8bb149a-4d43-7fa2-0d4c-37c21fee3e37}"
2025-07-05T01:11:07.818Z [ERROR] server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: failed to get log: index=2 error="log not found"
2025-07-05T01:11:07.819Z [INFO]  creating new snapshot: path=/tmp/raft240399447/snapshots/2-14-1751677867819.tmp
2025-07-05T01:11:07.819Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:07.820Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: copied to local snapshot: bytes=71
2025-07-05T01:11:07.820Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: snapshot restore progress: id=2-14-1751677867819 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:07.820Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: removing all old logs from log store
2025-07-05T01:11:07.820Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: compacting logs: from=1 to=2
2025-07-05T01:11:07.820Z [INFO]  server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37: Installed remote snapshot
2025-07-05T01:11:07.820Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: pipelining replication: peer="{Voter server-c8bb149a-4d43-7fa2-0d4c-37c21fee3e37 c8bb149a-4d43-7fa2-0d4c-37c21fee3e37}"
2025-07-05T01:11:07.820Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:07.820Z [ERROR] server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: failed to get log: index=2 error="log not found"
2025-07-05T01:11:07.820Z [INFO]  creating new snapshot: path=/tmp/raft3660586445/snapshots/2-14-1751677867820.tmp
2025-07-05T01:11:07.822Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:07.822Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: copied to local snapshot: bytes=71
2025-07-05T01:11:07.822Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: snapshot restore progress: id=2-14-1751677867820 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:07.822Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: removing all old logs from log store
2025-07-05T01:11:07.822Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: compacting logs: from=1 to=2
2025-07-05T01:11:07.822Z [INFO]  server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519: Installed remote snapshot
2025-07-05T01:11:07.822Z [INFO]  server-f6d49d47-fa8d-c3a6-c4f3-2fc652f8c660: pipelining replication: peer="{Voter server-4a4d631e-e3cc-ecfc-a398-d6fc2af48519 4a4d631e-e3cc-ecfc-a398-d6fc2af48519}"
2025-07-05T01:11:08.826Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:07.820463689 +0000 UTC m=+43.069200897 (2 nodes), 431ns from start of poll, 1.752717484s from cluster start. Timeout at 2025-07-05 01:11:08.826028823 +0000 UTC m=+44.074766062, 1.005565165s after stability
2025-07-05T01:11:08.840Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: aborting pipeline replication: peer="{Voter server-67c5578f-9ebd-6c07-f4f6-d1e86aff46d6 67c5578f-9ebd-6c07-f4f6-d1e86aff46d6}"
2025-07-05T01:11:08.840Z [INFO]  server-6bfd954f-df0a-ecac-c685-787ee3ca4d36: aborting pipeline replication: peer="{Voter server-66900aed-785e-de67-1b0a-3ac2d2de7aac 66900aed-785e-de67-1b0a-3ac2d2de7aac}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-07-05T01:11:08.842Z [INFO]  server-c67f7484-eda2-270b-e490-db21e601bc8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c67f7484-eda2-270b-e490-db21e601bc8c Address:c67f7484-eda2-270b-e490-db21e601bc8c} {Suffrage:Voter ID:server-322e7683-39f3-ae9d-08aa-67452cdb1fc2 Address:322e7683-39f3-ae9d-08aa-67452cdb1fc2} {Suffrage:Voter ID:server-d39790c4-6149-48e2-31ec-80389746810a Address:d39790c4-6149-48e2-31ec-80389746810a}]"
2025-07-05T01:11:08.842Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c67f7484-eda2-270b-e490-db21e601bc8c Address:c67f7484-eda2-270b-e490-db21e601bc8c} {Suffrage:Voter ID:server-322e7683-39f3-ae9d-08aa-67452cdb1fc2 Address:322e7683-39f3-ae9d-08aa-67452cdb1fc2} {Suffrage:Voter ID:server-d39790c4-6149-48e2-31ec-80389746810a Address:d39790c4-6149-48e2-31ec-80389746810a}]"
2025-07-05T01:11:08.842Z [INFO]  server-d39790c4-6149-48e2-31ec-80389746810a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c67f7484-eda2-270b-e490-db21e601bc8c Address:c67f7484-eda2-270b-e490-db21e601bc8c} {Suffrage:Voter ID:server-322e7683-39f3-ae9d-08aa-67452cdb1fc2 Address:322e7683-39f3-ae9d-08aa-67452cdb1fc2} {Suffrage:Voter ID:server-d39790c4-6149-48e2-31ec-80389746810a Address:d39790c4-6149-48e2-31ec-80389746810a}]"
2025-07-05T01:11:08.842Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:08.842Z [INFO]  server-c67f7484-eda2-270b-e490-db21e601bc8c: entering follower state: follower="Node at c67f7484-eda2-270b-e490-db21e601bc8c [Follower]" leader-address= leader-id=
2025-07-05T01:11:08.842Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: entering follower state: follower="Node at 322e7683-39f3-ae9d-08aa-67452cdb1fc2 [Follower]" leader-address= leader-id=
2025-07-05T01:11:08.842Z [INFO]  server-d39790c4-6149-48e2-31ec-80389746810a: entering follower state: follower="Node at d39790c4-6149-48e2-31ec-80389746810a [Follower]" leader-address= leader-id=
2025-07-05T01:11:09.534Z [WARN]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:09.534Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: entering candidate state: node="Node at 322e7683-39f3-ae9d-08aa-67452cdb1fc2 [Candidate]" term=2
2025-07-05T01:11:09.534Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: election won: term=2 tally=2
2025-07-05T01:11:09.534Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: entering leader state: leader="Node at 322e7683-39f3-ae9d-08aa-67452cdb1fc2 [Leader]"
2025-07-05T01:11:09.534Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: added peer, starting replication: peer=server-c67f7484-eda2-270b-e490-db21e601bc8c
2025-07-05T01:11:09.534Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: added peer, starting replication: peer=server-d39790c4-6149-48e2-31ec-80389746810a
2025-07-05T01:11:09.534Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: pipelining replication: peer="{Voter server-d39790c4-6149-48e2-31ec-80389746810a d39790c4-6149-48e2-31ec-80389746810a}"
2025-07-05T01:11:09.534Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: pipelining replication: peer="{Voter server-c67f7484-eda2-270b-e490-db21e601bc8c c67f7484-eda2-270b-e490-db21e601bc8c}"
2025-07-05T01:11:10.541Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:09.534993993 +0000 UTC m=+44.783731201 (1 nodes), 692.686657ms from start of poll, 692.978184ms from cluster start. Timeout at 2025-07-05 01:11:10.541144765 +0000 UTC m=+45.789881993, 1.006150792s after stability
2025-07-05T01:11:10.541Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: starting snapshot up to: index=12
2025-07-05T01:11:10.541Z [INFO]  creating new snapshot: path=/tmp/raft565500941/snapshots/2-12-1751677870541.tmp
2025-07-05T01:11:10.659Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: snapshot complete up to: index=12
2025-07-05T01:11:10.660Z [INFO]  creating new snapshot: path=/tmp/raft565500941/snapshots/2-23-1751677870660.tmp
2025-07-05T01:11:10.846Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: copied to local snapshot: bytes=71
2025-07-05T01:11:10.846Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: snapshot restore progress: id=2-23-1751677870660 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:10.846Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: restored user snapshot: index=23
2025-07-05T01:11:10.846Z [ERROR] server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: failed to get log: index=23 error="log not found"
2025-07-05T01:11:10.846Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: aborting pipeline replication: peer="{Voter server-d39790c4-6149-48e2-31ec-80389746810a d39790c4-6149-48e2-31ec-80389746810a}"
2025-07-05T01:11:10.846Z [ERROR] server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: failed to get log: index=23 error="log not found"
2025-07-05T01:11:10.846Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: aborting pipeline replication: peer="{Voter server-c67f7484-eda2-270b-e490-db21e601bc8c c67f7484-eda2-270b-e490-db21e601bc8c}"
2025-07-05T01:11:10.853Z [ERROR] server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: failed to get log: index=23 error="log not found"
2025-07-05T01:11:10.853Z [INFO]  creating new snapshot: path=/tmp/raft2521410437/snapshots/2-23-1751677870853.tmp
2025-07-05T01:11:10.854Z [INFO]  server-c67f7484-eda2-270b-e490-db21e601bc8c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:10.854Z [ERROR] server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: failed to get log: index=23 error="log not found"
2025-07-05T01:11:10.854Z [INFO]  creating new snapshot: path=/tmp/raft3018141077/snapshots/2-23-1751677870854.tmp
2025-07-05T01:11:10.855Z [INFO]  server-d39790c4-6149-48e2-31ec-80389746810a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:10.989Z [INFO]  server-d39790c4-6149-48e2-31ec-80389746810a: copied to local snapshot: bytes=71
2025-07-05T01:11:10.989Z [INFO]  server-d39790c4-6149-48e2-31ec-80389746810a: snapshot restore progress: id=2-23-1751677870854 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:10.989Z [INFO]  server-d39790c4-6149-48e2-31ec-80389746810a: Installed remote snapshot
2025-07-05T01:11:10.989Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: pipelining replication: peer="{Voter server-d39790c4-6149-48e2-31ec-80389746810a d39790c4-6149-48e2-31ec-80389746810a}"
2025-07-05T01:11:10.990Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:10.991Z [INFO]  server-c67f7484-eda2-270b-e490-db21e601bc8c: copied to local snapshot: bytes=71
2025-07-05T01:11:10.991Z [INFO]  server-c67f7484-eda2-270b-e490-db21e601bc8c: snapshot restore progress: id=2-23-1751677870853 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:10.991Z [INFO]  server-c67f7484-eda2-270b-e490-db21e601bc8c: Installed remote snapshot
2025-07-05T01:11:10.991Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: pipelining replication: peer="{Voter server-c67f7484-eda2-270b-e490-db21e601bc8c c67f7484-eda2-270b-e490-db21e601bc8c}"
2025-07-05T01:11:11.995Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:10.990084657 +0000 UTC m=+46.238821875 (2 nodes), 951ns from start of poll, 2.148068858s from cluster start. Timeout at 2025-07-05 01:11:11.995144514 +0000 UTC m=+47.243881732, 1.005059857s after stability
2025-07-05T01:11:12.003Z [WARN]  server-c67f7484-eda2-270b-e490-db21e601bc8c: skipping application of old log: index=24
2025-07-05T01:11:12.015Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: aborting pipeline replication: peer="{Voter server-d39790c4-6149-48e2-31ec-80389746810a d39790c4-6149-48e2-31ec-80389746810a}"
2025-07-05T01:11:12.015Z [INFO]  server-322e7683-39f3-ae9d-08aa-67452cdb1fc2: aborting pipeline replication: peer="{Voter server-c67f7484-eda2-270b-e490-db21e601bc8c c67f7484-eda2-270b-e490-db21e601bc8c}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-07-05T01:11:12.017Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d51a3d6-9f71-2774-f2de-364fc4637f17 Address:6d51a3d6-9f71-2774-f2de-364fc4637f17} {Suffrage:Voter ID:server-f4196ba9-516f-5274-b3dc-71f6a866938d Address:f4196ba9-516f-5274-b3dc-71f6a866938d} {Suffrage:Voter ID:server-ed47487f-b2bc-57bb-513a-bb642f95346d Address:ed47487f-b2bc-57bb-513a-bb642f95346d}]"
2025-07-05T01:11:12.017Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d51a3d6-9f71-2774-f2de-364fc4637f17 Address:6d51a3d6-9f71-2774-f2de-364fc4637f17} {Suffrage:Voter ID:server-f4196ba9-516f-5274-b3dc-71f6a866938d Address:f4196ba9-516f-5274-b3dc-71f6a866938d} {Suffrage:Voter ID:server-ed47487f-b2bc-57bb-513a-bb642f95346d Address:ed47487f-b2bc-57bb-513a-bb642f95346d}]"
2025-07-05T01:11:12.017Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d51a3d6-9f71-2774-f2de-364fc4637f17 Address:6d51a3d6-9f71-2774-f2de-364fc4637f17} {Suffrage:Voter ID:server-f4196ba9-516f-5274-b3dc-71f6a866938d Address:f4196ba9-516f-5274-b3dc-71f6a866938d} {Suffrage:Voter ID:server-ed47487f-b2bc-57bb-513a-bb642f95346d Address:ed47487f-b2bc-57bb-513a-bb642f95346d}]"
2025-07-05T01:11:12.017Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:12.017Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: entering follower state: follower="Node at 6d51a3d6-9f71-2774-f2de-364fc4637f17 [Follower]" leader-address= leader-id=
2025-07-05T01:11:12.017Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: entering follower state: follower="Node at f4196ba9-516f-5274-b3dc-71f6a866938d [Follower]" leader-address= leader-id=
2025-07-05T01:11:12.017Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: entering follower state: follower="Node at ed47487f-b2bc-57bb-513a-bb642f95346d [Follower]" leader-address= leader-id=
2025-07-05T01:11:12.745Z [WARN]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:12.745Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: entering candidate state: node="Node at 6d51a3d6-9f71-2774-f2de-364fc4637f17 [Candidate]" term=2
2025-07-05T01:11:12.745Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: election won: term=2 tally=2
2025-07-05T01:11:12.745Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: entering leader state: leader="Node at 6d51a3d6-9f71-2774-f2de-364fc4637f17 [Leader]"
2025-07-05T01:11:12.745Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: added peer, starting replication: peer=server-f4196ba9-516f-5274-b3dc-71f6a866938d
2025-07-05T01:11:12.745Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: added peer, starting replication: peer=server-ed47487f-b2bc-57bb-513a-bb642f95346d
2025-07-05T01:11:12.745Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: pipelining replication: peer="{Voter server-ed47487f-b2bc-57bb-513a-bb642f95346d ed47487f-b2bc-57bb-513a-bb642f95346d}"
2025-07-05T01:11:12.745Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: pipelining replication: peer="{Voter server-f4196ba9-516f-5274-b3dc-71f6a866938d f4196ba9-516f-5274-b3dc-71f6a866938d}"
2025-07-05T01:11:13.751Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:12.745658918 +0000 UTC m=+47.994396147 (1 nodes), 728.29348ms from start of poll, 728.577543ms from cluster start. Timeout at 2025-07-05 01:11:13.75166011 +0000 UTC m=+49.000397328, 1.006001181s after stability
2025-07-05T01:11:13.752Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: starting snapshot up to: index=12
2025-07-05T01:11:13.752Z [INFO]  creating new snapshot: path=/tmp/raft1982076611/snapshots/2-12-1751677873752.tmp
2025-07-05T01:11:13.867Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: snapshot complete up to: index=12
2025-07-05T01:11:13.871Z [INFO]  creating new snapshot: path=/tmp/raft1982076611/snapshots/2-23-1751677873871.tmp
2025-07-05T01:11:13.871Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: copied to local snapshot: bytes=71
2025-07-05T01:11:13.871Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: snapshot restore progress: id=2-23-1751677873871 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:13.871Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: removing all old logs from log store
2025-07-05T01:11:13.871Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: compacting logs: from=1 to=22
2025-07-05T01:11:13.871Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: restored user snapshot: index=23
2025-07-05T01:11:13.871Z [ERROR] server-6d51a3d6-9f71-2774-f2de-364fc4637f17: failed to get log: index=22 error="log not found"
2025-07-05T01:11:13.871Z [ERROR] server-6d51a3d6-9f71-2774-f2de-364fc4637f17: failed to get log: index=22 error="log not found"
2025-07-05T01:11:13.871Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: aborting pipeline replication: peer="{Voter server-f4196ba9-516f-5274-b3dc-71f6a866938d f4196ba9-516f-5274-b3dc-71f6a866938d}"
2025-07-05T01:11:13.871Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: aborting pipeline replication: peer="{Voter server-ed47487f-b2bc-57bb-513a-bb642f95346d ed47487f-b2bc-57bb-513a-bb642f95346d}"
2025-07-05T01:11:13.881Z [ERROR] server-6d51a3d6-9f71-2774-f2de-364fc4637f17: failed to get log: index=22 error="log not found"
2025-07-05T01:11:13.881Z [INFO]  creating new snapshot: path=/tmp/raft559815276/snapshots/2-23-1751677873881.tmp
2025-07-05T01:11:13.881Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:13.882Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: copied to local snapshot: bytes=71
2025-07-05T01:11:13.882Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: snapshot restore progress: id=2-23-1751677873881 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:13.882Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: removing all old logs from log store
2025-07-05T01:11:13.882Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: compacting logs: from=1 to=22
2025-07-05T01:11:13.882Z [INFO]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: Installed remote snapshot
2025-07-05T01:11:13.882Z [WARN]  server-f4196ba9-516f-5274-b3dc-71f6a866938d: skipping application of old log: index=22
2025-07-05T01:11:13.882Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: pipelining replication: peer="{Voter server-f4196ba9-516f-5274-b3dc-71f6a866938d f4196ba9-516f-5274-b3dc-71f6a866938d}"
2025-07-05T01:11:13.882Z [ERROR] server-6d51a3d6-9f71-2774-f2de-364fc4637f17: failed to get log: index=22 error="log not found"
2025-07-05T01:11:13.882Z [INFO]  creating new snapshot: path=/tmp/raft3825100448/snapshots/2-23-1751677873882.tmp
2025-07-05T01:11:13.882Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:13.882Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: copied to local snapshot: bytes=71
2025-07-05T01:11:13.882Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:13.884Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: snapshot restore progress: id=2-23-1751677873882 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:13.884Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: removing all old logs from log store
2025-07-05T01:11:13.884Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: compacting logs: from=1 to=22
2025-07-05T01:11:13.884Z [INFO]  server-ed47487f-b2bc-57bb-513a-bb642f95346d: Installed remote snapshot
2025-07-05T01:11:13.884Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: pipelining replication: peer="{Voter server-ed47487f-b2bc-57bb-513a-bb642f95346d ed47487f-b2bc-57bb-513a-bb642f95346d}"
2025-07-05T01:11:14.890Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:13.883014289 +0000 UTC m=+49.131751507 (2 nodes), 1.132µs from start of poll, 1.865932903s from cluster start. Timeout at 2025-07-05 01:11:14.890952213 +0000 UTC m=+50.139689421, 1.007937914s after stability
2025-07-05T01:11:14.902Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: aborting pipeline replication: peer="{Voter server-ed47487f-b2bc-57bb-513a-bb642f95346d ed47487f-b2bc-57bb-513a-bb642f95346d}"
2025-07-05T01:11:14.902Z [INFO]  server-6d51a3d6-9f71-2774-f2de-364fc4637f17: aborting pipeline replication: peer="{Voter server-f4196ba9-516f-5274-b3dc-71f6a866938d f4196ba9-516f-5274-b3dc-71f6a866938d}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-07-05T01:11:14.903Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18580028-ff20-542d-e063-9d0f436d75f0 Address:18580028-ff20-542d-e063-9d0f436d75f0} {Suffrage:Voter ID:server-d1780ae2-3d90-a7e5-d7f0-27be2c399203 Address:d1780ae2-3d90-a7e5-d7f0-27be2c399203} {Suffrage:Voter ID:server-d0964983-52dd-f89d-535b-88516b87615c Address:d0964983-52dd-f89d-535b-88516b87615c}]"
2025-07-05T01:11:14.904Z [INFO]  server-d1780ae2-3d90-a7e5-d7f0-27be2c399203: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18580028-ff20-542d-e063-9d0f436d75f0 Address:18580028-ff20-542d-e063-9d0f436d75f0} {Suffrage:Voter ID:server-d1780ae2-3d90-a7e5-d7f0-27be2c399203 Address:d1780ae2-3d90-a7e5-d7f0-27be2c399203} {Suffrage:Voter ID:server-d0964983-52dd-f89d-535b-88516b87615c Address:d0964983-52dd-f89d-535b-88516b87615c}]"
2025-07-05T01:11:14.904Z [INFO]  server-d0964983-52dd-f89d-535b-88516b87615c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18580028-ff20-542d-e063-9d0f436d75f0 Address:18580028-ff20-542d-e063-9d0f436d75f0} {Suffrage:Voter ID:server-d1780ae2-3d90-a7e5-d7f0-27be2c399203 Address:d1780ae2-3d90-a7e5-d7f0-27be2c399203} {Suffrage:Voter ID:server-d0964983-52dd-f89d-535b-88516b87615c Address:d0964983-52dd-f89d-535b-88516b87615c}]"
2025-07-05T01:11:14.904Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:14.904Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: entering follower state: follower="Node at 18580028-ff20-542d-e063-9d0f436d75f0 [Follower]" leader-address= leader-id=
2025-07-05T01:11:14.904Z [INFO]  server-d0964983-52dd-f89d-535b-88516b87615c: entering follower state: follower="Node at d0964983-52dd-f89d-535b-88516b87615c [Follower]" leader-address= leader-id=
2025-07-05T01:11:14.904Z [INFO]  server-d1780ae2-3d90-a7e5-d7f0-27be2c399203: entering follower state: follower="Node at d1780ae2-3d90-a7e5-d7f0-27be2c399203 [Follower]" leader-address= leader-id=
2025-07-05T01:11:15.434Z [WARN]  server-18580028-ff20-542d-e063-9d0f436d75f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:15.434Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: entering candidate state: node="Node at 18580028-ff20-542d-e063-9d0f436d75f0 [Candidate]" term=2
2025-07-05T01:11:15.434Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: election won: term=2 tally=2
2025-07-05T01:11:15.434Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: entering leader state: leader="Node at 18580028-ff20-542d-e063-9d0f436d75f0 [Leader]"
2025-07-05T01:11:15.434Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: added peer, starting replication: peer=server-d1780ae2-3d90-a7e5-d7f0-27be2c399203
2025-07-05T01:11:15.434Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: added peer, starting replication: peer=server-d0964983-52dd-f89d-535b-88516b87615c
2025-07-05T01:11:15.434Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: pipelining replication: peer="{Voter server-d0964983-52dd-f89d-535b-88516b87615c d0964983-52dd-f89d-535b-88516b87615c}"
2025-07-05T01:11:15.434Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: pipelining replication: peer="{Voter server-d1780ae2-3d90-a7e5-d7f0-27be2c399203 d1780ae2-3d90-a7e5-d7f0-27be2c399203}"
2025-07-05T01:11:16.439Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:15.434845222 +0000 UTC m=+50.683582440 (1 nodes), 530.727764ms from start of poll, 531.071239ms from cluster start. Timeout at 2025-07-05 01:11:16.439928643 +0000 UTC m=+51.688665851, 1.005083411s after stability
2025-07-05T01:11:16.440Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: starting snapshot up to: index=12
2025-07-05T01:11:16.440Z [INFO]  creating new snapshot: path=/tmp/raft1995253010/snapshots/2-12-1751677876440.tmp
2025-07-05T01:11:16.441Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: snapshot complete up to: index=12
2025-07-05T01:11:16.441Z [INFO]  server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f Address:7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f} {Suffrage:Voter ID:server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d Address:32a0b42f-5cc8-217d-7df6-4af4af8bd63d} {Suffrage:Voter ID:server-0b58ead8-5413-2152-c635-44a50e9bf1e5 Address:0b58ead8-5413-2152-c635-44a50e9bf1e5}]"
2025-07-05T01:11:16.441Z [INFO]  server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f Address:7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f} {Suffrage:Voter ID:server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d Address:32a0b42f-5cc8-217d-7df6-4af4af8bd63d} {Suffrage:Voter ID:server-0b58ead8-5413-2152-c635-44a50e9bf1e5 Address:0b58ead8-5413-2152-c635-44a50e9bf1e5}]"
2025-07-05T01:11:16.442Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f Address:7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f} {Suffrage:Voter ID:server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d Address:32a0b42f-5cc8-217d-7df6-4af4af8bd63d} {Suffrage:Voter ID:server-0b58ead8-5413-2152-c635-44a50e9bf1e5 Address:0b58ead8-5413-2152-c635-44a50e9bf1e5}]"
2025-07-05T01:11:16.442Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:16.442Z [INFO]  server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f: entering follower state: follower="Node at 7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f [Follower]" leader-address= leader-id=
2025-07-05T01:11:16.442Z [INFO]  server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d: entering follower state: follower="Node at 32a0b42f-5cc8-217d-7df6-4af4af8bd63d [Follower]" leader-address= leader-id=
2025-07-05T01:11:16.442Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: entering follower state: follower="Node at 0b58ead8-5413-2152-c635-44a50e9bf1e5 [Follower]" leader-address= leader-id=
2025-07-05T01:11:16.977Z [WARN]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:16.977Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: entering candidate state: node="Node at 0b58ead8-5413-2152-c635-44a50e9bf1e5 [Candidate]" term=2
2025-07-05T01:11:16.977Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: election won: term=2 tally=2
2025-07-05T01:11:16.977Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: entering leader state: leader="Node at 0b58ead8-5413-2152-c635-44a50e9bf1e5 [Leader]"
2025-07-05T01:11:16.977Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: added peer, starting replication: peer=server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f
2025-07-05T01:11:16.977Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: added peer, starting replication: peer=server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d
2025-07-05T01:11:16.978Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: pipelining replication: peer="{Voter server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d 32a0b42f-5cc8-217d-7df6-4af4af8bd63d}"
2025-07-05T01:11:16.978Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: pipelining replication: peer="{Voter server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f 7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f}"
2025-07-05T01:11:17.983Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:16.978119821 +0000 UTC m=+52.226857029 (1 nodes), 536.02705ms from start of poll, 536.297718ms from cluster start. Timeout at 2025-07-05 01:11:17.983956935 +0000 UTC m=+53.232694153, 1.005837124s after stability
2025-07-05T01:11:17.984Z [INFO]  creating new snapshot: path=/tmp/raft2015309892/snapshots/2-15-1751677877984.tmp
2025-07-05T01:11:17.984Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: copied to local snapshot: bytes=71
2025-07-05T01:11:17.991Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: snapshot restore progress: id=2-15-1751677877984 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:17.991Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: restored user snapshot: index=15
2025-07-05T01:11:17.991Z [ERROR] server-0b58ead8-5413-2152-c635-44a50e9bf1e5: failed to get log: index=3 error="log not found"
2025-07-05T01:11:17.991Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: aborting pipeline replication: peer="{Voter server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d 32a0b42f-5cc8-217d-7df6-4af4af8bd63d}"
2025-07-05T01:11:17.991Z [ERROR] server-0b58ead8-5413-2152-c635-44a50e9bf1e5: failed to get log: index=3 error="log not found"
2025-07-05T01:11:17.991Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: aborting pipeline replication: peer="{Voter server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f 7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f}"
2025-07-05T01:11:17.998Z [ERROR] server-0b58ead8-5413-2152-c635-44a50e9bf1e5: failed to get log: index=3 error="log not found"
2025-07-05T01:11:17.999Z [INFO]  creating new snapshot: path=/tmp/raft1562782485/snapshots/2-15-1751677877999.tmp
2025-07-05T01:11:17.999Z [INFO]  server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:17.999Z [INFO]  server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f: copied to local snapshot: bytes=71
2025-07-05T01:11:17.999Z [INFO]  server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f: snapshot restore progress: id=2-15-1751677877999 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:17.999Z [INFO]  server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f: Installed remote snapshot
2025-07-05T01:11:17.999Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: pipelining replication: peer="{Voter server-7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f 7d396d97-7a9d-cdd8-0ca0-07c7dfb5995f}"
2025-07-05T01:11:17.999Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:18.002Z [ERROR] server-0b58ead8-5413-2152-c635-44a50e9bf1e5: failed to get log: index=3 error="log not found"
2025-07-05T01:11:18.003Z [INFO]  creating new snapshot: path=/tmp/raft3615658467/snapshots/2-15-1751677878003.tmp
2025-07-05T01:11:18.003Z [INFO]  server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:18.003Z [INFO]  server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d: copied to local snapshot: bytes=71
2025-07-05T01:11:18.004Z [INFO]  server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d: snapshot restore progress: id=2-15-1751677878003 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:18.004Z [INFO]  server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d: Installed remote snapshot
2025-07-05T01:11:18.004Z [INFO]  server-0b58ead8-5413-2152-c635-44a50e9bf1e5: pipelining replication: peer="{Voter server-32a0b42f-5cc8-217d-7df6-4af4af8bd63d 32a0b42f-5cc8-217d-7df6-4af4af8bd63d}"
2025-07-05T01:11:19.005Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:17.999926118 +0000 UTC m=+53.248663336 (2 nodes), 531ns from start of poll, 1.558104025s from cluster start. Timeout at 2025-07-05 01:11:19.005312547 +0000 UTC m=+54.254049755, 1.005386419s after stability
2025-07-05T01:11:19.016Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: aborting pipeline replication: peer="{Voter server-d0964983-52dd-f89d-535b-88516b87615c d0964983-52dd-f89d-535b-88516b87615c}"
2025-07-05T01:11:19.016Z [INFO]  server-18580028-ff20-542d-e063-9d0f436d75f0: aborting pipeline replication: peer="{Voter server-d1780ae2-3d90-a7e5-d7f0-27be2c399203 d1780ae2-3d90-a7e5-d7f0-27be2c399203}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-07-05T01:11:19.018Z [INFO]  server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1 Address:8ad741cf-d5a7-c2e9-b617-a57d6c1652e1} {Suffrage:Voter ID:server-db5b5d6f-dc05-75a4-b237-fad4efac94b8 Address:db5b5d6f-dc05-75a4-b237-fad4efac94b8} {Suffrage:Voter ID:server-841c4cf9-4475-9643-6797-0d16d9f68b8b Address:841c4cf9-4475-9643-6797-0d16d9f68b8b}]"
2025-07-05T01:11:19.018Z [INFO]  server-db5b5d6f-dc05-75a4-b237-fad4efac94b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1 Address:8ad741cf-d5a7-c2e9-b617-a57d6c1652e1} {Suffrage:Voter ID:server-db5b5d6f-dc05-75a4-b237-fad4efac94b8 Address:db5b5d6f-dc05-75a4-b237-fad4efac94b8} {Suffrage:Voter ID:server-841c4cf9-4475-9643-6797-0d16d9f68b8b Address:841c4cf9-4475-9643-6797-0d16d9f68b8b}]"
2025-07-05T01:11:19.018Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1 Address:8ad741cf-d5a7-c2e9-b617-a57d6c1652e1} {Suffrage:Voter ID:server-db5b5d6f-dc05-75a4-b237-fad4efac94b8 Address:db5b5d6f-dc05-75a4-b237-fad4efac94b8} {Suffrage:Voter ID:server-841c4cf9-4475-9643-6797-0d16d9f68b8b Address:841c4cf9-4475-9643-6797-0d16d9f68b8b}]"
2025-07-05T01:11:19.018Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:19.018Z [INFO]  server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1: entering follower state: follower="Node at 8ad741cf-d5a7-c2e9-b617-a57d6c1652e1 [Follower]" leader-address= leader-id=
2025-07-05T01:11:19.018Z [INFO]  server-db5b5d6f-dc05-75a4-b237-fad4efac94b8: entering follower state: follower="Node at db5b5d6f-dc05-75a4-b237-fad4efac94b8 [Follower]" leader-address= leader-id=
2025-07-05T01:11:19.018Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: entering follower state: follower="Node at 841c4cf9-4475-9643-6797-0d16d9f68b8b [Follower]" leader-address= leader-id=
2025-07-05T01:11:19.618Z [WARN]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:19.618Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: entering candidate state: node="Node at 841c4cf9-4475-9643-6797-0d16d9f68b8b [Candidate]" term=2
2025-07-05T01:11:19.618Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: election won: term=2 tally=2
2025-07-05T01:11:19.618Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: entering leader state: leader="Node at 841c4cf9-4475-9643-6797-0d16d9f68b8b [Leader]"
2025-07-05T01:11:19.618Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: added peer, starting replication: peer=server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1
2025-07-05T01:11:19.618Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: added peer, starting replication: peer=server-db5b5d6f-dc05-75a4-b237-fad4efac94b8
2025-07-05T01:11:19.618Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: pipelining replication: peer="{Voter server-db5b5d6f-dc05-75a4-b237-fad4efac94b8 db5b5d6f-dc05-75a4-b237-fad4efac94b8}"
2025-07-05T01:11:19.618Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: pipelining replication: peer="{Voter server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1 8ad741cf-d5a7-c2e9-b617-a57d6c1652e1}"
2025-07-05T01:11:20.624Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:19.618616622 +0000 UTC m=+54.867353830 (1 nodes), 600.157846ms from start of poll, 600.372058ms from cluster start. Timeout at 2025-07-05 01:11:20.624629846 +0000 UTC m=+55.873367064, 1.006013234s after stability
2025-07-05T01:11:20.630Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: starting snapshot up to: index=12
2025-07-05T01:11:20.630Z [INFO]  creating new snapshot: path=/tmp/raft2168746734/snapshots/2-12-1751677880630.tmp
2025-07-05T01:11:20.631Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: snapshot complete up to: index=12
2025-07-05T01:11:20.632Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58558d95-3aec-374f-dd21-79447642e450 Address:58558d95-3aec-374f-dd21-79447642e450} {Suffrage:Voter ID:server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c Address:b70b95e0-4800-b18d-a5c1-b7705d0c3e6c} {Suffrage:Voter ID:server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9 Address:cb1a65bd-5a8b-8e2a-8712-43484dd68ec9}]"
2025-07-05T01:11:20.632Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58558d95-3aec-374f-dd21-79447642e450 Address:58558d95-3aec-374f-dd21-79447642e450} {Suffrage:Voter ID:server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c Address:b70b95e0-4800-b18d-a5c1-b7705d0c3e6c} {Suffrage:Voter ID:server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9 Address:cb1a65bd-5a8b-8e2a-8712-43484dd68ec9}]"
2025-07-05T01:11:20.632Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58558d95-3aec-374f-dd21-79447642e450 Address:58558d95-3aec-374f-dd21-79447642e450} {Suffrage:Voter ID:server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c Address:b70b95e0-4800-b18d-a5c1-b7705d0c3e6c} {Suffrage:Voter ID:server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9 Address:cb1a65bd-5a8b-8e2a-8712-43484dd68ec9}]"
2025-07-05T01:11:20.632Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:20.632Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: entering follower state: follower="Node at 58558d95-3aec-374f-dd21-79447642e450 [Follower]" leader-address= leader-id=
2025-07-05T01:11:20.632Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: entering follower state: follower="Node at b70b95e0-4800-b18d-a5c1-b7705d0c3e6c [Follower]" leader-address= leader-id=
2025-07-05T01:11:20.632Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: entering follower state: follower="Node at cb1a65bd-5a8b-8e2a-8712-43484dd68ec9 [Follower]" leader-address= leader-id=
2025-07-05T01:11:21.442Z [WARN]  server-58558d95-3aec-374f-dd21-79447642e450: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:21.442Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: entering candidate state: node="Node at 58558d95-3aec-374f-dd21-79447642e450 [Candidate]" term=2
2025-07-05T01:11:21.443Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: election won: term=2 tally=2
2025-07-05T01:11:21.443Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: entering leader state: leader="Node at 58558d95-3aec-374f-dd21-79447642e450 [Leader]"
2025-07-05T01:11:21.443Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: added peer, starting replication: peer=server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c
2025-07-05T01:11:21.443Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: added peer, starting replication: peer=server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9
2025-07-05T01:11:21.443Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: pipelining replication: peer="{Voter server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c b70b95e0-4800-b18d-a5c1-b7705d0c3e6c}"
2025-07-05T01:11:21.443Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: pipelining replication: peer="{Voter server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9 cb1a65bd-5a8b-8e2a-8712-43484dd68ec9}"
2025-07-05T01:11:22.451Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:21.443050615 +0000 UTC m=+56.691787823 (1 nodes), 810.439113ms from start of poll, 810.71447ms from cluster start. Timeout at 2025-07-05 01:11:22.451122078 +0000 UTC m=+57.699859296, 1.008071473s after stability
2025-07-05T01:11:22.451Z [INFO]  creating new snapshot: path=/tmp/raft3456754768/snapshots/2-15-1751677882451.tmp
2025-07-05T01:11:22.551Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: copied to local snapshot: bytes=71
2025-07-05T01:11:22.552Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: snapshot restore progress: id=2-15-1751677882451 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:22.552Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: removing all old logs from log store
2025-07-05T01:11:22.552Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: compacting logs: from=1 to=2
2025-07-05T01:11:22.552Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: restored user snapshot: index=15
2025-07-05T01:11:22.552Z [ERROR] server-58558d95-3aec-374f-dd21-79447642e450: failed to get log: index=2 error="log not found"
2025-07-05T01:11:22.552Z [ERROR] server-58558d95-3aec-374f-dd21-79447642e450: failed to get log: index=2 error="log not found"
2025-07-05T01:11:22.552Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: aborting pipeline replication: peer="{Voter server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c b70b95e0-4800-b18d-a5c1-b7705d0c3e6c}"
2025-07-05T01:11:22.552Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: aborting pipeline replication: peer="{Voter server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9 cb1a65bd-5a8b-8e2a-8712-43484dd68ec9}"
2025-07-05T01:11:22.562Z [ERROR] server-58558d95-3aec-374f-dd21-79447642e450: failed to get log: index=2 error="log not found"
2025-07-05T01:11:22.563Z [ERROR] server-58558d95-3aec-374f-dd21-79447642e450: failed to get log: index=2 error="log not found"
2025-07-05T01:11:22.563Z [INFO]  creating new snapshot: path=/tmp/raft2969973959/snapshots/2-15-1751677882563.tmp
2025-07-05T01:11:22.563Z [INFO]  creating new snapshot: path=/tmp/raft545269505/snapshots/2-15-1751677882563.tmp
2025-07-05T01:11:22.563Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:22.563Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:22.563Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: copied to local snapshot: bytes=71
2025-07-05T01:11:22.563Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: snapshot restore progress: id=2-15-1751677882563 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:22.563Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: removing all old logs from log store
2025-07-05T01:11:22.563Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: compacting logs: from=1 to=2
2025-07-05T01:11:22.563Z [INFO]  server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9: Installed remote snapshot
2025-07-05T01:11:22.563Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: copied to local snapshot: bytes=71
2025-07-05T01:11:22.564Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: snapshot restore progress: id=2-15-1751677882563 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:22.564Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: removing all old logs from log store
2025-07-05T01:11:22.564Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: compacting logs: from=1 to=2
2025-07-05T01:11:22.564Z [INFO]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: Installed remote snapshot
2025-07-05T01:11:22.564Z [WARN]  server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c: skipping application of old log: index=15
2025-07-05T01:11:22.564Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: pipelining replication: peer="{Voter server-b70b95e0-4800-b18d-a5c1-b7705d0c3e6c b70b95e0-4800-b18d-a5c1-b7705d0c3e6c}"
2025-07-05T01:11:22.564Z [INFO]  server-58558d95-3aec-374f-dd21-79447642e450: pipelining replication: peer="{Voter server-cb1a65bd-5a8b-8e2a-8712-43484dd68ec9 cb1a65bd-5a8b-8e2a-8712-43484dd68ec9}"
2025-07-05T01:11:22.564Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:23.570Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:22.564212323 +0000 UTC m=+57.812949541 (2 nodes), 632ns from start of poll, 1.931876188s from cluster start. Timeout at 2025-07-05 01:11:23.570194048 +0000 UTC m=+58.818931266, 1.005981725s after stability
2025-07-05T01:11:23.581Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: aborting pipeline replication: peer="{Voter server-db5b5d6f-dc05-75a4-b237-fad4efac94b8 db5b5d6f-dc05-75a4-b237-fad4efac94b8}"
2025-07-05T01:11:23.581Z [INFO]  server-841c4cf9-4475-9643-6797-0d16d9f68b8b: aborting pipeline replication: peer="{Voter server-8ad741cf-d5a7-c2e9-b617-a57d6c1652e1 8ad741cf-d5a7-c2e9-b617-a57d6c1652e1}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-07-05T01:11:23.585Z [INFO]  server-9f6131e2-9076-2d1b-349b-0342b7b622b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6131e2-9076-2d1b-349b-0342b7b622b2 Address:9f6131e2-9076-2d1b-349b-0342b7b622b2} {Suffrage:Voter ID:server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 Address:bea66a97-e105-6a33-e0f2-ebb9fc81f0b1} {Suffrage:Voter ID:server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663 Address:29bba3f9-811c-f2c4-cdb9-55e1d2eb4663}]"
2025-07-05T01:11:23.585Z [INFO]  server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6131e2-9076-2d1b-349b-0342b7b622b2 Address:9f6131e2-9076-2d1b-349b-0342b7b622b2} {Suffrage:Voter ID:server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 Address:bea66a97-e105-6a33-e0f2-ebb9fc81f0b1} {Suffrage:Voter ID:server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663 Address:29bba3f9-811c-f2c4-cdb9-55e1d2eb4663}]"
2025-07-05T01:11:23.585Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6131e2-9076-2d1b-349b-0342b7b622b2 Address:9f6131e2-9076-2d1b-349b-0342b7b622b2} {Suffrage:Voter ID:server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 Address:bea66a97-e105-6a33-e0f2-ebb9fc81f0b1} {Suffrage:Voter ID:server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663 Address:29bba3f9-811c-f2c4-cdb9-55e1d2eb4663}]"
2025-07-05T01:11:23.585Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:23.588Z [INFO]  server-9f6131e2-9076-2d1b-349b-0342b7b622b2: entering follower state: follower="Node at 9f6131e2-9076-2d1b-349b-0342b7b622b2 [Follower]" leader-address= leader-id=
2025-07-05T01:11:23.588Z [INFO]  server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1: entering follower state: follower="Node at bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 [Follower]" leader-address= leader-id=
2025-07-05T01:11:23.588Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: entering follower state: follower="Node at 29bba3f9-811c-f2c4-cdb9-55e1d2eb4663 [Follower]" leader-address= leader-id=
2025-07-05T01:11:24.329Z [WARN]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:24.329Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: entering candidate state: node="Node at 29bba3f9-811c-f2c4-cdb9-55e1d2eb4663 [Candidate]" term=2
2025-07-05T01:11:24.329Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: election won: term=2 tally=2
2025-07-05T01:11:24.329Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: entering leader state: leader="Node at 29bba3f9-811c-f2c4-cdb9-55e1d2eb4663 [Leader]"
2025-07-05T01:11:24.329Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: added peer, starting replication: peer=server-9f6131e2-9076-2d1b-349b-0342b7b622b2
2025-07-05T01:11:24.329Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: added peer, starting replication: peer=server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1
2025-07-05T01:11:24.329Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: pipelining replication: peer="{Voter server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 bea66a97-e105-6a33-e0f2-ebb9fc81f0b1}"
2025-07-05T01:11:24.330Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: pipelining replication: peer="{Voter server-9f6131e2-9076-2d1b-349b-0342b7b622b2 9f6131e2-9076-2d1b-349b-0342b7b622b2}"
2025-07-05T01:11:25.335Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:24.32967815 +0000 UTC m=+59.578415358 (1 nodes), 744.104716ms from start of poll, 745.019641ms from cluster start. Timeout at 2025-07-05 01:11:25.334991152 +0000 UTC m=+60.583728360, 1.005313002s after stability
2025-07-05T01:11:25.335Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: starting snapshot up to: index=12
2025-07-05T01:11:25.335Z [INFO]  creating new snapshot: path=/tmp/raft3741930944/snapshots/2-12-1751677885335.tmp
2025-07-05T01:11:25.335Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: snapshot complete up to: index=12
2025-07-05T01:11:25.336Z [INFO]  creating new snapshot: path=/tmp/raft3741930944/snapshots/2-113-1751677885336.tmp
2025-07-05T01:11:25.336Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: copied to local snapshot: bytes=71
2025-07-05T01:11:25.336Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: snapshot restore progress: id=2-113-1751677885336 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:25.336Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: restored user snapshot: index=113
2025-07-05T01:11:25.336Z [ERROR] server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: failed to get log: index=23 error="log not found"
2025-07-05T01:11:25.336Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: aborting pipeline replication: peer="{Voter server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 bea66a97-e105-6a33-e0f2-ebb9fc81f0b1}"
2025-07-05T01:11:25.337Z [ERROR] server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: failed to get log: index=23 error="log not found"
2025-07-05T01:11:25.337Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: aborting pipeline replication: peer="{Voter server-9f6131e2-9076-2d1b-349b-0342b7b622b2 9f6131e2-9076-2d1b-349b-0342b7b622b2}"
2025-07-05T01:11:25.343Z [ERROR] server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: failed to get log: index=23 error="log not found"
2025-07-05T01:11:25.343Z [ERROR] server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: failed to get log: index=23 error="log not found"
2025-07-05T01:11:25.343Z [INFO]  creating new snapshot: path=/tmp/raft2943389852/snapshots/2-113-1751677885343.tmp
2025-07-05T01:11:25.343Z [INFO]  creating new snapshot: path=/tmp/raft94532418/snapshots/2-113-1751677885343.tmp
2025-07-05T01:11:25.345Z [INFO]  server-9f6131e2-9076-2d1b-349b-0342b7b622b2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:25.345Z [INFO]  server-9f6131e2-9076-2d1b-349b-0342b7b622b2: copied to local snapshot: bytes=71
2025-07-05T01:11:25.345Z [INFO]  server-9f6131e2-9076-2d1b-349b-0342b7b622b2: snapshot restore progress: id=2-113-1751677885343 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:25.345Z [INFO]  server-9f6131e2-9076-2d1b-349b-0342b7b622b2: Installed remote snapshot
2025-07-05T01:11:25.345Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: pipelining replication: peer="{Voter server-9f6131e2-9076-2d1b-349b-0342b7b622b2 9f6131e2-9076-2d1b-349b-0342b7b622b2}"
2025-07-05T01:11:25.345Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:25.345Z [INFO]  server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:25.345Z [INFO]  server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1: copied to local snapshot: bytes=71
2025-07-05T01:11:25.346Z [INFO]  server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1: snapshot restore progress: id=2-113-1751677885343 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:25.346Z [INFO]  server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1: Installed remote snapshot
2025-07-05T01:11:25.346Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: pipelining replication: peer="{Voter server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 bea66a97-e105-6a33-e0f2-ebb9fc81f0b1}"
2025-07-05T01:11:26.350Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:25.34571548 +0000 UTC m=+60.594452698 (2 nodes), 721ns from start of poll, 1.761056981s from cluster start. Timeout at 2025-07-05 01:11:26.350919888 +0000 UTC m=+61.599657106, 1.005204408s after stability
2025-07-05T01:11:26.367Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: aborting pipeline replication: peer="{Voter server-bea66a97-e105-6a33-e0f2-ebb9fc81f0b1 bea66a97-e105-6a33-e0f2-ebb9fc81f0b1}"
2025-07-05T01:11:26.367Z [INFO]  server-29bba3f9-811c-f2c4-cdb9-55e1d2eb4663: aborting pipeline replication: peer="{Voter server-9f6131e2-9076-2d1b-349b-0342b7b622b2 9f6131e2-9076-2d1b-349b-0342b7b622b2}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-07-05T01:11:26.375Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-577c0213-06c9-d745-2fab-db2e84c35e69 Address:577c0213-06c9-d745-2fab-db2e84c35e69} {Suffrage:Voter ID:server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 Address:4d4c5f25-f7e5-e09b-e0ec-2461300c5f92} {Suffrage:Voter ID:server-67abea65-2b04-dfb1-b9c7-72cdf8040515 Address:67abea65-2b04-dfb1-b9c7-72cdf8040515}]"
2025-07-05T01:11:26.375Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: entering follower state: follower="Node at 577c0213-06c9-d745-2fab-db2e84c35e69 [Follower]" leader-address= leader-id=
2025-07-05T01:11:26.375Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-577c0213-06c9-d745-2fab-db2e84c35e69 Address:577c0213-06c9-d745-2fab-db2e84c35e69} {Suffrage:Voter ID:server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 Address:4d4c5f25-f7e5-e09b-e0ec-2461300c5f92} {Suffrage:Voter ID:server-67abea65-2b04-dfb1-b9c7-72cdf8040515 Address:67abea65-2b04-dfb1-b9c7-72cdf8040515}]"
2025-07-05T01:11:26.375Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: entering follower state: follower="Node at 4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 [Follower]" leader-address= leader-id=
2025-07-05T01:11:26.375Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-577c0213-06c9-d745-2fab-db2e84c35e69 Address:577c0213-06c9-d745-2fab-db2e84c35e69} {Suffrage:Voter ID:server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 Address:4d4c5f25-f7e5-e09b-e0ec-2461300c5f92} {Suffrage:Voter ID:server-67abea65-2b04-dfb1-b9c7-72cdf8040515 Address:67abea65-2b04-dfb1-b9c7-72cdf8040515}]"
2025-07-05T01:11:26.375Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:26.375Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: entering follower state: follower="Node at 67abea65-2b04-dfb1-b9c7-72cdf8040515 [Follower]" leader-address= leader-id=
2025-07-05T01:11:26.994Z [WARN]  server-577c0213-06c9-d745-2fab-db2e84c35e69: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:26.994Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: entering candidate state: node="Node at 577c0213-06c9-d745-2fab-db2e84c35e69 [Candidate]" term=2
2025-07-05T01:11:26.994Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: election won: term=2 tally=2
2025-07-05T01:11:26.994Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: entering leader state: leader="Node at 577c0213-06c9-d745-2fab-db2e84c35e69 [Leader]"
2025-07-05T01:11:26.994Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: added peer, starting replication: peer=server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92
2025-07-05T01:11:26.994Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: added peer, starting replication: peer=server-67abea65-2b04-dfb1-b9c7-72cdf8040515
2025-07-05T01:11:26.994Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: pipelining replication: peer="{Voter server-67abea65-2b04-dfb1-b9c7-72cdf8040515 67abea65-2b04-dfb1-b9c7-72cdf8040515}"
2025-07-05T01:11:26.994Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: pipelining replication: peer="{Voter server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 4d4c5f25-f7e5-e09b-e0ec-2461300c5f92}"
2025-07-05T01:11:27.999Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:26.994763605 +0000 UTC m=+62.243500813 (1 nodes), 618.8638ms from start of poll, 619.262758ms from cluster start. Timeout at 2025-07-05 01:11:27.999776443 +0000 UTC m=+63.248513661, 1.005012848s after stability
2025-07-05T01:11:28.000Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: starting snapshot up to: index=12
2025-07-05T01:11:28.000Z [INFO]  creating new snapshot: path=/tmp/raft3823330973/snapshots/2-12-1751677888000.tmp
2025-07-05T01:11:28.000Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: snapshot complete up to: index=12
2025-07-05T01:11:28.001Z [INFO]  creating new snapshot: path=/tmp/raft3823330973/snapshots/2-113-1751677888001.tmp
2025-07-05T01:11:28.001Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: copied to local snapshot: bytes=71
2025-07-05T01:11:28.002Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: snapshot restore progress: id=2-113-1751677888001 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:28.002Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: removing all old logs from log store
2025-07-05T01:11:28.002Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: compacting logs: from=1 to=22
2025-07-05T01:11:28.002Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: restored user snapshot: index=113
2025-07-05T01:11:28.002Z [ERROR] server-577c0213-06c9-d745-2fab-db2e84c35e69: failed to get log: index=22 error="log not found"
2025-07-05T01:11:28.002Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: aborting pipeline replication: peer="{Voter server-67abea65-2b04-dfb1-b9c7-72cdf8040515 67abea65-2b04-dfb1-b9c7-72cdf8040515}"
2025-07-05T01:11:28.002Z [ERROR] server-577c0213-06c9-d745-2fab-db2e84c35e69: failed to get log: index=20 error="log not found"
2025-07-05T01:11:28.002Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: aborting pipeline replication: peer="{Voter server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 4d4c5f25-f7e5-e09b-e0ec-2461300c5f92}"
2025-07-05T01:11:28.002Z [ERROR] server-577c0213-06c9-d745-2fab-db2e84c35e69: failed to get log: index=20 error="log not found"
2025-07-05T01:11:28.002Z [INFO]  creating new snapshot: path=/tmp/raft1594209328/snapshots/2-113-1751677888002.tmp
2025-07-05T01:11:28.002Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:28.007Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: copied to local snapshot: bytes=71
2025-07-05T01:11:28.007Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: snapshot restore progress: id=2-113-1751677888002 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:28.007Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: removing all old logs from log store
2025-07-05T01:11:28.007Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: compacting logs: from=1 to=20
2025-07-05T01:11:28.007Z [INFO]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: Installed remote snapshot
2025-07-05T01:11:28.007Z [WARN]  server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92: skipping application of old log: index=22
2025-07-05T01:11:28.007Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: pipelining replication: peer="{Voter server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 4d4c5f25-f7e5-e09b-e0ec-2461300c5f92}"
2025-07-05T01:11:28.007Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:28.007Z [ERROR] server-577c0213-06c9-d745-2fab-db2e84c35e69: failed to get log: index=22 error="log not found"
2025-07-05T01:11:28.008Z [INFO]  creating new snapshot: path=/tmp/raft409823713/snapshots/2-113-1751677888008.tmp
2025-07-05T01:11:28.008Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:28.008Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: copied to local snapshot: bytes=71
2025-07-05T01:11:28.008Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: snapshot restore progress: id=2-113-1751677888008 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:28.008Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: removing all old logs from log store
2025-07-05T01:11:28.008Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: compacting logs: from=1 to=22
2025-07-05T01:11:28.008Z [INFO]  server-67abea65-2b04-dfb1-b9c7-72cdf8040515: Installed remote snapshot
2025-07-05T01:11:28.019Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: pipelining replication: peer="{Voter server-67abea65-2b04-dfb1-b9c7-72cdf8040515 67abea65-2b04-dfb1-b9c7-72cdf8040515}"
2025-07-05T01:11:29.013Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:28.007903802 +0000 UTC m=+63.256641020 (2 nodes), 571ns from start of poll, 1.632402965s from cluster start. Timeout at 2025-07-05 01:11:29.013855772 +0000 UTC m=+64.262592980, 1.00595196s after stability
2025-07-05T01:11:29.025Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: aborting pipeline replication: peer="{Voter server-67abea65-2b04-dfb1-b9c7-72cdf8040515 67abea65-2b04-dfb1-b9c7-72cdf8040515}"
2025-07-05T01:11:29.025Z [INFO]  server-577c0213-06c9-d745-2fab-db2e84c35e69: aborting pipeline replication: peer="{Voter server-4d4c5f25-f7e5-e09b-e0ec-2461300c5f92 4d4c5f25-f7e5-e09b-e0ec-2461300c5f92}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-07-05T01:11:29.026Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-678c1fc3-a524-4c25-9662-3c1501249b54 Address:678c1fc3-a524-4c25-9662-3c1501249b54} {Suffrage:Voter ID:server-8655ac22-e90c-aa85-0fa5-f360764ddf92 Address:8655ac22-e90c-aa85-0fa5-f360764ddf92} {Suffrage:Voter ID:server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa Address:ddf4bc23-f72f-d9ee-7348-5100dbd1cafa}]"
2025-07-05T01:11:29.027Z [INFO]  server-8655ac22-e90c-aa85-0fa5-f360764ddf92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-678c1fc3-a524-4c25-9662-3c1501249b54 Address:678c1fc3-a524-4c25-9662-3c1501249b54} {Suffrage:Voter ID:server-8655ac22-e90c-aa85-0fa5-f360764ddf92 Address:8655ac22-e90c-aa85-0fa5-f360764ddf92} {Suffrage:Voter ID:server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa Address:ddf4bc23-f72f-d9ee-7348-5100dbd1cafa}]"
2025-07-05T01:11:29.027Z [INFO]  server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-678c1fc3-a524-4c25-9662-3c1501249b54 Address:678c1fc3-a524-4c25-9662-3c1501249b54} {Suffrage:Voter ID:server-8655ac22-e90c-aa85-0fa5-f360764ddf92 Address:8655ac22-e90c-aa85-0fa5-f360764ddf92} {Suffrage:Voter ID:server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa Address:ddf4bc23-f72f-d9ee-7348-5100dbd1cafa}]"
2025-07-05T01:11:29.027Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:29.027Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: entering follower state: follower="Node at 678c1fc3-a524-4c25-9662-3c1501249b54 [Follower]" leader-address= leader-id=
2025-07-05T01:11:29.027Z [INFO]  server-8655ac22-e90c-aa85-0fa5-f360764ddf92: entering follower state: follower="Node at 8655ac22-e90c-aa85-0fa5-f360764ddf92 [Follower]" leader-address= leader-id=
2025-07-05T01:11:29.027Z [INFO]  server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa: entering follower state: follower="Node at ddf4bc23-f72f-d9ee-7348-5100dbd1cafa [Follower]" leader-address= leader-id=
2025-07-05T01:11:29.763Z [WARN]  server-678c1fc3-a524-4c25-9662-3c1501249b54: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:29.763Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: entering candidate state: node="Node at 678c1fc3-a524-4c25-9662-3c1501249b54 [Candidate]" term=2
2025-07-05T01:11:29.763Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: election won: term=2 tally=2
2025-07-05T01:11:29.763Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: entering leader state: leader="Node at 678c1fc3-a524-4c25-9662-3c1501249b54 [Leader]"
2025-07-05T01:11:29.763Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: added peer, starting replication: peer=server-8655ac22-e90c-aa85-0fa5-f360764ddf92
2025-07-05T01:11:29.763Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: added peer, starting replication: peer=server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa
2025-07-05T01:11:29.763Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: pipelining replication: peer="{Voter server-8655ac22-e90c-aa85-0fa5-f360764ddf92 8655ac22-e90c-aa85-0fa5-f360764ddf92}"
2025-07-05T01:11:29.763Z [WARN]  server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa: rejecting vote request since our last term is greater: candidate=678c1fc3-a524-4c25-9662-3c1501249b54 last-term=2 last-candidate-term=1
2025-07-05T01:11:29.763Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: pipelining replication: peer="{Voter server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa ddf4bc23-f72f-d9ee-7348-5100dbd1cafa}"
2025-07-05T01:11:30.769Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:29.763954577 +0000 UTC m=+65.012691795 (1 nodes), 736.860884ms from start of poll, 737.157811ms from cluster start. Timeout at 2025-07-05 01:11:30.769884725 +0000 UTC m=+66.018621933, 1.005930138s after stability
2025-07-05T01:11:30.770Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: starting snapshot up to: index=12
2025-07-05T01:11:30.770Z [INFO]  creating new snapshot: path=/tmp/raft8568497/snapshots/2-12-1751677890770.tmp
2025-07-05T01:11:30.771Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: snapshot complete up to: index=12
2025-07-05T01:11:30.775Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c27ef8dd-087e-7fc9-86cc-0b37768beee8 Address:c27ef8dd-087e-7fc9-86cc-0b37768beee8} {Suffrage:Voter ID:server-004e80dd-f119-e8b2-537b-c290d2fe837f Address:004e80dd-f119-e8b2-537b-c290d2fe837f} {Suffrage:Voter ID:server-04e86d33-5724-8bf9-413f-bd4eab15adaf Address:04e86d33-5724-8bf9-413f-bd4eab15adaf}]"
2025-07-05T01:11:30.775Z [INFO]  server-004e80dd-f119-e8b2-537b-c290d2fe837f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c27ef8dd-087e-7fc9-86cc-0b37768beee8 Address:c27ef8dd-087e-7fc9-86cc-0b37768beee8} {Suffrage:Voter ID:server-004e80dd-f119-e8b2-537b-c290d2fe837f Address:004e80dd-f119-e8b2-537b-c290d2fe837f} {Suffrage:Voter ID:server-04e86d33-5724-8bf9-413f-bd4eab15adaf Address:04e86d33-5724-8bf9-413f-bd4eab15adaf}]"
2025-07-05T01:11:30.775Z [INFO]  server-04e86d33-5724-8bf9-413f-bd4eab15adaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c27ef8dd-087e-7fc9-86cc-0b37768beee8 Address:c27ef8dd-087e-7fc9-86cc-0b37768beee8} {Suffrage:Voter ID:server-004e80dd-f119-e8b2-537b-c290d2fe837f Address:004e80dd-f119-e8b2-537b-c290d2fe837f} {Suffrage:Voter ID:server-04e86d33-5724-8bf9-413f-bd4eab15adaf Address:04e86d33-5724-8bf9-413f-bd4eab15adaf}]"
2025-07-05T01:11:30.775Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:30.775Z [INFO]  server-004e80dd-f119-e8b2-537b-c290d2fe837f: entering follower state: follower="Node at 004e80dd-f119-e8b2-537b-c290d2fe837f [Follower]" leader-address= leader-id=
2025-07-05T01:11:30.775Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: entering follower state: follower="Node at c27ef8dd-087e-7fc9-86cc-0b37768beee8 [Follower]" leader-address= leader-id=
2025-07-05T01:11:30.775Z [INFO]  server-04e86d33-5724-8bf9-413f-bd4eab15adaf: entering follower state: follower="Node at 04e86d33-5724-8bf9-413f-bd4eab15adaf [Follower]" leader-address= leader-id=
2025-07-05T01:11:31.278Z [WARN]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:31.278Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: entering candidate state: node="Node at c27ef8dd-087e-7fc9-86cc-0b37768beee8 [Candidate]" term=2
2025-07-05T01:11:31.278Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: election won: term=2 tally=2
2025-07-05T01:11:31.278Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: entering leader state: leader="Node at c27ef8dd-087e-7fc9-86cc-0b37768beee8 [Leader]"
2025-07-05T01:11:31.278Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: added peer, starting replication: peer=server-004e80dd-f119-e8b2-537b-c290d2fe837f
2025-07-05T01:11:31.278Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: added peer, starting replication: peer=server-04e86d33-5724-8bf9-413f-bd4eab15adaf
2025-07-05T01:11:31.278Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: pipelining replication: peer="{Voter server-04e86d33-5724-8bf9-413f-bd4eab15adaf 04e86d33-5724-8bf9-413f-bd4eab15adaf}"
2025-07-05T01:11:31.278Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: pipelining replication: peer="{Voter server-004e80dd-f119-e8b2-537b-c290d2fe837f 004e80dd-f119-e8b2-537b-c290d2fe837f}"
2025-07-05T01:11:32.283Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:31.278420951 +0000 UTC m=+66.527158169 (1 nodes), 502.666971ms from start of poll, 502.941145ms from cluster start. Timeout at 2025-07-05 01:11:32.283896777 +0000 UTC m=+67.532634006, 1.005475837s after stability
2025-07-05T01:11:32.284Z [INFO]  creating new snapshot: path=/tmp/raft1359673907/snapshots/2-113-1751677892284.tmp
2025-07-05T01:11:32.285Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: copied to local snapshot: bytes=71
2025-07-05T01:11:32.285Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: snapshot restore progress: id=2-113-1751677892284 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:32.285Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: restored user snapshot: index=113
2025-07-05T01:11:32.285Z [ERROR] server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: failed to get log: index=3 error="log not found"
2025-07-05T01:11:32.285Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: aborting pipeline replication: peer="{Voter server-04e86d33-5724-8bf9-413f-bd4eab15adaf 04e86d33-5724-8bf9-413f-bd4eab15adaf}"
2025-07-05T01:11:32.285Z [ERROR] server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: failed to get log: index=3 error="log not found"
2025-07-05T01:11:32.285Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: aborting pipeline replication: peer="{Voter server-004e80dd-f119-e8b2-537b-c290d2fe837f 004e80dd-f119-e8b2-537b-c290d2fe837f}"
2025-07-05T01:11:32.292Z [ERROR] server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: failed to get log: index=3 error="log not found"
2025-07-05T01:11:32.292Z [INFO]  creating new snapshot: path=/tmp/raft2858715511/snapshots/2-113-1751677892292.tmp
2025-07-05T01:11:32.292Z [INFO]  server-04e86d33-5724-8bf9-413f-bd4eab15adaf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:32.292Z [ERROR] server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: failed to get log: index=3 error="log not found"
2025-07-05T01:11:32.293Z [INFO]  server-04e86d33-5724-8bf9-413f-bd4eab15adaf: copied to local snapshot: bytes=71
2025-07-05T01:11:32.293Z [INFO]  server-04e86d33-5724-8bf9-413f-bd4eab15adaf: snapshot restore progress: id=2-113-1751677892292 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:32.293Z [INFO]  creating new snapshot: path=/tmp/raft2615606133/snapshots/2-113-1751677892293.tmp
2025-07-05T01:11:32.293Z [INFO]  server-04e86d33-5724-8bf9-413f-bd4eab15adaf: Installed remote snapshot
2025-07-05T01:11:32.293Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: pipelining replication: peer="{Voter server-04e86d33-5724-8bf9-413f-bd4eab15adaf 04e86d33-5724-8bf9-413f-bd4eab15adaf}"
2025-07-05T01:11:32.293Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:32.293Z [INFO]  server-004e80dd-f119-e8b2-537b-c290d2fe837f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:32.293Z [INFO]  server-004e80dd-f119-e8b2-537b-c290d2fe837f: copied to local snapshot: bytes=71
2025-07-05T01:11:32.294Z [INFO]  server-004e80dd-f119-e8b2-537b-c290d2fe837f: snapshot restore progress: id=2-113-1751677892293 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:32.294Z [INFO]  server-004e80dd-f119-e8b2-537b-c290d2fe837f: Installed remote snapshot
2025-07-05T01:11:32.294Z [INFO]  server-c27ef8dd-087e-7fc9-86cc-0b37768beee8: pipelining replication: peer="{Voter server-004e80dd-f119-e8b2-537b-c290d2fe837f 004e80dd-f119-e8b2-537b-c290d2fe837f}"
2025-07-05T01:11:33.299Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:32.293623826 +0000 UTC m=+67.542361044 (2 nodes), 912ns from start of poll, 1.51814402s from cluster start. Timeout at 2025-07-05 01:11:33.29901314 +0000 UTC m=+68.547750368, 1.005389324s after stability
2025-07-05T01:11:33.310Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: aborting pipeline replication: peer="{Voter server-ddf4bc23-f72f-d9ee-7348-5100dbd1cafa ddf4bc23-f72f-d9ee-7348-5100dbd1cafa}"
2025-07-05T01:11:33.310Z [INFO]  server-678c1fc3-a524-4c25-9662-3c1501249b54: aborting pipeline replication: peer="{Voter server-8655ac22-e90c-aa85-0fa5-f360764ddf92 8655ac22-e90c-aa85-0fa5-f360764ddf92}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-07-05T01:11:33.312Z [INFO]  server-a6e4a0de-573e-3785-0045-2565ba16ee06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6e4a0de-573e-3785-0045-2565ba16ee06 Address:a6e4a0de-573e-3785-0045-2565ba16ee06} {Suffrage:Voter ID:server-c8b6f045-8199-0f14-b873-93723cd2c877 Address:c8b6f045-8199-0f14-b873-93723cd2c877} {Suffrage:Voter ID:server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db Address:21b0d997-b57d-3668-2e5d-1bb2b8f6f4db}]"
2025-07-05T01:11:33.312Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6e4a0de-573e-3785-0045-2565ba16ee06 Address:a6e4a0de-573e-3785-0045-2565ba16ee06} {Suffrage:Voter ID:server-c8b6f045-8199-0f14-b873-93723cd2c877 Address:c8b6f045-8199-0f14-b873-93723cd2c877} {Suffrage:Voter ID:server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db Address:21b0d997-b57d-3668-2e5d-1bb2b8f6f4db}]"
2025-07-05T01:11:33.312Z [INFO]  server-a6e4a0de-573e-3785-0045-2565ba16ee06: entering follower state: follower="Node at a6e4a0de-573e-3785-0045-2565ba16ee06 [Follower]" leader-address= leader-id=
2025-07-05T01:11:33.312Z [INFO]  server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6e4a0de-573e-3785-0045-2565ba16ee06 Address:a6e4a0de-573e-3785-0045-2565ba16ee06} {Suffrage:Voter ID:server-c8b6f045-8199-0f14-b873-93723cd2c877 Address:c8b6f045-8199-0f14-b873-93723cd2c877} {Suffrage:Voter ID:server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db Address:21b0d997-b57d-3668-2e5d-1bb2b8f6f4db}]"
2025-07-05T01:11:33.312Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:33.312Z [INFO]  server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db: entering follower state: follower="Node at 21b0d997-b57d-3668-2e5d-1bb2b8f6f4db [Follower]" leader-address= leader-id=
2025-07-05T01:11:33.312Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: entering follower state: follower="Node at c8b6f045-8199-0f14-b873-93723cd2c877 [Follower]" leader-address= leader-id=
2025-07-05T01:11:33.818Z [WARN]  server-c8b6f045-8199-0f14-b873-93723cd2c877: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:33.818Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: entering candidate state: node="Node at c8b6f045-8199-0f14-b873-93723cd2c877 [Candidate]" term=2
2025-07-05T01:11:33.818Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: election won: term=2 tally=2
2025-07-05T01:11:33.818Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: entering leader state: leader="Node at c8b6f045-8199-0f14-b873-93723cd2c877 [Leader]"
2025-07-05T01:11:33.818Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: added peer, starting replication: peer=server-a6e4a0de-573e-3785-0045-2565ba16ee06
2025-07-05T01:11:33.818Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: added peer, starting replication: peer=server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db
2025-07-05T01:11:33.818Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: pipelining replication: peer="{Voter server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db 21b0d997-b57d-3668-2e5d-1bb2b8f6f4db}"
2025-07-05T01:11:33.818Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: pipelining replication: peer="{Voter server-a6e4a0de-573e-3785-0045-2565ba16ee06 a6e4a0de-573e-3785-0045-2565ba16ee06}"
2025-07-05T01:11:34.825Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:33.818985308 +0000 UTC m=+69.067722536 (1 nodes), 506.624591ms from start of poll, 506.937047ms from cluster start. Timeout at 2025-07-05 01:11:34.825043216 +0000 UTC m=+70.073780434, 1.006057898s after stability
2025-07-05T01:11:34.825Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: starting snapshot up to: index=12
2025-07-05T01:11:34.825Z [INFO]  creating new snapshot: path=/tmp/raft2703903902/snapshots/2-12-1751677894825.tmp
2025-07-05T01:11:34.826Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: snapshot complete up to: index=12
2025-07-05T01:11:34.828Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60a8b4ef-94bf-3852-c07e-aea202bff505 Address:60a8b4ef-94bf-3852-c07e-aea202bff505} {Suffrage:Voter ID:server-d94995a5-3da1-6020-0b09-d61cc46541aa Address:d94995a5-3da1-6020-0b09-d61cc46541aa} {Suffrage:Voter ID:server-9d078d78-0e60-97ed-3fa3-4eb21b519c66 Address:9d078d78-0e60-97ed-3fa3-4eb21b519c66}]"
2025-07-05T01:11:34.828Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60a8b4ef-94bf-3852-c07e-aea202bff505 Address:60a8b4ef-94bf-3852-c07e-aea202bff505} {Suffrage:Voter ID:server-d94995a5-3da1-6020-0b09-d61cc46541aa Address:d94995a5-3da1-6020-0b09-d61cc46541aa} {Suffrage:Voter ID:server-9d078d78-0e60-97ed-3fa3-4eb21b519c66 Address:9d078d78-0e60-97ed-3fa3-4eb21b519c66}]"
2025-07-05T01:11:34.828Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: entering follower state: follower="Node at 60a8b4ef-94bf-3852-c07e-aea202bff505 [Follower]" leader-address= leader-id=
2025-07-05T01:11:34.828Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: entering follower state: follower="Node at d94995a5-3da1-6020-0b09-d61cc46541aa [Follower]" leader-address= leader-id=
2025-07-05T01:11:34.828Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60a8b4ef-94bf-3852-c07e-aea202bff505 Address:60a8b4ef-94bf-3852-c07e-aea202bff505} {Suffrage:Voter ID:server-d94995a5-3da1-6020-0b09-d61cc46541aa Address:d94995a5-3da1-6020-0b09-d61cc46541aa} {Suffrage:Voter ID:server-9d078d78-0e60-97ed-3fa3-4eb21b519c66 Address:9d078d78-0e60-97ed-3fa3-4eb21b519c66}]"
2025-07-05T01:11:34.828Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:34.828Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: entering follower state: follower="Node at 9d078d78-0e60-97ed-3fa3-4eb21b519c66 [Follower]" leader-address= leader-id=
2025-07-05T01:11:35.587Z [WARN]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:35.587Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: entering candidate state: node="Node at d94995a5-3da1-6020-0b09-d61cc46541aa [Candidate]" term=2
2025-07-05T01:11:35.587Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: election won: term=2 tally=2
2025-07-05T01:11:35.587Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: entering leader state: leader="Node at d94995a5-3da1-6020-0b09-d61cc46541aa [Leader]"
2025-07-05T01:11:35.587Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: added peer, starting replication: peer=server-60a8b4ef-94bf-3852-c07e-aea202bff505
2025-07-05T01:11:35.587Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: added peer, starting replication: peer=server-9d078d78-0e60-97ed-3fa3-4eb21b519c66
2025-07-05T01:11:35.587Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: pipelining replication: peer="{Voter server-9d078d78-0e60-97ed-3fa3-4eb21b519c66 9d078d78-0e60-97ed-3fa3-4eb21b519c66}"
2025-07-05T01:11:35.587Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: pipelining replication: peer="{Voter server-60a8b4ef-94bf-3852-c07e-aea202bff505 60a8b4ef-94bf-3852-c07e-aea202bff505}"
2025-07-05T01:11:36.595Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:35.587774487 +0000 UTC m=+70.836511705 (1 nodes), 759.543664ms from start of poll, 759.970334ms from cluster start. Timeout at 2025-07-05 01:11:36.595131431 +0000 UTC m=+71.843868649, 1.007356944s after stability
2025-07-05T01:11:36.595Z [INFO]  creating new snapshot: path=/tmp/raft3108737416/snapshots/2-113-1751677896595.tmp
2025-07-05T01:11:36.595Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: copied to local snapshot: bytes=71
2025-07-05T01:11:36.596Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: snapshot restore progress: id=2-113-1751677896595 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:36.596Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: removing all old logs from log store
2025-07-05T01:11:36.596Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: compacting logs: from=1 to=2
2025-07-05T01:11:36.596Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: restored user snapshot: index=113
2025-07-05T01:11:36.596Z [ERROR] server-d94995a5-3da1-6020-0b09-d61cc46541aa: failed to get log: index=2 error="log not found"
2025-07-05T01:11:36.596Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: aborting pipeline replication: peer="{Voter server-9d078d78-0e60-97ed-3fa3-4eb21b519c66 9d078d78-0e60-97ed-3fa3-4eb21b519c66}"
2025-07-05T01:11:36.596Z [ERROR] server-d94995a5-3da1-6020-0b09-d61cc46541aa: failed to get log: index=2 error="log not found"
2025-07-05T01:11:36.596Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: aborting pipeline replication: peer="{Voter server-60a8b4ef-94bf-3852-c07e-aea202bff505 60a8b4ef-94bf-3852-c07e-aea202bff505}"
2025-07-05T01:11:36.603Z [ERROR] server-d94995a5-3da1-6020-0b09-d61cc46541aa: failed to get log: index=2 error="log not found"
2025-07-05T01:11:36.603Z [INFO]  creating new snapshot: path=/tmp/raft1490215743/snapshots/2-113-1751677896603.tmp
2025-07-05T01:11:36.604Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:36.604Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: copied to local snapshot: bytes=71
2025-07-05T01:11:36.604Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: snapshot restore progress: id=2-113-1751677896603 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:36.604Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: removing all old logs from log store
2025-07-05T01:11:36.604Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: compacting logs: from=1 to=2
2025-07-05T01:11:36.604Z [INFO]  server-9d078d78-0e60-97ed-3fa3-4eb21b519c66: Installed remote snapshot
2025-07-05T01:11:36.604Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: pipelining replication: peer="{Voter server-9d078d78-0e60-97ed-3fa3-4eb21b519c66 9d078d78-0e60-97ed-3fa3-4eb21b519c66}"
2025-07-05T01:11:36.604Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:36.606Z [ERROR] server-d94995a5-3da1-6020-0b09-d61cc46541aa: failed to get log: index=2 error="log not found"
2025-07-05T01:11:36.606Z [INFO]  creating new snapshot: path=/tmp/raft729364510/snapshots/2-113-1751677896606.tmp
2025-07-05T01:11:36.606Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:36.606Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: copied to local snapshot: bytes=71
2025-07-05T01:11:36.606Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: snapshot restore progress: id=2-113-1751677896606 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:36.606Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: removing all old logs from log store
2025-07-05T01:11:36.606Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: compacting logs: from=1 to=2
2025-07-05T01:11:36.606Z [INFO]  server-60a8b4ef-94bf-3852-c07e-aea202bff505: Installed remote snapshot
2025-07-05T01:11:36.606Z [INFO]  server-d94995a5-3da1-6020-0b09-d61cc46541aa: pipelining replication: peer="{Voter server-60a8b4ef-94bf-3852-c07e-aea202bff505 60a8b4ef-94bf-3852-c07e-aea202bff505}"
2025-07-05T01:11:37.610Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:36.604839714 +0000 UTC m=+71.853576932 (2 nodes), 731ns from start of poll, 1.777035561s from cluster start. Timeout at 2025-07-05 01:11:37.610838271 +0000 UTC m=+72.859575479, 1.005998547s after stability
2025-07-05T01:11:37.622Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: aborting pipeline replication: peer="{Voter server-21b0d997-b57d-3668-2e5d-1bb2b8f6f4db 21b0d997-b57d-3668-2e5d-1bb2b8f6f4db}"
2025-07-05T01:11:37.623Z [INFO]  server-c8b6f045-8199-0f14-b873-93723cd2c877: aborting pipeline replication: peer="{Voter server-a6e4a0de-573e-3785-0045-2565ba16ee06 a6e4a0de-573e-3785-0045-2565ba16ee06}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-07-05T01:11:37.624Z [INFO]  server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0 Address:c354ccf0-90ab-1b5b-280f-db6b1c7357a0} {Suffrage:Voter ID:server-52404b33-08bd-b242-7b8e-13f4f87becba Address:52404b33-08bd-b242-7b8e-13f4f87becba} {Suffrage:Voter ID:server-8a908409-1d20-889d-a028-609ecb20a5d8 Address:8a908409-1d20-889d-a028-609ecb20a5d8}]"
2025-07-05T01:11:37.624Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0 Address:c354ccf0-90ab-1b5b-280f-db6b1c7357a0} {Suffrage:Voter ID:server-52404b33-08bd-b242-7b8e-13f4f87becba Address:52404b33-08bd-b242-7b8e-13f4f87becba} {Suffrage:Voter ID:server-8a908409-1d20-889d-a028-609ecb20a5d8 Address:8a908409-1d20-889d-a028-609ecb20a5d8}]"
2025-07-05T01:11:37.624Z [INFO]  server-8a908409-1d20-889d-a028-609ecb20a5d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0 Address:c354ccf0-90ab-1b5b-280f-db6b1c7357a0} {Suffrage:Voter ID:server-52404b33-08bd-b242-7b8e-13f4f87becba Address:52404b33-08bd-b242-7b8e-13f4f87becba} {Suffrage:Voter ID:server-8a908409-1d20-889d-a028-609ecb20a5d8 Address:8a908409-1d20-889d-a028-609ecb20a5d8}]"
2025-07-05T01:11:37.624Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:37.624Z [INFO]  server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0: entering follower state: follower="Node at c354ccf0-90ab-1b5b-280f-db6b1c7357a0 [Follower]" leader-address= leader-id=
2025-07-05T01:11:37.624Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: entering follower state: follower="Node at 52404b33-08bd-b242-7b8e-13f4f87becba [Follower]" leader-address= leader-id=
2025-07-05T01:11:37.624Z [INFO]  server-8a908409-1d20-889d-a028-609ecb20a5d8: entering follower state: follower="Node at 8a908409-1d20-889d-a028-609ecb20a5d8 [Follower]" leader-address= leader-id=
2025-07-05T01:11:38.440Z [WARN]  server-52404b33-08bd-b242-7b8e-13f4f87becba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:38.440Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: entering candidate state: node="Node at 52404b33-08bd-b242-7b8e-13f4f87becba [Candidate]" term=2
2025-07-05T01:11:38.440Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: election won: term=2 tally=2
2025-07-05T01:11:38.440Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: entering leader state: leader="Node at 52404b33-08bd-b242-7b8e-13f4f87becba [Leader]"
2025-07-05T01:11:38.440Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: added peer, starting replication: peer=server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0
2025-07-05T01:11:38.440Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: added peer, starting replication: peer=server-8a908409-1d20-889d-a028-609ecb20a5d8
2025-07-05T01:11:38.440Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: pipelining replication: peer="{Voter server-8a908409-1d20-889d-a028-609ecb20a5d8 8a908409-1d20-889d-a028-609ecb20a5d8}"
2025-07-05T01:11:38.441Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: pipelining replication: peer="{Voter server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0 c354ccf0-90ab-1b5b-280f-db6b1c7357a0}"
2025-07-05T01:11:39.445Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:38.440769001 +0000 UTC m=+73.689506209 (1 nodes), 816.33074ms from start of poll, 816.572033ms from cluster start. Timeout at 2025-07-05 01:11:39.445914027 +0000 UTC m=+74.694651235, 1.005145026s after stability
2025-07-05T01:11:39.446Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: starting snapshot up to: index=12
2025-07-05T01:11:39.446Z [INFO]  creating new snapshot: path=/tmp/raft2704368153/snapshots/2-12-1751677899446.tmp
2025-07-05T01:11:39.446Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: snapshot complete up to: index=12
2025-07-05T01:11:39.447Z [INFO]  creating new snapshot: path=/tmp/raft2704368153/snapshots/2-1013-1751677899447.tmp
2025-07-05T01:11:39.447Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: copied to local snapshot: bytes=71
2025-07-05T01:11:39.453Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: snapshot restore progress: id=2-1013-1751677899447 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:39.453Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: restored user snapshot: index=1013
2025-07-05T01:11:39.454Z [ERROR] server-52404b33-08bd-b242-7b8e-13f4f87becba: failed to get log: index=23 error="log not found"
2025-07-05T01:11:39.454Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: aborting pipeline replication: peer="{Voter server-8a908409-1d20-889d-a028-609ecb20a5d8 8a908409-1d20-889d-a028-609ecb20a5d8}"
2025-07-05T01:11:39.455Z [ERROR] server-52404b33-08bd-b242-7b8e-13f4f87becba: failed to get log: index=23 error="log not found"
2025-07-05T01:11:39.455Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: aborting pipeline replication: peer="{Voter server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0 c354ccf0-90ab-1b5b-280f-db6b1c7357a0}"
2025-07-05T01:11:39.463Z [ERROR] server-52404b33-08bd-b242-7b8e-13f4f87becba: failed to get log: index=23 error="log not found"
2025-07-05T01:11:39.463Z [ERROR] server-52404b33-08bd-b242-7b8e-13f4f87becba: failed to get log: index=23 error="log not found"
2025-07-05T01:11:39.463Z [INFO]  creating new snapshot: path=/tmp/raft480131855/snapshots/2-1013-1751677899463.tmp
2025-07-05T01:11:39.463Z [INFO]  server-8a908409-1d20-889d-a028-609ecb20a5d8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:39.464Z [INFO]  server-8a908409-1d20-889d-a028-609ecb20a5d8: copied to local snapshot: bytes=71
2025-07-05T01:11:39.464Z [INFO]  server-8a908409-1d20-889d-a028-609ecb20a5d8: snapshot restore progress: id=2-1013-1751677899463 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:39.464Z [INFO]  server-8a908409-1d20-889d-a028-609ecb20a5d8: Installed remote snapshot
2025-07-05T01:11:39.464Z [WARN]  server-8a908409-1d20-889d-a028-609ecb20a5d8: skipping application of old log: index=22
2025-07-05T01:11:39.464Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: pipelining replication: peer="{Voter server-8a908409-1d20-889d-a028-609ecb20a5d8 8a908409-1d20-889d-a028-609ecb20a5d8}"
2025-07-05T01:11:39.464Z [INFO]  creating new snapshot: path=/tmp/raft3711837475/snapshots/2-1013-1751677899464.tmp
2025-07-05T01:11:39.464Z [INFO]  server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:39.464Z [INFO]  server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0: copied to local snapshot: bytes=71
2025-07-05T01:11:39.464Z [INFO]  server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0: snapshot restore progress: id=2-1013-1751677899464 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:39.464Z [INFO]  server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0: Installed remote snapshot
2025-07-05T01:11:39.464Z [WARN]  server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0: skipping application of old log: index=22
2025-07-05T01:11:39.464Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: pipelining replication: peer="{Voter server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0 c354ccf0-90ab-1b5b-280f-db6b1c7357a0}"
2025-07-05T01:11:39.464Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:40.475Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:39.46484865 +0000 UTC m=+74.713585858 (2 nodes), 361ns from start of poll, 1.840651682s from cluster start. Timeout at 2025-07-05 01:11:40.474957894 +0000 UTC m=+75.723695112, 1.010109254s after stability
2025-07-05T01:11:40.487Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: aborting pipeline replication: peer="{Voter server-8a908409-1d20-889d-a028-609ecb20a5d8 8a908409-1d20-889d-a028-609ecb20a5d8}"
2025-07-05T01:11:40.487Z [INFO]  server-52404b33-08bd-b242-7b8e-13f4f87becba: aborting pipeline replication: peer="{Voter server-c354ccf0-90ab-1b5b-280f-db6b1c7357a0 c354ccf0-90ab-1b5b-280f-db6b1c7357a0}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-07-05T01:11:40.489Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 Address:63ebbeeb-a3f5-e823-b2a8-8ed14cc61150} {Suffrage:Voter ID:server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 Address:bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946} {Suffrage:Voter ID:server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4 Address:43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4}]"
2025-07-05T01:11:40.489Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: entering follower state: follower="Node at 63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 [Follower]" leader-address= leader-id=
2025-07-05T01:11:40.489Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 Address:63ebbeeb-a3f5-e823-b2a8-8ed14cc61150} {Suffrage:Voter ID:server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 Address:bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946} {Suffrage:Voter ID:server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4 Address:43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4}]"
2025-07-05T01:11:40.489Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: entering follower state: follower="Node at bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 [Follower]" leader-address= leader-id=
2025-07-05T01:11:40.489Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 Address:63ebbeeb-a3f5-e823-b2a8-8ed14cc61150} {Suffrage:Voter ID:server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 Address:bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946} {Suffrage:Voter ID:server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4 Address:43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4}]"
2025-07-05T01:11:40.489Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:40.489Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: entering follower state: follower="Node at 43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4 [Follower]" leader-address= leader-id=
2025-07-05T01:11:41.122Z [WARN]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:41.122Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: entering candidate state: node="Node at 43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4 [Candidate]" term=2
2025-07-05T01:11:41.123Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: election won: term=2 tally=2
2025-07-05T01:11:41.123Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: entering leader state: leader="Node at 43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4 [Leader]"
2025-07-05T01:11:41.123Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: added peer, starting replication: peer=server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150
2025-07-05T01:11:41.123Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: added peer, starting replication: peer=server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946
2025-07-05T01:11:41.123Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: pipelining replication: peer="{Voter server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946}"
2025-07-05T01:11:41.123Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: pipelining replication: peer="{Voter server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 63ebbeeb-a3f5-e823-b2a8-8ed14cc61150}"
2025-07-05T01:11:42.128Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:41.123151888 +0000 UTC m=+76.371889116 (1 nodes), 633.345773ms from start of poll, 633.765009ms from cluster start. Timeout at 2025-07-05 01:11:42.128683028 +0000 UTC m=+77.377420236, 1.00553112s after stability
2025-07-05T01:11:42.129Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: starting snapshot up to: index=12
2025-07-05T01:11:42.129Z [INFO]  creating new snapshot: path=/tmp/raft1727306598/snapshots/2-12-1751677902129.tmp
2025-07-05T01:11:42.130Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: snapshot complete up to: index=12
2025-07-05T01:11:42.130Z [INFO]  creating new snapshot: path=/tmp/raft1727306598/snapshots/2-1013-1751677902130.tmp
2025-07-05T01:11:42.130Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: copied to local snapshot: bytes=71
2025-07-05T01:11:42.131Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: snapshot restore progress: id=2-1013-1751677902130 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:42.131Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: removing all old logs from log store
2025-07-05T01:11:42.131Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: compacting logs: from=1 to=22
2025-07-05T01:11:42.131Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: restored user snapshot: index=1013
2025-07-05T01:11:42.131Z [ERROR] server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: failed to get log: index=22 error="log not found"
2025-07-05T01:11:42.131Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: aborting pipeline replication: peer="{Voter server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946}"
2025-07-05T01:11:42.131Z [ERROR] server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: failed to get log: index=22 error="log not found"
2025-07-05T01:11:42.131Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: aborting pipeline replication: peer="{Voter server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 63ebbeeb-a3f5-e823-b2a8-8ed14cc61150}"
2025-07-05T01:11:42.136Z [ERROR] server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: failed to get log: index=22 error="log not found"
2025-07-05T01:11:42.137Z [INFO]  creating new snapshot: path=/tmp/raft4259676501/snapshots/2-1013-1751677902137.tmp
2025-07-05T01:11:42.137Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:42.137Z [ERROR] server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: failed to get log: index=22 error="log not found"
2025-07-05T01:11:42.137Z [INFO]  creating new snapshot: path=/tmp/raft2378035879/snapshots/2-1013-1751677902137.tmp
2025-07-05T01:11:42.137Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: copied to local snapshot: bytes=71
2025-07-05T01:11:42.137Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:42.137Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: snapshot restore progress: id=2-1013-1751677902137 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:42.137Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: removing all old logs from log store
2025-07-05T01:11:42.137Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: compacting logs: from=1 to=22
2025-07-05T01:11:42.137Z [INFO]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: Installed remote snapshot
2025-07-05T01:11:42.137Z [WARN]  server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946: skipping application of old log: index=22
2025-07-05T01:11:42.137Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: pipelining replication: peer="{Voter server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946}"
2025-07-05T01:11:42.137Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:42.137Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: copied to local snapshot: bytes=71
2025-07-05T01:11:42.138Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: snapshot restore progress: id=2-1013-1751677902137 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:42.138Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: removing all old logs from log store
2025-07-05T01:11:42.138Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: compacting logs: from=1 to=22
2025-07-05T01:11:42.138Z [INFO]  server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150: Installed remote snapshot
2025-07-05T01:11:42.147Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: pipelining replication: peer="{Voter server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 63ebbeeb-a3f5-e823-b2a8-8ed14cc61150}"
2025-07-05T01:11:43.147Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:42.137977095 +0000 UTC m=+77.386714313 (2 nodes), 1.333µs from start of poll, 1.648590206s from cluster start. Timeout at 2025-07-05 01:11:43.147186873 +0000 UTC m=+78.395924091, 1.009209778s after stability
2025-07-05T01:11:43.163Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: aborting pipeline replication: peer="{Voter server-bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946 bc0edb58-1c6c-8b1e-f9fb-fbb3c5630946}"
2025-07-05T01:11:43.163Z [INFO]  server-43ec14f4-4d85-f74b-d4b3-0ad42c50cdc4: aborting pipeline replication: peer="{Voter server-63ebbeeb-a3f5-e823-b2a8-8ed14cc61150 63ebbeeb-a3f5-e823-b2a8-8ed14cc61150}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-07-05T01:11:43.165Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7 Address:6733f73e-6788-d5e5-a1ba-c5447f9ea7e7} {Suffrage:Voter ID:server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4 Address:f7a1ffe0-f99a-87fe-4334-bda63919d1a4} {Suffrage:Voter ID:server-f03c008c-4339-0a40-df06-25437855d024 Address:f03c008c-4339-0a40-df06-25437855d024}]"
2025-07-05T01:11:43.165Z [INFO]  server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7 Address:6733f73e-6788-d5e5-a1ba-c5447f9ea7e7} {Suffrage:Voter ID:server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4 Address:f7a1ffe0-f99a-87fe-4334-bda63919d1a4} {Suffrage:Voter ID:server-f03c008c-4339-0a40-df06-25437855d024 Address:f03c008c-4339-0a40-df06-25437855d024}]"
2025-07-05T01:11:43.165Z [INFO]  server-f03c008c-4339-0a40-df06-25437855d024: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7 Address:6733f73e-6788-d5e5-a1ba-c5447f9ea7e7} {Suffrage:Voter ID:server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4 Address:f7a1ffe0-f99a-87fe-4334-bda63919d1a4} {Suffrage:Voter ID:server-f03c008c-4339-0a40-df06-25437855d024 Address:f03c008c-4339-0a40-df06-25437855d024}]"
2025-07-05T01:11:43.165Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:43.165Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: entering follower state: follower="Node at 6733f73e-6788-d5e5-a1ba-c5447f9ea7e7 [Follower]" leader-address= leader-id=
2025-07-05T01:11:43.165Z [INFO]  server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4: entering follower state: follower="Node at f7a1ffe0-f99a-87fe-4334-bda63919d1a4 [Follower]" leader-address= leader-id=
2025-07-05T01:11:43.165Z [INFO]  server-f03c008c-4339-0a40-df06-25437855d024: entering follower state: follower="Node at f03c008c-4339-0a40-df06-25437855d024 [Follower]" leader-address= leader-id=
2025-07-05T01:11:43.735Z [WARN]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:43.735Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: entering candidate state: node="Node at 6733f73e-6788-d5e5-a1ba-c5447f9ea7e7 [Candidate]" term=2
2025-07-05T01:11:43.735Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: election won: term=2 tally=2
2025-07-05T01:11:43.735Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: entering leader state: leader="Node at 6733f73e-6788-d5e5-a1ba-c5447f9ea7e7 [Leader]"
2025-07-05T01:11:43.735Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: added peer, starting replication: peer=server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4
2025-07-05T01:11:43.735Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: added peer, starting replication: peer=server-f03c008c-4339-0a40-df06-25437855d024
2025-07-05T01:11:43.735Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: pipelining replication: peer="{Voter server-f03c008c-4339-0a40-df06-25437855d024 f03c008c-4339-0a40-df06-25437855d024}"
2025-07-05T01:11:43.735Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: pipelining replication: peer="{Voter server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4 f7a1ffe0-f99a-87fe-4334-bda63919d1a4}"
2025-07-05T01:11:44.740Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:43.735422594 +0000 UTC m=+78.984159802 (1 nodes), 570.00571ms from start of poll, 570.32014ms from cluster start. Timeout at 2025-07-05 01:11:44.74042307 +0000 UTC m=+79.989160288, 1.005000486s after stability
2025-07-05T01:11:44.741Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: starting snapshot up to: index=12
2025-07-05T01:11:44.741Z [INFO]  creating new snapshot: path=/tmp/raft1853882755/snapshots/2-12-1751677904741.tmp
2025-07-05T01:11:44.741Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: snapshot complete up to: index=12
2025-07-05T01:11:44.742Z [INFO]  server-623ee27c-a635-7ccc-4650-7900bab344b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-623ee27c-a635-7ccc-4650-7900bab344b2 Address:623ee27c-a635-7ccc-4650-7900bab344b2} {Suffrage:Voter ID:server-5c98501a-8fe4-0909-a504-5f34715009c9 Address:5c98501a-8fe4-0909-a504-5f34715009c9} {Suffrage:Voter ID:server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9 Address:9acc4a59-cb4d-f84a-bc7b-7436a29f84c9}]"
2025-07-05T01:11:44.742Z [INFO]  server-623ee27c-a635-7ccc-4650-7900bab344b2: entering follower state: follower="Node at 623ee27c-a635-7ccc-4650-7900bab344b2 [Follower]" leader-address= leader-id=
2025-07-05T01:11:44.746Z [INFO]  server-5c98501a-8fe4-0909-a504-5f34715009c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-623ee27c-a635-7ccc-4650-7900bab344b2 Address:623ee27c-a635-7ccc-4650-7900bab344b2} {Suffrage:Voter ID:server-5c98501a-8fe4-0909-a504-5f34715009c9 Address:5c98501a-8fe4-0909-a504-5f34715009c9} {Suffrage:Voter ID:server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9 Address:9acc4a59-cb4d-f84a-bc7b-7436a29f84c9}]"
2025-07-05T01:11:44.747Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-623ee27c-a635-7ccc-4650-7900bab344b2 Address:623ee27c-a635-7ccc-4650-7900bab344b2} {Suffrage:Voter ID:server-5c98501a-8fe4-0909-a504-5f34715009c9 Address:5c98501a-8fe4-0909-a504-5f34715009c9} {Suffrage:Voter ID:server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9 Address:9acc4a59-cb4d-f84a-bc7b-7436a29f84c9}]"
2025-07-05T01:11:44.747Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:44.747Z [INFO]  server-5c98501a-8fe4-0909-a504-5f34715009c9: entering follower state: follower="Node at 5c98501a-8fe4-0909-a504-5f34715009c9 [Follower]" leader-address= leader-id=
2025-07-05T01:11:44.747Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: entering follower state: follower="Node at 9acc4a59-cb4d-f84a-bc7b-7436a29f84c9 [Follower]" leader-address= leader-id=
2025-07-05T01:11:45.286Z [WARN]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:45.286Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: entering candidate state: node="Node at 9acc4a59-cb4d-f84a-bc7b-7436a29f84c9 [Candidate]" term=2
2025-07-05T01:11:45.286Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: election won: term=2 tally=2
2025-07-05T01:11:45.286Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: entering leader state: leader="Node at 9acc4a59-cb4d-f84a-bc7b-7436a29f84c9 [Leader]"
2025-07-05T01:11:45.286Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: added peer, starting replication: peer=server-623ee27c-a635-7ccc-4650-7900bab344b2
2025-07-05T01:11:45.286Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: added peer, starting replication: peer=server-5c98501a-8fe4-0909-a504-5f34715009c9
2025-07-05T01:11:45.286Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: pipelining replication: peer="{Voter server-5c98501a-8fe4-0909-a504-5f34715009c9 5c98501a-8fe4-0909-a504-5f34715009c9}"
2025-07-05T01:11:45.286Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: pipelining replication: peer="{Voter server-623ee27c-a635-7ccc-4650-7900bab344b2 623ee27c-a635-7ccc-4650-7900bab344b2}"
2025-07-05T01:11:46.292Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:45.286541472 +0000 UTC m=+80.535278690 (1 nodes), 539.462562ms from start of poll, 543.783895ms from cluster start. Timeout at 2025-07-05 01:11:46.292469005 +0000 UTC m=+81.541206213, 1.005927523s after stability
2025-07-05T01:11:46.292Z [INFO]  creating new snapshot: path=/tmp/raft1322813428/snapshots/2-1013-1751677906292.tmp
2025-07-05T01:11:46.293Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: copied to local snapshot: bytes=71
2025-07-05T01:11:46.293Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: snapshot restore progress: id=2-1013-1751677906292 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:46.293Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: restored user snapshot: index=1013
2025-07-05T01:11:46.293Z [ERROR] server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: failed to get log: index=3 error="log not found"
2025-07-05T01:11:46.293Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: aborting pipeline replication: peer="{Voter server-5c98501a-8fe4-0909-a504-5f34715009c9 5c98501a-8fe4-0909-a504-5f34715009c9}"
2025-07-05T01:11:46.293Z [ERROR] server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: failed to get log: index=3 error="log not found"
2025-07-05T01:11:46.293Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: aborting pipeline replication: peer="{Voter server-623ee27c-a635-7ccc-4650-7900bab344b2 623ee27c-a635-7ccc-4650-7900bab344b2}"
2025-07-05T01:11:46.300Z [ERROR] server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: failed to get log: index=3 error="log not found"
2025-07-05T01:11:46.300Z [INFO]  creating new snapshot: path=/tmp/raft4219502011/snapshots/2-1013-1751677906300.tmp
2025-07-05T01:11:46.304Z [INFO]  server-623ee27c-a635-7ccc-4650-7900bab344b2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:46.311Z [INFO]  server-623ee27c-a635-7ccc-4650-7900bab344b2: copied to local snapshot: bytes=71
2025-07-05T01:11:46.311Z [INFO]  server-623ee27c-a635-7ccc-4650-7900bab344b2: snapshot restore progress: id=2-1013-1751677906300 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:46.311Z [INFO]  server-623ee27c-a635-7ccc-4650-7900bab344b2: Installed remote snapshot
2025-07-05T01:11:46.311Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: pipelining replication: peer="{Voter server-623ee27c-a635-7ccc-4650-7900bab344b2 623ee27c-a635-7ccc-4650-7900bab344b2}"
2025-07-05T01:11:46.311Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:46.311Z [ERROR] server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: failed to get log: index=3 error="log not found"
2025-07-05T01:11:46.312Z [INFO]  creating new snapshot: path=/tmp/raft2405822249/snapshots/2-1013-1751677906312.tmp
2025-07-05T01:11:46.312Z [INFO]  server-5c98501a-8fe4-0909-a504-5f34715009c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:46.312Z [INFO]  server-5c98501a-8fe4-0909-a504-5f34715009c9: copied to local snapshot: bytes=71
2025-07-05T01:11:46.313Z [INFO]  server-5c98501a-8fe4-0909-a504-5f34715009c9: snapshot restore progress: id=2-1013-1751677906312 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:46.313Z [INFO]  server-5c98501a-8fe4-0909-a504-5f34715009c9: Installed remote snapshot
2025-07-05T01:11:46.313Z [INFO]  server-9acc4a59-cb4d-f84a-bc7b-7436a29f84c9: pipelining replication: peer="{Voter server-5c98501a-8fe4-0909-a504-5f34715009c9 5c98501a-8fe4-0909-a504-5f34715009c9}"
2025-07-05T01:11:47.316Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:46.311763673 +0000 UTC m=+81.560500881 (2 nodes), 601ns from start of poll, 1.569006086s from cluster start. Timeout at 2025-07-05 01:11:47.316766012 +0000 UTC m=+82.565503230, 1.005002349s after stability
2025-07-05T01:11:47.330Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: aborting pipeline replication: peer="{Voter server-f03c008c-4339-0a40-df06-25437855d024 f03c008c-4339-0a40-df06-25437855d024}"
2025-07-05T01:11:47.330Z [INFO]  server-6733f73e-6788-d5e5-a1ba-c5447f9ea7e7: aborting pipeline replication: peer="{Voter server-f7a1ffe0-f99a-87fe-4334-bda63919d1a4 f7a1ffe0-f99a-87fe-4334-bda63919d1a4}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-07-05T01:11:47.334Z [INFO]  server-21204b5a-5140-d47b-cece-020fde396091: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21204b5a-5140-d47b-cece-020fde396091 Address:21204b5a-5140-d47b-cece-020fde396091} {Suffrage:Voter ID:server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac Address:eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac} {Suffrage:Voter ID:server-75965c00-8a58-8106-1e50-b4fb57be6359 Address:75965c00-8a58-8106-1e50-b4fb57be6359}]"
2025-07-05T01:11:47.334Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21204b5a-5140-d47b-cece-020fde396091 Address:21204b5a-5140-d47b-cece-020fde396091} {Suffrage:Voter ID:server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac Address:eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac} {Suffrage:Voter ID:server-75965c00-8a58-8106-1e50-b4fb57be6359 Address:75965c00-8a58-8106-1e50-b4fb57be6359}]"
2025-07-05T01:11:47.334Z [INFO]  server-75965c00-8a58-8106-1e50-b4fb57be6359: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21204b5a-5140-d47b-cece-020fde396091 Address:21204b5a-5140-d47b-cece-020fde396091} {Suffrage:Voter ID:server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac Address:eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac} {Suffrage:Voter ID:server-75965c00-8a58-8106-1e50-b4fb57be6359 Address:75965c00-8a58-8106-1e50-b4fb57be6359}]"
2025-07-05T01:11:47.334Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:47.334Z [INFO]  server-21204b5a-5140-d47b-cece-020fde396091: entering follower state: follower="Node at 21204b5a-5140-d47b-cece-020fde396091 [Follower]" leader-address= leader-id=
2025-07-05T01:11:47.334Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: entering follower state: follower="Node at eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac [Follower]" leader-address= leader-id=
2025-07-05T01:11:47.334Z [INFO]  server-75965c00-8a58-8106-1e50-b4fb57be6359: entering follower state: follower="Node at 75965c00-8a58-8106-1e50-b4fb57be6359 [Follower]" leader-address= leader-id=
2025-07-05T01:11:47.977Z [WARN]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:47.977Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: entering candidate state: node="Node at eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac [Candidate]" term=2
2025-07-05T01:11:47.977Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: election won: term=2 tally=2
2025-07-05T01:11:47.977Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: entering leader state: leader="Node at eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac [Leader]"
2025-07-05T01:11:47.977Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: added peer, starting replication: peer=server-21204b5a-5140-d47b-cece-020fde396091
2025-07-05T01:11:47.977Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: added peer, starting replication: peer=server-75965c00-8a58-8106-1e50-b4fb57be6359
2025-07-05T01:11:47.977Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: pipelining replication: peer="{Voter server-75965c00-8a58-8106-1e50-b4fb57be6359 75965c00-8a58-8106-1e50-b4fb57be6359}"
2025-07-05T01:11:47.977Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: pipelining replication: peer="{Voter server-21204b5a-5140-d47b-cece-020fde396091 21204b5a-5140-d47b-cece-020fde396091}"
2025-07-05T01:11:47.977Z [WARN]  server-21204b5a-5140-d47b-cece-020fde396091: rejecting vote request since our last term is greater: candidate=eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac last-term=2 last-candidate-term=1
2025-07-05T01:11:48.983Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:47.977869861 +0000 UTC m=+83.226607079 (1 nodes), 643.101915ms from start of poll, 643.404052ms from cluster start. Timeout at 2025-07-05 01:11:48.983903654 +0000 UTC m=+84.232640872, 1.006033793s after stability
2025-07-05T01:11:48.988Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: starting snapshot up to: index=12
2025-07-05T01:11:48.988Z [INFO]  creating new snapshot: path=/tmp/raft2822665526/snapshots/2-12-1751677908988.tmp
2025-07-05T01:11:48.988Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: snapshot complete up to: index=12
2025-07-05T01:11:48.996Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e Address:74fe82f1-6c33-f5ee-f7a0-2bd1556b955e} {Suffrage:Voter ID:server-9a07f265-adb2-473c-3afb-680415d97618 Address:9a07f265-adb2-473c-3afb-680415d97618} {Suffrage:Voter ID:server-0e38e340-7d80-2897-75e9-c3c79bccca77 Address:0e38e340-7d80-2897-75e9-c3c79bccca77}]"
2025-07-05T01:11:48.996Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: entering follower state: follower="Node at 74fe82f1-6c33-f5ee-f7a0-2bd1556b955e [Follower]" leader-address= leader-id=
2025-07-05T01:11:48.996Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e Address:74fe82f1-6c33-f5ee-f7a0-2bd1556b955e} {Suffrage:Voter ID:server-9a07f265-adb2-473c-3afb-680415d97618 Address:9a07f265-adb2-473c-3afb-680415d97618} {Suffrage:Voter ID:server-0e38e340-7d80-2897-75e9-c3c79bccca77 Address:0e38e340-7d80-2897-75e9-c3c79bccca77}]"
2025-07-05T01:11:48.997Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e Address:74fe82f1-6c33-f5ee-f7a0-2bd1556b955e} {Suffrage:Voter ID:server-9a07f265-adb2-473c-3afb-680415d97618 Address:9a07f265-adb2-473c-3afb-680415d97618} {Suffrage:Voter ID:server-0e38e340-7d80-2897-75e9-c3c79bccca77 Address:0e38e340-7d80-2897-75e9-c3c79bccca77}]"
2025-07-05T01:11:48.997Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:48.997Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: entering follower state: follower="Node at 9a07f265-adb2-473c-3afb-680415d97618 [Follower]" leader-address= leader-id=
2025-07-05T01:11:48.997Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: entering follower state: follower="Node at 0e38e340-7d80-2897-75e9-c3c79bccca77 [Follower]" leader-address= leader-id=
2025-07-05T01:11:49.721Z [WARN]  server-9a07f265-adb2-473c-3afb-680415d97618: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:49.721Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: entering candidate state: node="Node at 9a07f265-adb2-473c-3afb-680415d97618 [Candidate]" term=2
2025-07-05T01:11:49.721Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: election won: term=2 tally=2
2025-07-05T01:11:49.721Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: entering leader state: leader="Node at 9a07f265-adb2-473c-3afb-680415d97618 [Leader]"
2025-07-05T01:11:49.721Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: added peer, starting replication: peer=server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e
2025-07-05T01:11:49.721Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: added peer, starting replication: peer=server-0e38e340-7d80-2897-75e9-c3c79bccca77
2025-07-05T01:11:49.721Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: pipelining replication: peer="{Voter server-0e38e340-7d80-2897-75e9-c3c79bccca77 0e38e340-7d80-2897-75e9-c3c79bccca77}"
2025-07-05T01:11:49.721Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: pipelining replication: peer="{Voter server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e 74fe82f1-6c33-f5ee-f7a0-2bd1556b955e}"
2025-07-05T01:11:50.727Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:49.721559809 +0000 UTC m=+84.970297027 (1 nodes), 724.441888ms from start of poll, 724.974066ms from cluster start. Timeout at 2025-07-05 01:11:50.727466494 +0000 UTC m=+85.976203702, 1.005906675s after stability
2025-07-05T01:11:50.727Z [INFO]  creating new snapshot: path=/tmp/raft2778479238/snapshots/2-1013-1751677910727.tmp
2025-07-05T01:11:50.728Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: copied to local snapshot: bytes=71
2025-07-05T01:11:50.728Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: snapshot restore progress: id=2-1013-1751677910727 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:50.728Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: removing all old logs from log store
2025-07-05T01:11:50.728Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: compacting logs: from=1 to=2
2025-07-05T01:11:50.728Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: restored user snapshot: index=1013
2025-07-05T01:11:50.728Z [ERROR] server-9a07f265-adb2-473c-3afb-680415d97618: failed to get log: index=2 error="log not found"
2025-07-05T01:11:50.728Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: aborting pipeline replication: peer="{Voter server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e 74fe82f1-6c33-f5ee-f7a0-2bd1556b955e}"
2025-07-05T01:11:50.728Z [ERROR] server-9a07f265-adb2-473c-3afb-680415d97618: failed to get log: index=2 error="log not found"
2025-07-05T01:11:50.728Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: aborting pipeline replication: peer="{Voter server-0e38e340-7d80-2897-75e9-c3c79bccca77 0e38e340-7d80-2897-75e9-c3c79bccca77}"
2025-07-05T01:11:50.739Z [ERROR] server-9a07f265-adb2-473c-3afb-680415d97618: failed to get log: index=2 error="log not found"
2025-07-05T01:11:50.739Z [INFO]  creating new snapshot: path=/tmp/raft102198389/snapshots/2-1013-1751677910739.tmp
2025-07-05T01:11:50.739Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:50.739Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: copied to local snapshot: bytes=71
2025-07-05T01:11:50.739Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: snapshot restore progress: id=2-1013-1751677910739 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:50.739Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: removing all old logs from log store
2025-07-05T01:11:50.739Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: compacting logs: from=1 to=2
2025-07-05T01:11:50.739Z [INFO]  server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e: Installed remote snapshot
2025-07-05T01:11:50.739Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: pipelining replication: peer="{Voter server-74fe82f1-6c33-f5ee-f7a0-2bd1556b955e 74fe82f1-6c33-f5ee-f7a0-2bd1556b955e}"
2025-07-05T01:11:50.739Z [ERROR] server-9a07f265-adb2-473c-3afb-680415d97618: failed to get log: index=2 error="log not found"
2025-07-05T01:11:50.740Z [INFO]  creating new snapshot: path=/tmp/raft3989615803/snapshots/2-1013-1751677910740.tmp
2025-07-05T01:11:50.740Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:50.740Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: copied to local snapshot: bytes=71
2025-07-05T01:11:50.740Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: snapshot restore progress: id=2-1013-1751677910740 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:50.740Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: removing all old logs from log store
2025-07-05T01:11:50.740Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: compacting logs: from=1 to=2
2025-07-05T01:11:50.740Z [INFO]  server-0e38e340-7d80-2897-75e9-c3c79bccca77: Installed remote snapshot
2025-07-05T01:11:50.740Z [INFO]  server-9a07f265-adb2-473c-3afb-680415d97618: pipelining replication: peer="{Voter server-0e38e340-7d80-2897-75e9-c3c79bccca77 0e38e340-7d80-2897-75e9-c3c79bccca77}"
2025-07-05T01:11:50.740Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:51.751Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:50.740765159 +0000 UTC m=+85.989502367 (2 nodes), 681ns from start of poll, 1.744179406s from cluster start. Timeout at 2025-07-05 01:11:51.751071493 +0000 UTC m=+86.999808701, 1.010306334s after stability
2025-07-05T01:11:51.767Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: aborting pipeline replication: peer="{Voter server-75965c00-8a58-8106-1e50-b4fb57be6359 75965c00-8a58-8106-1e50-b4fb57be6359}"
2025-07-05T01:11:51.767Z [INFO]  server-eb22d6e8-b088-c2a2-e55d-7d40ac60d2ac: aborting pipeline replication: peer="{Voter server-21204b5a-5140-d47b-cece-020fde396091 21204b5a-5140-d47b-cece-020fde396091}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-07-05T01:11:51.768Z [INFO]  server-8c0a1432-22b5-369a-198e-252251ece905: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c0a1432-22b5-369a-198e-252251ece905 Address:8c0a1432-22b5-369a-198e-252251ece905} {Suffrage:Voter ID:server-447e6e01-9607-1483-e4a8-4459872e5f10 Address:447e6e01-9607-1483-e4a8-4459872e5f10} {Suffrage:Voter ID:server-652828fc-e78b-8319-9cde-396a16c61452 Address:652828fc-e78b-8319-9cde-396a16c61452}]"
2025-07-05T01:11:51.768Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c0a1432-22b5-369a-198e-252251ece905 Address:8c0a1432-22b5-369a-198e-252251ece905} {Suffrage:Voter ID:server-447e6e01-9607-1483-e4a8-4459872e5f10 Address:447e6e01-9607-1483-e4a8-4459872e5f10} {Suffrage:Voter ID:server-652828fc-e78b-8319-9cde-396a16c61452 Address:652828fc-e78b-8319-9cde-396a16c61452}]"
2025-07-05T01:11:51.768Z [INFO]  server-652828fc-e78b-8319-9cde-396a16c61452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c0a1432-22b5-369a-198e-252251ece905 Address:8c0a1432-22b5-369a-198e-252251ece905} {Suffrage:Voter ID:server-447e6e01-9607-1483-e4a8-4459872e5f10 Address:447e6e01-9607-1483-e4a8-4459872e5f10} {Suffrage:Voter ID:server-652828fc-e78b-8319-9cde-396a16c61452 Address:652828fc-e78b-8319-9cde-396a16c61452}]"
2025-07-05T01:11:51.768Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:51.768Z [INFO]  server-8c0a1432-22b5-369a-198e-252251ece905: entering follower state: follower="Node at 8c0a1432-22b5-369a-198e-252251ece905 [Follower]" leader-address= leader-id=
2025-07-05T01:11:51.768Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: entering follower state: follower="Node at 447e6e01-9607-1483-e4a8-4459872e5f10 [Follower]" leader-address= leader-id=
2025-07-05T01:11:51.768Z [INFO]  server-652828fc-e78b-8319-9cde-396a16c61452: entering follower state: follower="Node at 652828fc-e78b-8319-9cde-396a16c61452 [Follower]" leader-address= leader-id=
2025-07-05T01:11:52.505Z [WARN]  server-447e6e01-9607-1483-e4a8-4459872e5f10: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:52.505Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: entering candidate state: node="Node at 447e6e01-9607-1483-e4a8-4459872e5f10 [Candidate]" term=2
2025-07-05T01:11:52.505Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: election won: term=2 tally=2
2025-07-05T01:11:52.505Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: entering leader state: leader="Node at 447e6e01-9607-1483-e4a8-4459872e5f10 [Leader]"
2025-07-05T01:11:52.505Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: added peer, starting replication: peer=server-8c0a1432-22b5-369a-198e-252251ece905
2025-07-05T01:11:52.505Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: added peer, starting replication: peer=server-652828fc-e78b-8319-9cde-396a16c61452
2025-07-05T01:11:52.505Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: pipelining replication: peer="{Voter server-8c0a1432-22b5-369a-198e-252251ece905 8c0a1432-22b5-369a-198e-252251ece905}"
2025-07-05T01:11:52.505Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: pipelining replication: peer="{Voter server-652828fc-e78b-8319-9cde-396a16c61452 652828fc-e78b-8319-9cde-396a16c61452}"
2025-07-05T01:11:53.511Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:52.505734296 +0000 UTC m=+87.754471514 (1 nodes), 737.048546ms from start of poll, 737.259602ms from cluster start. Timeout at 2025-07-05 01:11:53.511501609 +0000 UTC m=+88.760238817, 1.005767303s after stability
2025-07-05T01:11:53.511Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: starting snapshot up to: index=12
2025-07-05T01:11:53.512Z [INFO]  creating new snapshot: path=/tmp/raft1907921977/snapshots/2-12-1751677913512.tmp
2025-07-05T01:11:53.512Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: snapshot complete up to: index=12
2025-07-05T01:11:53.518Z [INFO]  creating new snapshot: path=/tmp/raft1907921977/snapshots/2-10013-1751677913518.tmp
2025-07-05T01:11:53.518Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: copied to local snapshot: bytes=71
2025-07-05T01:11:53.519Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: snapshot restore progress: id=2-10013-1751677913518 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:53.519Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: restored user snapshot: index=10013
2025-07-05T01:11:53.519Z [ERROR] server-447e6e01-9607-1483-e4a8-4459872e5f10: failed to get log: index=23 error="log not found"
2025-07-05T01:11:53.519Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: aborting pipeline replication: peer="{Voter server-652828fc-e78b-8319-9cde-396a16c61452 652828fc-e78b-8319-9cde-396a16c61452}"
2025-07-05T01:11:53.519Z [ERROR] server-447e6e01-9607-1483-e4a8-4459872e5f10: failed to get log: index=23 error="log not found"
2025-07-05T01:11:53.519Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: aborting pipeline replication: peer="{Voter server-8c0a1432-22b5-369a-198e-252251ece905 8c0a1432-22b5-369a-198e-252251ece905}"
2025-07-05T01:11:53.526Z [ERROR] server-447e6e01-9607-1483-e4a8-4459872e5f10: failed to get log: index=23 error="log not found"
2025-07-05T01:11:53.527Z [INFO]  creating new snapshot: path=/tmp/raft1263413107/snapshots/2-10013-1751677913527.tmp
2025-07-05T01:11:53.527Z [INFO]  server-652828fc-e78b-8319-9cde-396a16c61452: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:53.527Z [INFO]  server-652828fc-e78b-8319-9cde-396a16c61452: copied to local snapshot: bytes=71
2025-07-05T01:11:53.527Z [INFO]  server-652828fc-e78b-8319-9cde-396a16c61452: snapshot restore progress: id=2-10013-1751677913527 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:53.527Z [INFO]  server-652828fc-e78b-8319-9cde-396a16c61452: Installed remote snapshot
2025-07-05T01:11:53.527Z [WARN]  server-652828fc-e78b-8319-9cde-396a16c61452: skipping application of old log: index=22
2025-07-05T01:11:53.527Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: pipelining replication: peer="{Voter server-652828fc-e78b-8319-9cde-396a16c61452 652828fc-e78b-8319-9cde-396a16c61452}"
2025-07-05T01:11:53.527Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:53.528Z [ERROR] server-447e6e01-9607-1483-e4a8-4459872e5f10: failed to get log: index=23 error="log not found"
2025-07-05T01:11:53.529Z [INFO]  creating new snapshot: path=/tmp/raft1584414464/snapshots/2-10013-1751677913529.tmp
2025-07-05T01:11:53.529Z [INFO]  server-8c0a1432-22b5-369a-198e-252251ece905: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:53.529Z [INFO]  server-8c0a1432-22b5-369a-198e-252251ece905: copied to local snapshot: bytes=71
2025-07-05T01:11:53.529Z [INFO]  server-8c0a1432-22b5-369a-198e-252251ece905: snapshot restore progress: id=2-10013-1751677913529 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:53.529Z [INFO]  server-8c0a1432-22b5-369a-198e-252251ece905: Installed remote snapshot
2025-07-05T01:11:53.529Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: pipelining replication: peer="{Voter server-8c0a1432-22b5-369a-198e-252251ece905 8c0a1432-22b5-369a-198e-252251ece905}"
2025-07-05T01:11:54.533Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:53.527672249 +0000 UTC m=+88.776409457 (2 nodes), 391ns from start of poll, 1.759197545s from cluster start. Timeout at 2025-07-05 01:11:54.533121255 +0000 UTC m=+89.781858463, 1.005449006s after stability
2025-07-05T01:11:54.545Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: aborting pipeline replication: peer="{Voter server-652828fc-e78b-8319-9cde-396a16c61452 652828fc-e78b-8319-9cde-396a16c61452}"
2025-07-05T01:11:54.545Z [INFO]  server-447e6e01-9607-1483-e4a8-4459872e5f10: aborting pipeline replication: peer="{Voter server-8c0a1432-22b5-369a-198e-252251ece905 8c0a1432-22b5-369a-198e-252251ece905}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-07-05T01:11:54.546Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26 Address:f5d9b88d-c1ff-9cdc-f283-d600ab948b26} {Suffrage:Voter ID:server-a8a1b863-5e99-443b-4100-536f108fd302 Address:a8a1b863-5e99-443b-4100-536f108fd302} {Suffrage:Voter ID:server-25938c6a-d0a5-318d-51c7-d59f6eab7744 Address:25938c6a-d0a5-318d-51c7-d59f6eab7744}]"
2025-07-05T01:11:54.546Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26 Address:f5d9b88d-c1ff-9cdc-f283-d600ab948b26} {Suffrage:Voter ID:server-a8a1b863-5e99-443b-4100-536f108fd302 Address:a8a1b863-5e99-443b-4100-536f108fd302} {Suffrage:Voter ID:server-25938c6a-d0a5-318d-51c7-d59f6eab7744 Address:25938c6a-d0a5-318d-51c7-d59f6eab7744}]"
2025-07-05T01:11:54.547Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26 Address:f5d9b88d-c1ff-9cdc-f283-d600ab948b26} {Suffrage:Voter ID:server-a8a1b863-5e99-443b-4100-536f108fd302 Address:a8a1b863-5e99-443b-4100-536f108fd302} {Suffrage:Voter ID:server-25938c6a-d0a5-318d-51c7-d59f6eab7744 Address:25938c6a-d0a5-318d-51c7-d59f6eab7744}]"
2025-07-05T01:11:54.547Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:54.547Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: entering follower state: follower="Node at f5d9b88d-c1ff-9cdc-f283-d600ab948b26 [Follower]" leader-address= leader-id=
2025-07-05T01:11:54.547Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: entering follower state: follower="Node at a8a1b863-5e99-443b-4100-536f108fd302 [Follower]" leader-address= leader-id=
2025-07-05T01:11:54.547Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: entering follower state: follower="Node at 25938c6a-d0a5-318d-51c7-d59f6eab7744 [Follower]" leader-address= leader-id=
2025-07-05T01:11:55.053Z [WARN]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:55.053Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: entering candidate state: node="Node at 25938c6a-d0a5-318d-51c7-d59f6eab7744 [Candidate]" term=2
2025-07-05T01:11:55.053Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: election won: term=2 tally=2
2025-07-05T01:11:55.053Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: entering leader state: leader="Node at 25938c6a-d0a5-318d-51c7-d59f6eab7744 [Leader]"
2025-07-05T01:11:55.053Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: added peer, starting replication: peer=server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26
2025-07-05T01:11:55.053Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: added peer, starting replication: peer=server-a8a1b863-5e99-443b-4100-536f108fd302
2025-07-05T01:11:55.053Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: pipelining replication: peer="{Voter server-a8a1b863-5e99-443b-4100-536f108fd302 a8a1b863-5e99-443b-4100-536f108fd302}"
2025-07-05T01:11:55.057Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: pipelining replication: peer="{Voter server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26 f5d9b88d-c1ff-9cdc-f283-d600ab948b26}"
2025-07-05T01:11:56.063Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:55.057212576 +0000 UTC m=+90.305949795 (1 nodes), 510.150422ms from start of poll, 510.417834ms from cluster start. Timeout at 2025-07-05 01:11:56.063151762 +0000 UTC m=+91.311888970, 1.005939175s after stability
2025-07-05T01:11:56.063Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: starting snapshot up to: index=12
2025-07-05T01:11:56.063Z [INFO]  creating new snapshot: path=/tmp/raft3536075960/snapshots/2-12-1751677916063.tmp
2025-07-05T01:11:56.067Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: snapshot complete up to: index=12
2025-07-05T01:11:56.071Z [INFO]  creating new snapshot: path=/tmp/raft3536075960/snapshots/2-10013-1751677916071.tmp
2025-07-05T01:11:56.073Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: copied to local snapshot: bytes=71
2025-07-05T01:11:56.074Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: snapshot restore progress: id=2-10013-1751677916071 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:56.074Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: removing all old logs from log store
2025-07-05T01:11:56.074Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: compacting logs: from=1 to=22
2025-07-05T01:11:56.074Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: restored user snapshot: index=10013
2025-07-05T01:11:56.074Z [ERROR] server-25938c6a-d0a5-318d-51c7-d59f6eab7744: failed to get log: index=22 error="log not found"
2025-07-05T01:11:56.074Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: aborting pipeline replication: peer="{Voter server-a8a1b863-5e99-443b-4100-536f108fd302 a8a1b863-5e99-443b-4100-536f108fd302}"
2025-07-05T01:11:56.074Z [ERROR] server-25938c6a-d0a5-318d-51c7-d59f6eab7744: failed to get log: index=22 error="log not found"
2025-07-05T01:11:56.074Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: aborting pipeline replication: peer="{Voter server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26 f5d9b88d-c1ff-9cdc-f283-d600ab948b26}"
2025-07-05T01:11:56.080Z [ERROR] server-25938c6a-d0a5-318d-51c7-d59f6eab7744: failed to get log: index=22 error="log not found"
2025-07-05T01:11:56.080Z [INFO]  creating new snapshot: path=/tmp/raft1045734538/snapshots/2-10013-1751677916080.tmp
2025-07-05T01:11:56.080Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:56.081Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: copied to local snapshot: bytes=71
2025-07-05T01:11:56.081Z [ERROR] server-25938c6a-d0a5-318d-51c7-d59f6eab7744: failed to get log: index=22 error="log not found"
2025-07-05T01:11:56.082Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: snapshot restore progress: id=2-10013-1751677916080 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:56.082Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: removing all old logs from log store
2025-07-05T01:11:56.082Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: compacting logs: from=1 to=22
2025-07-05T01:11:56.082Z [INFO]  server-a8a1b863-5e99-443b-4100-536f108fd302: Installed remote snapshot
2025-07-05T01:11:56.082Z [WARN]  server-a8a1b863-5e99-443b-4100-536f108fd302: skipping application of old log: index=22
2025-07-05T01:11:56.082Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: pipelining replication: peer="{Voter server-a8a1b863-5e99-443b-4100-536f108fd302 a8a1b863-5e99-443b-4100-536f108fd302}"
2025-07-05T01:11:56.082Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:11:56.082Z [INFO]  creating new snapshot: path=/tmp/raft1050331756/snapshots/2-10013-1751677916082.tmp
2025-07-05T01:11:56.082Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:56.083Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: copied to local snapshot: bytes=71
2025-07-05T01:11:56.083Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: snapshot restore progress: id=2-10013-1751677916082 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:11:56.083Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: removing all old logs from log store
2025-07-05T01:11:56.083Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: compacting logs: from=1 to=22
2025-07-05T01:11:56.083Z [INFO]  server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26: Installed remote snapshot
2025-07-05T01:11:56.083Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: pipelining replication: peer="{Voter server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26 f5d9b88d-c1ff-9cdc-f283-d600ab948b26}"
2025-07-05T01:11:57.088Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:11:56.082639552 +0000 UTC m=+91.331376760 (2 nodes), 952ns from start of poll, 1.535844799s from cluster start. Timeout at 2025-07-05 01:11:57.088144002 +0000 UTC m=+92.336881210, 1.00550445s after stability
2025-07-05T01:11:57.103Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: aborting pipeline replication: peer="{Voter server-a8a1b863-5e99-443b-4100-536f108fd302 a8a1b863-5e99-443b-4100-536f108fd302}"
2025-07-05T01:11:57.103Z [INFO]  server-25938c6a-d0a5-318d-51c7-d59f6eab7744: aborting pipeline replication: peer="{Voter server-f5d9b88d-c1ff-9cdc-f283-d600ab948b26 f5d9b88d-c1ff-9cdc-f283-d600ab948b26}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-07-05T01:11:57.104Z [INFO]  server-1c36887c-91b6-b767-24a4-48e2240d2123: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c36887c-91b6-b767-24a4-48e2240d2123 Address:1c36887c-91b6-b767-24a4-48e2240d2123} {Suffrage:Voter ID:server-1bb24478-9b63-3bb3-09b4-d6d867401f9b Address:1bb24478-9b63-3bb3-09b4-d6d867401f9b} {Suffrage:Voter ID:server-c1bade13-b7f8-be78-45ec-b7de70e3b047 Address:c1bade13-b7f8-be78-45ec-b7de70e3b047}]"
2025-07-05T01:11:57.104Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c36887c-91b6-b767-24a4-48e2240d2123 Address:1c36887c-91b6-b767-24a4-48e2240d2123} {Suffrage:Voter ID:server-1bb24478-9b63-3bb3-09b4-d6d867401f9b Address:1bb24478-9b63-3bb3-09b4-d6d867401f9b} {Suffrage:Voter ID:server-c1bade13-b7f8-be78-45ec-b7de70e3b047 Address:c1bade13-b7f8-be78-45ec-b7de70e3b047}]"
2025-07-05T01:11:57.104Z [INFO]  server-c1bade13-b7f8-be78-45ec-b7de70e3b047: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c36887c-91b6-b767-24a4-48e2240d2123 Address:1c36887c-91b6-b767-24a4-48e2240d2123} {Suffrage:Voter ID:server-1bb24478-9b63-3bb3-09b4-d6d867401f9b Address:1bb24478-9b63-3bb3-09b4-d6d867401f9b} {Suffrage:Voter ID:server-c1bade13-b7f8-be78-45ec-b7de70e3b047 Address:c1bade13-b7f8-be78-45ec-b7de70e3b047}]"
2025-07-05T01:11:57.104Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:57.104Z [INFO]  server-1c36887c-91b6-b767-24a4-48e2240d2123: entering follower state: follower="Node at 1c36887c-91b6-b767-24a4-48e2240d2123 [Follower]" leader-address= leader-id=
2025-07-05T01:11:57.104Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: entering follower state: follower="Node at 1bb24478-9b63-3bb3-09b4-d6d867401f9b [Follower]" leader-address= leader-id=
2025-07-05T01:11:57.104Z [INFO]  server-c1bade13-b7f8-be78-45ec-b7de70e3b047: entering follower state: follower="Node at c1bade13-b7f8-be78-45ec-b7de70e3b047 [Follower]" leader-address= leader-id=
2025-07-05T01:11:57.718Z [WARN]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:57.718Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: entering candidate state: node="Node at 1bb24478-9b63-3bb3-09b4-d6d867401f9b [Candidate]" term=2
2025-07-05T01:11:57.718Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: election won: term=2 tally=2
2025-07-05T01:11:57.718Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: entering leader state: leader="Node at 1bb24478-9b63-3bb3-09b4-d6d867401f9b [Leader]"
2025-07-05T01:11:57.718Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: added peer, starting replication: peer=server-1c36887c-91b6-b767-24a4-48e2240d2123
2025-07-05T01:11:57.718Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: added peer, starting replication: peer=server-c1bade13-b7f8-be78-45ec-b7de70e3b047
2025-07-05T01:11:57.718Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: pipelining replication: peer="{Voter server-c1bade13-b7f8-be78-45ec-b7de70e3b047 c1bade13-b7f8-be78-45ec-b7de70e3b047}"
2025-07-05T01:11:57.718Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: pipelining replication: peer="{Voter server-1c36887c-91b6-b767-24a4-48e2240d2123 1c36887c-91b6-b767-24a4-48e2240d2123}"
2025-07-05T01:11:58.727Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:57.718465575 +0000 UTC m=+92.967202783 (1 nodes), 613.757635ms from start of poll, 613.949335ms from cluster start. Timeout at 2025-07-05 01:11:58.727562621 +0000 UTC m=+93.976299840, 1.009097057s after stability
2025-07-05T01:11:58.728Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: starting snapshot up to: index=12
2025-07-05T01:11:58.728Z [INFO]  creating new snapshot: path=/tmp/raft824050718/snapshots/2-12-1751677918728.tmp
2025-07-05T01:11:58.729Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: snapshot complete up to: index=12
2025-07-05T01:11:58.743Z [INFO]  server-f273cf70-c230-937b-ebaa-b3045ab138c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f273cf70-c230-937b-ebaa-b3045ab138c0 Address:f273cf70-c230-937b-ebaa-b3045ab138c0} {Suffrage:Voter ID:server-d4965292-e637-0918-b0ff-4da90cd82f15 Address:d4965292-e637-0918-b0ff-4da90cd82f15} {Suffrage:Voter ID:server-130b1fb0-0419-e53a-376b-8cd92823433c Address:130b1fb0-0419-e53a-376b-8cd92823433c}]"
2025-07-05T01:11:58.743Z [INFO]  server-d4965292-e637-0918-b0ff-4da90cd82f15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f273cf70-c230-937b-ebaa-b3045ab138c0 Address:f273cf70-c230-937b-ebaa-b3045ab138c0} {Suffrage:Voter ID:server-d4965292-e637-0918-b0ff-4da90cd82f15 Address:d4965292-e637-0918-b0ff-4da90cd82f15} {Suffrage:Voter ID:server-130b1fb0-0419-e53a-376b-8cd92823433c Address:130b1fb0-0419-e53a-376b-8cd92823433c}]"
2025-07-05T01:11:58.743Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f273cf70-c230-937b-ebaa-b3045ab138c0 Address:f273cf70-c230-937b-ebaa-b3045ab138c0} {Suffrage:Voter ID:server-d4965292-e637-0918-b0ff-4da90cd82f15 Address:d4965292-e637-0918-b0ff-4da90cd82f15} {Suffrage:Voter ID:server-130b1fb0-0419-e53a-376b-8cd92823433c Address:130b1fb0-0419-e53a-376b-8cd92823433c}]"
2025-07-05T01:11:58.743Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:11:58.743Z [INFO]  server-f273cf70-c230-937b-ebaa-b3045ab138c0: entering follower state: follower="Node at f273cf70-c230-937b-ebaa-b3045ab138c0 [Follower]" leader-address= leader-id=
2025-07-05T01:11:58.743Z [INFO]  server-d4965292-e637-0918-b0ff-4da90cd82f15: entering follower state: follower="Node at d4965292-e637-0918-b0ff-4da90cd82f15 [Follower]" leader-address= leader-id=
2025-07-05T01:11:58.743Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: entering follower state: follower="Node at 130b1fb0-0419-e53a-376b-8cd92823433c [Follower]" leader-address= leader-id=
2025-07-05T01:11:59.246Z [WARN]  server-130b1fb0-0419-e53a-376b-8cd92823433c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:11:59.246Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: entering candidate state: node="Node at 130b1fb0-0419-e53a-376b-8cd92823433c [Candidate]" term=2
2025-07-05T01:11:59.246Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: election won: term=2 tally=2
2025-07-05T01:11:59.246Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: entering leader state: leader="Node at 130b1fb0-0419-e53a-376b-8cd92823433c [Leader]"
2025-07-05T01:11:59.246Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: added peer, starting replication: peer=server-f273cf70-c230-937b-ebaa-b3045ab138c0
2025-07-05T01:11:59.246Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: added peer, starting replication: peer=server-d4965292-e637-0918-b0ff-4da90cd82f15
2025-07-05T01:11:59.246Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: pipelining replication: peer="{Voter server-d4965292-e637-0918-b0ff-4da90cd82f15 d4965292-e637-0918-b0ff-4da90cd82f15}"
2025-07-05T01:11:59.246Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: pipelining replication: peer="{Voter server-f273cf70-c230-937b-ebaa-b3045ab138c0 f273cf70-c230-937b-ebaa-b3045ab138c0}"
2025-07-05T01:12:00.251Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:11:59.246515528 +0000 UTC m=+94.495252746 (1 nodes), 502.7466ms from start of poll, 503.147812ms from cluster start. Timeout at 2025-07-05 01:12:00.251683668 +0000 UTC m=+95.500420886, 1.00516814s after stability
2025-07-05T01:12:00.251Z [INFO]  creating new snapshot: path=/tmp/raft376020549/snapshots/2-10013-1751677920251.tmp
2025-07-05T01:12:00.252Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: copied to local snapshot: bytes=71
2025-07-05T01:12:00.252Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: snapshot restore progress: id=2-10013-1751677920251 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:00.252Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: restored user snapshot: index=10013
2025-07-05T01:12:00.252Z [ERROR] server-130b1fb0-0419-e53a-376b-8cd92823433c: failed to get log: index=3 error="log not found"
2025-07-05T01:12:00.252Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: aborting pipeline replication: peer="{Voter server-f273cf70-c230-937b-ebaa-b3045ab138c0 f273cf70-c230-937b-ebaa-b3045ab138c0}"
2025-07-05T01:12:00.252Z [ERROR] server-130b1fb0-0419-e53a-376b-8cd92823433c: failed to get log: index=3 error="log not found"
2025-07-05T01:12:00.252Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: aborting pipeline replication: peer="{Voter server-d4965292-e637-0918-b0ff-4da90cd82f15 d4965292-e637-0918-b0ff-4da90cd82f15}"
2025-07-05T01:12:00.262Z [ERROR] server-130b1fb0-0419-e53a-376b-8cd92823433c: failed to get log: index=3 error="log not found"
2025-07-05T01:12:00.263Z [INFO]  creating new snapshot: path=/tmp/raft1794461458/snapshots/2-10013-1751677920263.tmp
2025-07-05T01:12:00.263Z [ERROR] server-130b1fb0-0419-e53a-376b-8cd92823433c: failed to get log: index=3 error="log not found"
2025-07-05T01:12:00.263Z [INFO]  creating new snapshot: path=/tmp/raft1827280846/snapshots/2-10013-1751677920263.tmp
2025-07-05T01:12:00.263Z [INFO]  server-d4965292-e637-0918-b0ff-4da90cd82f15: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:00.264Z [INFO]  server-d4965292-e637-0918-b0ff-4da90cd82f15: copied to local snapshot: bytes=71
2025-07-05T01:12:00.264Z [INFO]  server-f273cf70-c230-937b-ebaa-b3045ab138c0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:00.264Z [INFO]  server-f273cf70-c230-937b-ebaa-b3045ab138c0: copied to local snapshot: bytes=71
2025-07-05T01:12:00.264Z [INFO]  server-f273cf70-c230-937b-ebaa-b3045ab138c0: snapshot restore progress: id=2-10013-1751677920263 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:00.264Z [INFO]  server-f273cf70-c230-937b-ebaa-b3045ab138c0: Installed remote snapshot
2025-07-05T01:12:00.264Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: pipelining replication: peer="{Voter server-f273cf70-c230-937b-ebaa-b3045ab138c0 f273cf70-c230-937b-ebaa-b3045ab138c0}"
2025-07-05T01:12:00.264Z [INFO]  server-d4965292-e637-0918-b0ff-4da90cd82f15: snapshot restore progress: id=2-10013-1751677920263 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:00.264Z [INFO]  server-d4965292-e637-0918-b0ff-4da90cd82f15: Installed remote snapshot
2025-07-05T01:12:00.264Z [INFO]  server-130b1fb0-0419-e53a-376b-8cd92823433c: pipelining replication: peer="{Voter server-d4965292-e637-0918-b0ff-4da90cd82f15 d4965292-e637-0918-b0ff-4da90cd82f15}"
2025-07-05T01:12:00.264Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:01.271Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:00.264843563 +0000 UTC m=+95.513580781 (2 nodes), 942ns from start of poll, 1.521475847s from cluster start. Timeout at 2025-07-05 01:12:01.270967535 +0000 UTC m=+96.519704743, 1.006123962s after stability
2025-07-05T01:12:01.287Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: aborting pipeline replication: peer="{Voter server-c1bade13-b7f8-be78-45ec-b7de70e3b047 c1bade13-b7f8-be78-45ec-b7de70e3b047}"
2025-07-05T01:12:01.287Z [INFO]  server-1bb24478-9b63-3bb3-09b4-d6d867401f9b: aborting pipeline replication: peer="{Voter server-1c36887c-91b6-b767-24a4-48e2240d2123 1c36887c-91b6-b767-24a4-48e2240d2123}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-07-05T01:12:01.288Z [INFO]  server-c8d91f20-ff9c-358c-4c6e-5603842c43b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8d91f20-ff9c-358c-4c6e-5603842c43b5 Address:c8d91f20-ff9c-358c-4c6e-5603842c43b5} {Suffrage:Voter ID:server-4d838492-b948-40da-f8f3-16b2646f3917 Address:4d838492-b948-40da-f8f3-16b2646f3917} {Suffrage:Voter ID:server-95d29895-c167-116b-dae3-ca6760f2c424 Address:95d29895-c167-116b-dae3-ca6760f2c424}]"
2025-07-05T01:12:01.288Z [INFO]  server-4d838492-b948-40da-f8f3-16b2646f3917: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8d91f20-ff9c-358c-4c6e-5603842c43b5 Address:c8d91f20-ff9c-358c-4c6e-5603842c43b5} {Suffrage:Voter ID:server-4d838492-b948-40da-f8f3-16b2646f3917 Address:4d838492-b948-40da-f8f3-16b2646f3917} {Suffrage:Voter ID:server-95d29895-c167-116b-dae3-ca6760f2c424 Address:95d29895-c167-116b-dae3-ca6760f2c424}]"
2025-07-05T01:12:01.288Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8d91f20-ff9c-358c-4c6e-5603842c43b5 Address:c8d91f20-ff9c-358c-4c6e-5603842c43b5} {Suffrage:Voter ID:server-4d838492-b948-40da-f8f3-16b2646f3917 Address:4d838492-b948-40da-f8f3-16b2646f3917} {Suffrage:Voter ID:server-95d29895-c167-116b-dae3-ca6760f2c424 Address:95d29895-c167-116b-dae3-ca6760f2c424}]"
2025-07-05T01:12:01.288Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:01.288Z [INFO]  server-c8d91f20-ff9c-358c-4c6e-5603842c43b5: entering follower state: follower="Node at c8d91f20-ff9c-358c-4c6e-5603842c43b5 [Follower]" leader-address= leader-id=
2025-07-05T01:12:01.288Z [INFO]  server-4d838492-b948-40da-f8f3-16b2646f3917: entering follower state: follower="Node at 4d838492-b948-40da-f8f3-16b2646f3917 [Follower]" leader-address= leader-id=
2025-07-05T01:12:01.288Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: entering follower state: follower="Node at 95d29895-c167-116b-dae3-ca6760f2c424 [Follower]" leader-address= leader-id=
2025-07-05T01:12:01.956Z [WARN]  server-95d29895-c167-116b-dae3-ca6760f2c424: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:01.956Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: entering candidate state: node="Node at 95d29895-c167-116b-dae3-ca6760f2c424 [Candidate]" term=2
2025-07-05T01:12:01.956Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: election won: term=2 tally=2
2025-07-05T01:12:01.956Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: entering leader state: leader="Node at 95d29895-c167-116b-dae3-ca6760f2c424 [Leader]"
2025-07-05T01:12:01.956Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: added peer, starting replication: peer=server-c8d91f20-ff9c-358c-4c6e-5603842c43b5
2025-07-05T01:12:01.956Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: added peer, starting replication: peer=server-4d838492-b948-40da-f8f3-16b2646f3917
2025-07-05T01:12:01.956Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: pipelining replication: peer="{Voter server-4d838492-b948-40da-f8f3-16b2646f3917 4d838492-b948-40da-f8f3-16b2646f3917}"
2025-07-05T01:12:01.956Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: pipelining replication: peer="{Voter server-c8d91f20-ff9c-358c-4c6e-5603842c43b5 c8d91f20-ff9c-358c-4c6e-5603842c43b5}"
2025-07-05T01:12:02.962Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:01.956707307 +0000 UTC m=+97.205444535 (1 nodes), 667.895975ms from start of poll, 668.09062ms from cluster start. Timeout at 2025-07-05 01:12:02.962011882 +0000 UTC m=+98.210749110, 1.005304575s after stability
2025-07-05T01:12:02.962Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: starting snapshot up to: index=12
2025-07-05T01:12:02.962Z [INFO]  creating new snapshot: path=/tmp/raft1198038365/snapshots/2-12-1751677922962.tmp
2025-07-05T01:12:03.124Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: snapshot complete up to: index=12
2025-07-05T01:12:03.126Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8 Address:ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8} {Suffrage:Voter ID:server-66b32b0b-f13a-278e-de8b-55342df925d2 Address:66b32b0b-f13a-278e-de8b-55342df925d2} {Suffrage:Voter ID:server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef Address:2fa960cb-1dcc-901c-1334-1eaf2c7e91ef}]"
2025-07-05T01:12:03.127Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8 Address:ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8} {Suffrage:Voter ID:server-66b32b0b-f13a-278e-de8b-55342df925d2 Address:66b32b0b-f13a-278e-de8b-55342df925d2} {Suffrage:Voter ID:server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef Address:2fa960cb-1dcc-901c-1334-1eaf2c7e91ef}]"
2025-07-05T01:12:03.127Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8 Address:ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8} {Suffrage:Voter ID:server-66b32b0b-f13a-278e-de8b-55342df925d2 Address:66b32b0b-f13a-278e-de8b-55342df925d2} {Suffrage:Voter ID:server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef Address:2fa960cb-1dcc-901c-1334-1eaf2c7e91ef}]"
2025-07-05T01:12:03.127Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:03.147Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: entering follower state: follower="Node at ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8 [Follower]" leader-address= leader-id=
2025-07-05T01:12:03.147Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: entering follower state: follower="Node at 66b32b0b-f13a-278e-de8b-55342df925d2 [Follower]" leader-address= leader-id=
2025-07-05T01:12:03.147Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: entering follower state: follower="Node at 2fa960cb-1dcc-901c-1334-1eaf2c7e91ef [Follower]" leader-address= leader-id=
2025-07-05T01:12:03.680Z [WARN]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:03.680Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: entering candidate state: node="Node at 2fa960cb-1dcc-901c-1334-1eaf2c7e91ef [Candidate]" term=2
2025-07-05T01:12:03.680Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: election won: term=2 tally=2
2025-07-05T01:12:03.680Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: entering leader state: leader="Node at 2fa960cb-1dcc-901c-1334-1eaf2c7e91ef [Leader]"
2025-07-05T01:12:03.680Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: added peer, starting replication: peer=server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8
2025-07-05T01:12:03.680Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: added peer, starting replication: peer=server-66b32b0b-f13a-278e-de8b-55342df925d2
2025-07-05T01:12:03.680Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: pipelining replication: peer="{Voter server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8 ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8}"
2025-07-05T01:12:03.680Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: pipelining replication: peer="{Voter server-66b32b0b-f13a-278e-de8b-55342df925d2 66b32b0b-f13a-278e-de8b-55342df925d2}"
2025-07-05T01:12:04.686Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:03.681039689 +0000 UTC m=+98.929776907 (1 nodes), 553.58982ms from start of poll, 554.621445ms from cluster start. Timeout at 2025-07-05 01:12:04.686602339 +0000 UTC m=+99.935339557, 1.00556265s after stability
2025-07-05T01:12:04.686Z [INFO]  creating new snapshot: path=/tmp/raft3788354176/snapshots/2-10013-1751677924686.tmp
2025-07-05T01:12:04.687Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: copied to local snapshot: bytes=71
2025-07-05T01:12:04.687Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: snapshot restore progress: id=2-10013-1751677924686 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:04.687Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: removing all old logs from log store
2025-07-05T01:12:04.687Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: compacting logs: from=1 to=2
2025-07-05T01:12:04.687Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: restored user snapshot: index=10013
2025-07-05T01:12:04.687Z [ERROR] server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: failed to get log: index=2 error="log not found"
2025-07-05T01:12:04.687Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: aborting pipeline replication: peer="{Voter server-66b32b0b-f13a-278e-de8b-55342df925d2 66b32b0b-f13a-278e-de8b-55342df925d2}"
2025-07-05T01:12:04.687Z [ERROR] server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: failed to get log: index=2 error="log not found"
2025-07-05T01:12:04.687Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: aborting pipeline replication: peer="{Voter server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8 ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8}"
2025-07-05T01:12:04.694Z [ERROR] server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: failed to get log: index=2 error="log not found"
2025-07-05T01:12:04.694Z [INFO]  creating new snapshot: path=/tmp/raft738888316/snapshots/2-10013-1751677924694.tmp
2025-07-05T01:12:04.694Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:04.694Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: copied to local snapshot: bytes=71
2025-07-05T01:12:04.695Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: snapshot restore progress: id=2-10013-1751677924694 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:04.695Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: removing all old logs from log store
2025-07-05T01:12:04.695Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: compacting logs: from=1 to=2
2025-07-05T01:12:04.695Z [INFO]  server-66b32b0b-f13a-278e-de8b-55342df925d2: Installed remote snapshot
2025-07-05T01:12:04.695Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: pipelining replication: peer="{Voter server-66b32b0b-f13a-278e-de8b-55342df925d2 66b32b0b-f13a-278e-de8b-55342df925d2}"
2025-07-05T01:12:04.695Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:04.696Z [ERROR] server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: failed to get log: index=2 error="log not found"
2025-07-05T01:12:04.696Z [INFO]  creating new snapshot: path=/tmp/raft1652973781/snapshots/2-10013-1751677924696.tmp
2025-07-05T01:12:04.696Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:04.696Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: copied to local snapshot: bytes=71
2025-07-05T01:12:04.697Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: snapshot restore progress: id=2-10013-1751677924696 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-05T01:12:04.697Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: removing all old logs from log store
2025-07-05T01:12:04.697Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: compacting logs: from=1 to=2
2025-07-05T01:12:04.697Z [INFO]  server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8: Installed remote snapshot
2025-07-05T01:12:04.697Z [INFO]  server-2fa960cb-1dcc-901c-1334-1eaf2c7e91ef: pipelining replication: peer="{Voter server-ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8 ffcd62f1-2d0b-00b2-ba4c-f33ef20981e8}"
2025-07-05T01:12:05.701Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:04.695199398 +0000 UTC m=+99.943936606 (2 nodes), 731ns from start of poll, 1.568781144s from cluster start. Timeout at 2025-07-05 01:12:05.701181094 +0000 UTC m=+100.949918302, 1.005981696s after stability
2025-07-05T01:12:05.720Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: aborting pipeline replication: peer="{Voter server-4d838492-b948-40da-f8f3-16b2646f3917 4d838492-b948-40da-f8f3-16b2646f3917}"
2025-07-05T01:12:05.720Z [INFO]  server-95d29895-c167-116b-dae3-ca6760f2c424: aborting pipeline replication: peer="{Voter server-c8d91f20-ff9c-358c-4c6e-5603842c43b5 c8d91f20-ff9c-358c-4c6e-5603842c43b5}"
--- PASS: TestRaft_UserRestore (86.17s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.67s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.93s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.59s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.60s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.58s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (3.17s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.89s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.11s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.57s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.79s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.65s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.29s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.31s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.68s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.17s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.78s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.18s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.43s)
=== RUN   TestRaft_SendSnapshotFollower
2025-07-05T01:12:05.724Z [INFO]  server-d695700f-45d0-a59d-8234-66de712d1a3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d695700f-45d0-a59d-8234-66de712d1a3b Address:d695700f-45d0-a59d-8234-66de712d1a3b} {Suffrage:Voter ID:server-53a43b47-e376-6559-b329-41ef32b5ea98 Address:53a43b47-e376-6559-b329-41ef32b5ea98} {Suffrage:Voter ID:server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b Address:575e5db6-f9a4-0d02-c9d8-735f3e099a5b}]"
2025-07-05T01:12:05.724Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d695700f-45d0-a59d-8234-66de712d1a3b Address:d695700f-45d0-a59d-8234-66de712d1a3b} {Suffrage:Voter ID:server-53a43b47-e376-6559-b329-41ef32b5ea98 Address:53a43b47-e376-6559-b329-41ef32b5ea98} {Suffrage:Voter ID:server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b Address:575e5db6-f9a4-0d02-c9d8-735f3e099a5b}]"
2025-07-05T01:12:05.725Z [INFO]  server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d695700f-45d0-a59d-8234-66de712d1a3b Address:d695700f-45d0-a59d-8234-66de712d1a3b} {Suffrage:Voter ID:server-53a43b47-e376-6559-b329-41ef32b5ea98 Address:53a43b47-e376-6559-b329-41ef32b5ea98} {Suffrage:Voter ID:server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b Address:575e5db6-f9a4-0d02-c9d8-735f3e099a5b}]"
2025-07-05T01:12:05.725Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:05.735Z [INFO]  server-d695700f-45d0-a59d-8234-66de712d1a3b: entering follower state: follower="Node at d695700f-45d0-a59d-8234-66de712d1a3b [Follower]" leader-address= leader-id=
2025-07-05T01:12:05.735Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: entering follower state: follower="Node at 53a43b47-e376-6559-b329-41ef32b5ea98 [Follower]" leader-address= leader-id=
2025-07-05T01:12:05.735Z [INFO]  server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b: entering follower state: follower="Node at 575e5db6-f9a4-0d02-c9d8-735f3e099a5b [Follower]" leader-address= leader-id=
2025-07-05T01:12:05.820Z [WARN]  server-53a43b47-e376-6559-b329-41ef32b5ea98: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:05.820Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: entering candidate state: node="Node at 53a43b47-e376-6559-b329-41ef32b5ea98 [Candidate]" term=2
2025-07-05T01:12:05.820Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: election won: term=2 tally=2
2025-07-05T01:12:05.820Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: entering leader state: leader="Node at 53a43b47-e376-6559-b329-41ef32b5ea98 [Leader]"
2025-07-05T01:12:05.820Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: added peer, starting replication: peer=server-d695700f-45d0-a59d-8234-66de712d1a3b
2025-07-05T01:12:05.820Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: added peer, starting replication: peer=server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b
2025-07-05T01:12:05.821Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: pipelining replication: peer="{Voter server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b 575e5db6-f9a4-0d02-c9d8-735f3e099a5b}"
2025-07-05T01:12:05.821Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: pipelining replication: peer="{Voter server-d695700f-45d0-a59d-8234-66de712d1a3b d695700f-45d0-a59d-8234-66de712d1a3b}"
2025-07-05T01:12:05.926Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:05.821193016 +0000 UTC m=+101.069930234 (2 nodes), 96.12777ms from start of poll, 96.403537ms from cluster start. Timeout at 2025-07-05 01:12:05.926388858 +0000 UTC m=+101.175126076, 105.195842ms after stability
2025-07-05T01:12:05.926Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:06.032Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:05.926791293 +0000 UTC m=+101.175528511 (1 nodes), 2.846µs from start of poll, 202.001814ms from cluster start. Timeout at 2025-07-05 01:12:06.032127799 +0000 UTC m=+101.280865017, 105.336506ms after stability
2025-07-05T01:12:06.032Z [ERROR] server-53a43b47-e376-6559-b329-41ef32b5ea98: failed to pipeline appendEntries: peer="{Voter server-d695700f-45d0-a59d-8234-66de712d1a3b d695700f-45d0-a59d-8234-66de712d1a3b}" error="append pipeline closed"
2025-07-05T01:12:06.032Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: aborting pipeline replication: peer="{Voter server-d695700f-45d0-a59d-8234-66de712d1a3b d695700f-45d0-a59d-8234-66de712d1a3b}"
2025-07-05T01:12:06.032Z [ERROR] server-53a43b47-e376-6559-b329-41ef32b5ea98: failed to appendEntries to: peer="{Voter server-d695700f-45d0-a59d-8234-66de712d1a3b d695700f-45d0-a59d-8234-66de712d1a3b}" error="failed to connect to peer: d695700f-45d0-a59d-8234-66de712d1a3b"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-07-05T01:12:06.034Z [ERROR] server-d695700f-45d0-a59d-8234-66de712d1a3b: failed to take snapshot: error="nothing new to snapshot"
2025-07-05T01:12:06.034Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: starting snapshot up to: index=102
2025-07-05T01:12:06.034Z [INFO]  creating new snapshot: path=/tmp/raft3749774953/snapshots/2-102-1751677926034.tmp
2025-07-05T01:12:06.035Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: compacting logs: from=1 to=92
2025-07-05T01:12:06.035Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: snapshot complete up to: index=102
2025-07-05T01:12:06.035Z [INFO]  server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b: starting snapshot up to: index=98
2025-07-05T01:12:06.035Z [INFO]  creating new snapshot: path=/tmp/raft2834815423/snapshots/2-98-1751677926035.tmp
2025-07-05T01:12:06.035Z [INFO]  server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b: compacting logs: from=1 to=92
2025-07-05T01:12:06.035Z [INFO]  server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b: snapshot complete up to: index=98
2025-07-05T01:12:06.042Z [ERROR] server-53a43b47-e376-6559-b329-41ef32b5ea98: failed to get log: index=2 error="log not found"
2025-07-05T01:12:06.043Z [INFO]  creating new snapshot: path=/tmp/raft780023595/snapshots/2-102-1751677926043.tmp
2025-07-05T01:12:06.043Z [INFO]  server-d695700f-45d0-a59d-8234-66de712d1a3b: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-07-05T01:12:06.043Z [INFO]  server-d695700f-45d0-a59d-8234-66de712d1a3b: copied to local snapshot: bytes=693
2025-07-05T01:12:06.044Z [INFO]  server-d695700f-45d0-a59d-8234-66de712d1a3b: snapshot restore progress: id=2-102-1751677926043 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-05T01:12:06.044Z [INFO]  server-d695700f-45d0-a59d-8234-66de712d1a3b: Installed remote snapshot
2025-07-05T01:12:06.044Z [WARN]  server-d695700f-45d0-a59d-8234-66de712d1a3b: skipping application of old log: index=102
2025-07-05T01:12:06.044Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: pipelining replication: peer="{Voter server-d695700f-45d0-a59d-8234-66de712d1a3b d695700f-45d0-a59d-8234-66de712d1a3b}"
2025-07-05T01:12:06.048Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: aborting pipeline replication: peer="{Voter server-575e5db6-f9a4-0d02-c9d8-735f3e099a5b 575e5db6-f9a4-0d02-c9d8-735f3e099a5b}"
2025-07-05T01:12:06.048Z [INFO]  server-53a43b47-e376-6559-b329-41ef32b5ea98: aborting pipeline replication: peer="{Voter server-d695700f-45d0-a59d-8234-66de712d1a3b d695700f-45d0-a59d-8234-66de712d1a3b}"
--- PASS: TestRaft_SendSnapshotFollower (0.33s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-07-05T01:12:06.050Z [INFO]  server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e Address:cd214d6f-4e7e-5908-ed38-1c0d2f93315e} {Suffrage:Voter ID:server-06665f1e-f353-37d1-2683-53932302e17d Address:06665f1e-f353-37d1-2683-53932302e17d} {Suffrage:Voter ID:server-9b3ccc45-8146-f578-aac0-6814564e4090 Address:9b3ccc45-8146-f578-aac0-6814564e4090}]"
2025-07-05T01:12:06.050Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e Address:cd214d6f-4e7e-5908-ed38-1c0d2f93315e} {Suffrage:Voter ID:server-06665f1e-f353-37d1-2683-53932302e17d Address:06665f1e-f353-37d1-2683-53932302e17d} {Suffrage:Voter ID:server-9b3ccc45-8146-f578-aac0-6814564e4090 Address:9b3ccc45-8146-f578-aac0-6814564e4090}]"
2025-07-05T01:12:06.050Z [INFO]  server-9b3ccc45-8146-f578-aac0-6814564e4090: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e Address:cd214d6f-4e7e-5908-ed38-1c0d2f93315e} {Suffrage:Voter ID:server-06665f1e-f353-37d1-2683-53932302e17d Address:06665f1e-f353-37d1-2683-53932302e17d} {Suffrage:Voter ID:server-9b3ccc45-8146-f578-aac0-6814564e4090 Address:9b3ccc45-8146-f578-aac0-6814564e4090}]"
2025-07-05T01:12:06.050Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:06.050Z [INFO]  server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e: entering follower state: follower="Node at cd214d6f-4e7e-5908-ed38-1c0d2f93315e [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.050Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: entering follower state: follower="Node at 06665f1e-f353-37d1-2683-53932302e17d [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.050Z [INFO]  server-9b3ccc45-8146-f578-aac0-6814564e4090: entering follower state: follower="Node at 9b3ccc45-8146-f578-aac0-6814564e4090 [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.105Z [WARN]  server-06665f1e-f353-37d1-2683-53932302e17d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:06.105Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: entering candidate state: node="Node at 06665f1e-f353-37d1-2683-53932302e17d [Candidate]" term=2
2025-07-05T01:12:06.105Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: election won: term=2 tally=2
2025-07-05T01:12:06.105Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: entering leader state: leader="Node at 06665f1e-f353-37d1-2683-53932302e17d [Leader]"
2025-07-05T01:12:06.105Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: added peer, starting replication: peer=server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e
2025-07-05T01:12:06.105Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: added peer, starting replication: peer=server-9b3ccc45-8146-f578-aac0-6814564e4090
2025-07-05T01:12:06.105Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: pipelining replication: peer="{Voter server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e cd214d6f-4e7e-5908-ed38-1c0d2f93315e}"
2025-07-05T01:12:06.106Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: pipelining replication: peer="{Voter server-9b3ccc45-8146-f578-aac0-6814564e4090 9b3ccc45-8146-f578-aac0-6814564e4090}"
2025-07-05T01:12:06.211Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:06.106052578 +0000 UTC m=+101.354789796 (2 nodes), 55.4826ms from start of poll, 55.898039ms from cluster start. Timeout at 2025-07-05 01:12:06.211474674 +0000 UTC m=+101.460211892, 105.422096ms after stability
2025-07-05T01:12:06.211Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:06.317Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:06.211555656 +0000 UTC m=+101.460292884 (1 nodes), 1.643µs from start of poll, 161.401127ms from cluster start. Timeout at 2025-07-05 01:12:06.317311949 +0000 UTC m=+101.566049177, 105.756293ms after stability
2025-07-05T01:12:06.317Z [ERROR] server-06665f1e-f353-37d1-2683-53932302e17d: failed to pipeline appendEntries: peer="{Voter server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e cd214d6f-4e7e-5908-ed38-1c0d2f93315e}" error="append pipeline closed"
2025-07-05T01:12:06.317Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: aborting pipeline replication: peer="{Voter server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e cd214d6f-4e7e-5908-ed38-1c0d2f93315e}"
2025-07-05T01:12:06.317Z [ERROR] server-06665f1e-f353-37d1-2683-53932302e17d: failed to appendEntries to: peer="{Voter server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e cd214d6f-4e7e-5908-ed38-1c0d2f93315e}" error="failed to connect to peer: cd214d6f-4e7e-5908-ed38-1c0d2f93315e"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-07-05T01:12:06.319Z [ERROR] server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e: failed to take snapshot: error="nothing new to snapshot"
2025-07-05T01:12:06.319Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: starting snapshot up to: index=102
2025-07-05T01:12:06.319Z [INFO]  creating new snapshot: path=/tmp/raft298395764/snapshots/2-102-1751677926319.tmp
2025-07-05T01:12:06.320Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: compacting logs: from=1 to=92
2025-07-05T01:12:06.320Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: snapshot complete up to: index=102
2025-07-05T01:12:06.320Z [INFO]  server-9b3ccc45-8146-f578-aac0-6814564e4090: starting snapshot up to: index=100
2025-07-05T01:12:06.320Z [INFO]  creating new snapshot: path=/tmp/raft1159211566/snapshots/2-100-1751677926320.tmp
2025-07-05T01:12:06.320Z [INFO]  server-9b3ccc45-8146-f578-aac0-6814564e4090: compacting logs: from=1 to=92
2025-07-05T01:12:06.320Z [INFO]  server-9b3ccc45-8146-f578-aac0-6814564e4090: snapshot complete up to: index=100
2025-07-05T01:12:06.328Z [ERROR] server-06665f1e-f353-37d1-2683-53932302e17d: failed to get log: index=2 error="log not found"
2025-07-05T01:12:06.331Z [ERROR] server-06665f1e-f353-37d1-2683-53932302e17d: failed to heartbeat to: peer=cd214d6f-4e7e-5908-ed38-1c0d2f93315e backoff time=10ms error="failed to connect to peer: cd214d6f-4e7e-5908-ed38-1c0d2f93315e"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-07-05T01:12:06.340Z [INFO]  creating new snapshot: path=/tmp/raft748809194/snapshots/2-102-1751677926340.tmp
2025-07-05T01:12:06.340Z [INFO]  server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-07-05T01:12:06.340Z [INFO]  server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e: copied to local snapshot: bytes=693
2025-07-05T01:12:06.340Z [INFO]  server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e: snapshot restore progress: id=2-102-1751677926340 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-05T01:12:06.340Z [INFO]  server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e: Installed remote snapshot
2025-07-05T01:12:06.341Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: pipelining replication: peer="{Voter server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e cd214d6f-4e7e-5908-ed38-1c0d2f93315e}"
2025-07-05T01:12:06.351Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: aborting pipeline replication: peer="{Voter server-9b3ccc45-8146-f578-aac0-6814564e4090 9b3ccc45-8146-f578-aac0-6814564e4090}"
2025-07-05T01:12:06.351Z [INFO]  server-06665f1e-f353-37d1-2683-53932302e17d: aborting pipeline replication: peer="{Voter server-cd214d6f-4e7e-5908-ed38-1c0d2f93315e cd214d6f-4e7e-5908-ed38-1c0d2f93315e}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s)
=== RUN   TestRaft_ReJoinFollower
2025-07-05T01:12:06.353Z [INFO]  server-c241a206-918e-68e7-3c8a-863fd4092381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c241a206-918e-68e7-3c8a-863fd4092381 Address:c241a206-918e-68e7-3c8a-863fd4092381} {Suffrage:Voter ID:server-65d27d65-604c-cc7f-0130-92499177246b Address:65d27d65-604c-cc7f-0130-92499177246b} {Suffrage:Voter ID:server-86c2028e-a0e2-0c39-d96e-942d886c2b16 Address:86c2028e-a0e2-0c39-d96e-942d886c2b16}]"
2025-07-05T01:12:06.353Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c241a206-918e-68e7-3c8a-863fd4092381 Address:c241a206-918e-68e7-3c8a-863fd4092381} {Suffrage:Voter ID:server-65d27d65-604c-cc7f-0130-92499177246b Address:65d27d65-604c-cc7f-0130-92499177246b} {Suffrage:Voter ID:server-86c2028e-a0e2-0c39-d96e-942d886c2b16 Address:86c2028e-a0e2-0c39-d96e-942d886c2b16}]"
2025-07-05T01:12:06.353Z [INFO]  server-86c2028e-a0e2-0c39-d96e-942d886c2b16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c241a206-918e-68e7-3c8a-863fd4092381 Address:c241a206-918e-68e7-3c8a-863fd4092381} {Suffrage:Voter ID:server-65d27d65-604c-cc7f-0130-92499177246b Address:65d27d65-604c-cc7f-0130-92499177246b} {Suffrage:Voter ID:server-86c2028e-a0e2-0c39-d96e-942d886c2b16 Address:86c2028e-a0e2-0c39-d96e-942d886c2b16}]"
2025-07-05T01:12:06.353Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:06.353Z [INFO]  server-c241a206-918e-68e7-3c8a-863fd4092381: entering follower state: follower="Node at c241a206-918e-68e7-3c8a-863fd4092381 [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.353Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: entering follower state: follower="Node at 65d27d65-604c-cc7f-0130-92499177246b [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.353Z [INFO]  server-86c2028e-a0e2-0c39-d96e-942d886c2b16: entering follower state: follower="Node at 86c2028e-a0e2-0c39-d96e-942d886c2b16 [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.405Z [WARN]  server-65d27d65-604c-cc7f-0130-92499177246b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:06.405Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: entering candidate state: node="Node at 65d27d65-604c-cc7f-0130-92499177246b [Candidate]" term=2
2025-07-05T01:12:06.405Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: election won: term=2 tally=2
2025-07-05T01:12:06.405Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: entering leader state: leader="Node at 65d27d65-604c-cc7f-0130-92499177246b [Leader]"
2025-07-05T01:12:06.405Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: added peer, starting replication: peer=server-c241a206-918e-68e7-3c8a-863fd4092381
2025-07-05T01:12:06.405Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: added peer, starting replication: peer=server-86c2028e-a0e2-0c39-d96e-942d886c2b16
2025-07-05T01:12:06.405Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: pipelining replication: peer="{Voter server-86c2028e-a0e2-0c39-d96e-942d886c2b16 86c2028e-a0e2-0c39-d96e-942d886c2b16}"
2025-07-05T01:12:06.405Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: pipelining replication: peer="{Voter server-c241a206-918e-68e7-3c8a-863fd4092381 c241a206-918e-68e7-3c8a-863fd4092381}"
2025-07-05T01:12:06.510Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:06.405557118 +0000 UTC m=+101.654294347 (1 nodes), 51.934418ms from start of poll, 52.177704ms from cluster start. Timeout at 2025-07-05 01:12:06.510826418 +0000 UTC m=+101.759563646, 105.269299ms after stability
2025-07-05T01:12:06.515Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:06.621Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:06.51598982 +0000 UTC m=+101.764727028 (2 nodes), 1.413µs from start of poll, 162.610385ms from cluster start. Timeout at 2025-07-05 01:12:06.62111014 +0000 UTC m=+101.869847368, 105.12034ms after stability
2025-07-05T01:12:06.621Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: updating configuration: command=RemoveServer server-id=server-c241a206-918e-68e7-3c8a-863fd4092381 server-addr= servers="[{Suffrage:Voter ID:server-65d27d65-604c-cc7f-0130-92499177246b Address:65d27d65-604c-cc7f-0130-92499177246b} {Suffrage:Voter ID:server-86c2028e-a0e2-0c39-d96e-942d886c2b16 Address:86c2028e-a0e2-0c39-d96e-942d886c2b16}]"
2025-07-05T01:12:06.621Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: removed peer, stopping replication: peer=server-c241a206-918e-68e7-3c8a-863fd4092381 last-index=3
2025-07-05T01:12:06.621Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: aborting pipeline replication: peer="{Voter server-c241a206-918e-68e7-3c8a-863fd4092381 c241a206-918e-68e7-3c8a-863fd4092381}"
2025-07-05T01:12:06.698Z [WARN]  server-c241a206-918e-68e7-3c8a-863fd4092381: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-07-05T01:12:06.727Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: updating configuration: command=AddVoter server-id=server-c241a206-918e-68e7-3c8a-863fd4092381 server-addr=c241a206-918e-68e7-3c8a-863fd4092381 servers="[{Suffrage:Voter ID:server-65d27d65-604c-cc7f-0130-92499177246b Address:65d27d65-604c-cc7f-0130-92499177246b} {Suffrage:Voter ID:server-86c2028e-a0e2-0c39-d96e-942d886c2b16 Address:86c2028e-a0e2-0c39-d96e-942d886c2b16} {Suffrage:Voter ID:server-c241a206-918e-68e7-3c8a-863fd4092381 Address:c241a206-918e-68e7-3c8a-863fd4092381}]"
2025-07-05T01:12:06.727Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: added peer, starting replication: peer=server-c241a206-918e-68e7-3c8a-863fd4092381
2025-07-05T01:12:06.727Z [WARN]  server-c241a206-918e-68e7-3c8a-863fd4092381: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-07-05T01:12:06.727Z [WARN]  server-65d27d65-604c-cc7f-0130-92499177246b: appendEntries rejected, sending older logs: peer="{Voter server-c241a206-918e-68e7-3c8a-863fd4092381 c241a206-918e-68e7-3c8a-863fd4092381}" next=4
2025-07-05T01:12:06.727Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: pipelining replication: peer="{Voter server-c241a206-918e-68e7-3c8a-863fd4092381 c241a206-918e-68e7-3c8a-863fd4092381}"
2025-07-05T01:12:06.727Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:06.832Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:06.727714263 +0000 UTC m=+101.976451481 (1 nodes), 521ns from start of poll, 374.334838ms from cluster start. Timeout at 2025-07-05 01:12:06.832767969 +0000 UTC m=+102.081505177, 105.053696ms after stability
2025-07-05T01:12:06.832Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: aborting pipeline replication: peer="{Voter server-86c2028e-a0e2-0c39-d96e-942d886c2b16 86c2028e-a0e2-0c39-d96e-942d886c2b16}"
2025-07-05T01:12:06.832Z [INFO]  server-65d27d65-604c-cc7f-0130-92499177246b: aborting pipeline replication: peer="{Voter server-c241a206-918e-68e7-3c8a-863fd4092381 c241a206-918e-68e7-3c8a-863fd4092381}"
--- PASS: TestRaft_ReJoinFollower (0.48s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-07-05T01:12:06.833Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e48533d-a036-3fa2-29a8-5294186c7262 Address:8e48533d-a036-3fa2-29a8-5294186c7262} {Suffrage:Voter ID:server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 Address:98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}]"
2025-07-05T01:12:06.833Z [INFO]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e48533d-a036-3fa2-29a8-5294186c7262 Address:8e48533d-a036-3fa2-29a8-5294186c7262} {Suffrage:Voter ID:server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 Address:98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}]"
2025-07-05T01:12:06.833Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:06.833Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: entering follower state: follower="Node at 8e48533d-a036-3fa2-29a8-5294186c7262 [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.833Z [INFO]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: entering follower state: follower="Node at 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 [Follower]" leader-address= leader-id=
2025-07-05T01:12:06.890Z [WARN]  server-8e48533d-a036-3fa2-29a8-5294186c7262: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:06.890Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: entering candidate state: node="Node at 8e48533d-a036-3fa2-29a8-5294186c7262 [Candidate]" term=2
2025-07-05T01:12:06.890Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: election won: term=2 tally=2
2025-07-05T01:12:06.890Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: entering leader state: leader="Node at 8e48533d-a036-3fa2-29a8-5294186c7262 [Leader]"
2025-07-05T01:12:06.890Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: added peer, starting replication: peer=server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960
2025-07-05T01:12:06.891Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: pipelining replication: peer="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}"
2025-07-05T01:12:06.996Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:06.891097176 +0000 UTC m=+102.139834384 (1 nodes), 57.156599ms from start of poll, 57.356754ms from cluster start. Timeout at 2025-07-05 01:12:06.996626734 +0000 UTC m=+102.245363942, 105.529558ms after stability
2025-07-05T01:12:07.003Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:07.108Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:07.003126481 +0000 UTC m=+102.251863689 (1 nodes), 732ns from start of poll, 169.386059ms from cluster start. Timeout at 2025-07-05 01:12:07.10848593 +0000 UTC m=+102.357223138, 105.359449ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 [Follower]
2025-07-05T01:12:07.119Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to pipeline appendEntries: peer="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}" error="append pipeline closed"
2025-07-05T01:12:07.119Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: aborting pipeline replication: peer="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}"
2025-07-05T01:12:07.119Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to heartbeat to: peer=98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 backoff time=10ms error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960"
2025-07-05T01:12:07.128Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to appendEntries to: peer="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}" error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960"
2025-07-05T01:12:07.139Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to heartbeat to: peer=98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 backoff time=10ms error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960"
2025-07-05T01:12:07.149Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to appendEntries to: peer="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}" error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960"
2025-07-05T01:12:07.158Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to heartbeat to: peer=98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 backoff time=10ms error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960"
2025-07-05T01:12:07.158Z [WARN]  server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to contact: server-id=server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 time=51.028759ms
2025-07-05T01:12:07.158Z [WARN]  server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to contact quorum of nodes, stepping down
2025-07-05T01:12:07.158Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: entering follower state: follower="Node at 8e48533d-a036-3fa2-29a8-5294186c7262 [Follower]" leader-address= leader-id=
2025-07-05T01:12:07.158Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:07.165Z [WARN]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: heartbeat timeout reached, starting election: last-leader-addr=8e48533d-a036-3fa2-29a8-5294186c7262 last-leader-id=server-8e48533d-a036-3fa2-29a8-5294186c7262
2025-07-05T01:12:07.165Z [INFO]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: entering candidate state: node="Node at 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 [Candidate]" term=3
2025-07-05T01:12:07.165Z [ERROR] server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: failed to make requestVote RPC: target="{Voter server-8e48533d-a036-3fa2-29a8-5294186c7262 8e48533d-a036-3fa2-29a8-5294186c7262}" error="failed to connect to peer: 8e48533d-a036-3fa2-29a8-5294186c7262" term=3
2025-07-05T01:12:07.166Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to appendEntries to: peer="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}" error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960"
2025-07-05T01:12:07.241Z [WARN]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: Election timeout reached, restarting election
2025-07-05T01:12:07.241Z [INFO]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: entering candidate state: node="Node at 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 [Candidate]" term=4
2025-07-05T01:12:07.242Z [ERROR] server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: failed to make requestVote RPC: target="{Voter server-8e48533d-a036-3fa2-29a8-5294186c7262 8e48533d-a036-3fa2-29a8-5294186c7262}" error="failed to connect to peer: 8e48533d-a036-3fa2-29a8-5294186c7262" term=4
2025-07-05T01:12:07.243Z [WARN]  server-8e48533d-a036-3fa2-29a8-5294186c7262: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:07.243Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: entering candidate state: node="Node at 8e48533d-a036-3fa2-29a8-5294186c7262 [Candidate]" term=3
2025-07-05T01:12:07.243Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to make requestVote RPC: target="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}" error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960" term=3
2025-07-05T01:12:07.332Z [WARN]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: Election timeout reached, restarting election
2025-07-05T01:12:07.332Z [INFO]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: entering candidate state: node="Node at 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 [Candidate]" term=5
2025-07-05T01:12:07.332Z [ERROR] server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: failed to make requestVote RPC: target="{Voter server-8e48533d-a036-3fa2-29a8-5294186c7262 8e48533d-a036-3fa2-29a8-5294186c7262}" error="failed to connect to peer: 8e48533d-a036-3fa2-29a8-5294186c7262" term=5
2025-07-05T01:12:07.333Z [WARN]  server-8e48533d-a036-3fa2-29a8-5294186c7262: Election timeout reached, restarting election
2025-07-05T01:12:07.333Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: entering candidate state: node="Node at 8e48533d-a036-3fa2-29a8-5294186c7262 [Candidate]" term=4
2025-07-05T01:12:07.333Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to make requestVote RPC: target="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}" error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960" term=4
2025-07-05T01:12:07.348Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:07.243445329 +0000 UTC m=+102.492182537 (0 nodes), 85.084624ms from start of poll, 409.704907ms from cluster start. Timeout at 2025-07-05 01:12:07.348649837 +0000 UTC m=+102.597387045, 105.204508ms after stability
2025-07-05T01:12:07.400Z [WARN]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: Election timeout reached, restarting election
2025-07-05T01:12:07.400Z [INFO]  server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: entering candidate state: node="Node at 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 [Candidate]" term=6
2025-07-05T01:12:07.400Z [ERROR] server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960: failed to make requestVote RPC: target="{Voter server-8e48533d-a036-3fa2-29a8-5294186c7262 8e48533d-a036-3fa2-29a8-5294186c7262}" error="failed to connect to peer: 8e48533d-a036-3fa2-29a8-5294186c7262" term=6
2025-07-05T01:12:07.400Z [WARN]  server-8e48533d-a036-3fa2-29a8-5294186c7262: Election timeout reached, restarting election
2025-07-05T01:12:07.400Z [INFO]  server-8e48533d-a036-3fa2-29a8-5294186c7262: entering candidate state: node="Node at 8e48533d-a036-3fa2-29a8-5294186c7262 [Candidate]" term=5
2025-07-05T01:12:07.400Z [ERROR] server-8e48533d-a036-3fa2-29a8-5294186c7262: failed to make requestVote RPC: target="{Voter server-98f7bf22-ebb3-f549-c0e4-f4d89d7c5960 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960}" error="failed to connect to peer: 98f7bf22-ebb3-f549-c0e4-f4d89d7c5960" term=5
--- PASS: TestRaft_LeaderLeaseExpire (0.62s)
=== RUN   TestRaft_Barrier
2025-07-05T01:12:07.455Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbb144ca-eeb2-e2da-4194-b754866a3b44 Address:fbb144ca-eeb2-e2da-4194-b754866a3b44} {Suffrage:Voter ID:server-a59879b1-44f3-1de5-e03b-f96a9129bc4c Address:a59879b1-44f3-1de5-e03b-f96a9129bc4c} {Suffrage:Voter ID:server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740 Address:a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740}]"
2025-07-05T01:12:07.458Z [INFO]  server-a59879b1-44f3-1de5-e03b-f96a9129bc4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbb144ca-eeb2-e2da-4194-b754866a3b44 Address:fbb144ca-eeb2-e2da-4194-b754866a3b44} {Suffrage:Voter ID:server-a59879b1-44f3-1de5-e03b-f96a9129bc4c Address:a59879b1-44f3-1de5-e03b-f96a9129bc4c} {Suffrage:Voter ID:server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740 Address:a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740}]"
2025-07-05T01:12:07.459Z [INFO]  server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbb144ca-eeb2-e2da-4194-b754866a3b44 Address:fbb144ca-eeb2-e2da-4194-b754866a3b44} {Suffrage:Voter ID:server-a59879b1-44f3-1de5-e03b-f96a9129bc4c Address:a59879b1-44f3-1de5-e03b-f96a9129bc4c} {Suffrage:Voter ID:server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740 Address:a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740}]"
2025-07-05T01:12:07.459Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:07.459Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: entering follower state: follower="Node at fbb144ca-eeb2-e2da-4194-b754866a3b44 [Follower]" leader-address= leader-id=
2025-07-05T01:12:07.459Z [INFO]  server-a59879b1-44f3-1de5-e03b-f96a9129bc4c: entering follower state: follower="Node at a59879b1-44f3-1de5-e03b-f96a9129bc4c [Follower]" leader-address= leader-id=
2025-07-05T01:12:07.459Z [INFO]  server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740: entering follower state: follower="Node at a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740 [Follower]" leader-address= leader-id=
2025-07-05T01:12:07.515Z [WARN]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:07.515Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: entering candidate state: node="Node at fbb144ca-eeb2-e2da-4194-b754866a3b44 [Candidate]" term=2
2025-07-05T01:12:07.515Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: election won: term=2 tally=2
2025-07-05T01:12:07.515Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: entering leader state: leader="Node at fbb144ca-eeb2-e2da-4194-b754866a3b44 [Leader]"
2025-07-05T01:12:07.515Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: added peer, starting replication: peer=server-a59879b1-44f3-1de5-e03b-f96a9129bc4c
2025-07-05T01:12:07.515Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: added peer, starting replication: peer=server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740
2025-07-05T01:12:07.515Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: pipelining replication: peer="{Voter server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740 a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740}"
2025-07-05T01:12:07.515Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: pipelining replication: peer="{Voter server-a59879b1-44f3-1de5-e03b-f96a9129bc4c a59879b1-44f3-1de5-e03b-f96a9129bc4c}"
2025-07-05T01:12:07.621Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:07.515420802 +0000 UTC m=+102.764158020 (1 nodes), 56.370624ms from start of poll, 60.319068ms from cluster start. Timeout at 2025-07-05 01:12:07.621397469 +0000 UTC m=+102.870134677, 105.976657ms after stability
2025-07-05T01:12:07.634Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: aborting pipeline replication: peer="{Voter server-a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740 a5e4ec1b-5d6f-cea9-de25-e62d5c8c5740}"
2025-07-05T01:12:07.634Z [INFO]  server-fbb144ca-eeb2-e2da-4194-b754866a3b44: aborting pipeline replication: peer="{Voter server-a59879b1-44f3-1de5-e03b-f96a9129bc4c a59879b1-44f3-1de5-e03b-f96a9129bc4c}"
--- PASS: TestRaft_Barrier (0.18s)
=== RUN   TestRaft_VerifyLeader
2025-07-05T01:12:07.639Z [INFO]  server-814480c1-0285-8aa6-8062-05531567c518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-814480c1-0285-8aa6-8062-05531567c518 Address:814480c1-0285-8aa6-8062-05531567c518} {Suffrage:Voter ID:server-734ef5a7-9786-03d8-e700-a8795022d42c Address:734ef5a7-9786-03d8-e700-a8795022d42c} {Suffrage:Voter ID:server-066511b5-d290-19b8-8218-19c562551b35 Address:066511b5-d290-19b8-8218-19c562551b35}]"
2025-07-05T01:12:07.639Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-814480c1-0285-8aa6-8062-05531567c518 Address:814480c1-0285-8aa6-8062-05531567c518} {Suffrage:Voter ID:server-734ef5a7-9786-03d8-e700-a8795022d42c Address:734ef5a7-9786-03d8-e700-a8795022d42c} {Suffrage:Voter ID:server-066511b5-d290-19b8-8218-19c562551b35 Address:066511b5-d290-19b8-8218-19c562551b35}]"
2025-07-05T01:12:07.639Z [INFO]  server-066511b5-d290-19b8-8218-19c562551b35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-814480c1-0285-8aa6-8062-05531567c518 Address:814480c1-0285-8aa6-8062-05531567c518} {Suffrage:Voter ID:server-734ef5a7-9786-03d8-e700-a8795022d42c Address:734ef5a7-9786-03d8-e700-a8795022d42c} {Suffrage:Voter ID:server-066511b5-d290-19b8-8218-19c562551b35 Address:066511b5-d290-19b8-8218-19c562551b35}]"
2025-07-05T01:12:07.639Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:07.639Z [INFO]  server-814480c1-0285-8aa6-8062-05531567c518: entering follower state: follower="Node at 814480c1-0285-8aa6-8062-05531567c518 [Follower]" leader-address= leader-id=
2025-07-05T01:12:07.639Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: entering follower state: follower="Node at 734ef5a7-9786-03d8-e700-a8795022d42c [Follower]" leader-address= leader-id=
2025-07-05T01:12:07.639Z [INFO]  server-066511b5-d290-19b8-8218-19c562551b35: entering follower state: follower="Node at 066511b5-d290-19b8-8218-19c562551b35 [Follower]" leader-address= leader-id=
2025-07-05T01:12:07.709Z [WARN]  server-734ef5a7-9786-03d8-e700-a8795022d42c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:07.709Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: entering candidate state: node="Node at 734ef5a7-9786-03d8-e700-a8795022d42c [Candidate]" term=2
2025-07-05T01:12:07.709Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: election won: term=2 tally=2
2025-07-05T01:12:07.709Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: entering leader state: leader="Node at 734ef5a7-9786-03d8-e700-a8795022d42c [Leader]"
2025-07-05T01:12:07.709Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: added peer, starting replication: peer=server-814480c1-0285-8aa6-8062-05531567c518
2025-07-05T01:12:07.709Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: added peer, starting replication: peer=server-066511b5-d290-19b8-8218-19c562551b35
2025-07-05T01:12:07.709Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: pipelining replication: peer="{Voter server-066511b5-d290-19b8-8218-19c562551b35 066511b5-d290-19b8-8218-19c562551b35}"
2025-07-05T01:12:07.710Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: pipelining replication: peer="{Voter server-814480c1-0285-8aa6-8062-05531567c518 814480c1-0285-8aa6-8062-05531567c518}"
2025-07-05T01:12:07.816Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:07.710088795 +0000 UTC m=+102.958826003 (1 nodes), 70.722595ms from start of poll, 71.05052ms from cluster start. Timeout at 2025-07-05 01:12:07.81607025 +0000 UTC m=+103.064807458, 105.981455ms after stability
2025-07-05T01:12:07.816Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: aborting pipeline replication: peer="{Voter server-814480c1-0285-8aa6-8062-05531567c518 814480c1-0285-8aa6-8062-05531567c518}"
2025-07-05T01:12:07.816Z [INFO]  server-734ef5a7-9786-03d8-e700-a8795022d42c: aborting pipeline replication: peer="{Voter server-066511b5-d290-19b8-8218-19c562551b35 066511b5-d290-19b8-8218-19c562551b35}"
2025-07-05T01:12:08.317Z [ERROR] server-734ef5a7-9786-03d8-e700-a8795022d42c: failed to heartbeat to: peer=814480c1-0285-8aa6-8062-05531567c518 backoff time=10ms error="command timed out"
--- PASS: TestRaft_VerifyLeader (0.68s)
=== RUN   TestRaft_VerifyLeader_Single
2025-07-05T01:12:08.318Z [INFO]  server-6094a47d-5e47-0a84-4c34-4fbc82403797: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6094a47d-5e47-0a84-4c34-4fbc82403797 Address:6094a47d-5e47-0a84-4c34-4fbc82403797}]"
2025-07-05T01:12:08.318Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:08.318Z [INFO]  server-6094a47d-5e47-0a84-4c34-4fbc82403797: entering follower state: follower="Node at 6094a47d-5e47-0a84-4c34-4fbc82403797 [Follower]" leader-address= leader-id=
2025-07-05T01:12:08.374Z [WARN]  server-6094a47d-5e47-0a84-4c34-4fbc82403797: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:08.374Z [INFO]  server-6094a47d-5e47-0a84-4c34-4fbc82403797: entering candidate state: node="Node at 6094a47d-5e47-0a84-4c34-4fbc82403797 [Candidate]" term=2
2025-07-05T01:12:08.374Z [INFO]  server-6094a47d-5e47-0a84-4c34-4fbc82403797: election won: term=2 tally=1
2025-07-05T01:12:08.374Z [INFO]  server-6094a47d-5e47-0a84-4c34-4fbc82403797: entering leader state: leader="Node at 6094a47d-5e47-0a84-4c34-4fbc82403797 [Leader]"
2025-07-05T01:12:08.480Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:08.374126886 +0000 UTC m=+103.622864094 (1 nodes), 55.82916ms from start of poll, 56.042369ms from cluster start. Timeout at 2025-07-05 01:12:08.480420847 +0000 UTC m=+103.729158055, 106.293961ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.16s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-07-05T01:12:08.481Z [INFO]  server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b Address:20d7d117-e1a1-67a7-dc9c-60c3b836d82b} {Suffrage:Voter ID:server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031 Address:06f30e1f-1f5d-e1c4-a415-0cdd8ab35031}]"
2025-07-05T01:12:08.482Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b Address:20d7d117-e1a1-67a7-dc9c-60c3b836d82b} {Suffrage:Voter ID:server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031 Address:06f30e1f-1f5d-e1c4-a415-0cdd8ab35031}]"
2025-07-05T01:12:08.482Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:08.485Z [INFO]  server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b: entering follower state: follower="Node at 20d7d117-e1a1-67a7-dc9c-60c3b836d82b [Follower]" leader-address= leader-id=
2025-07-05T01:12:08.485Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: entering follower state: follower="Node at 06f30e1f-1f5d-e1c4-a415-0cdd8ab35031 [Follower]" leader-address= leader-id=
2025-07-05T01:12:08.543Z [WARN]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:08.543Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: entering candidate state: node="Node at 06f30e1f-1f5d-e1c4-a415-0cdd8ab35031 [Candidate]" term=2
2025-07-05T01:12:08.544Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: election won: term=2 tally=2
2025-07-05T01:12:08.544Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: entering leader state: leader="Node at 06f30e1f-1f5d-e1c4-a415-0cdd8ab35031 [Leader]"
2025-07-05T01:12:08.544Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: added peer, starting replication: peer=server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b
2025-07-05T01:12:08.544Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: pipelining replication: peer="{Voter server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b 20d7d117-e1a1-67a7-dc9c-60c3b836d82b}"
2025-07-05T01:12:08.655Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:08.544191557 +0000 UTC m=+103.792928765 (1 nodes), 61.870707ms from start of poll, 62.627406ms from cluster start. Timeout at 2025-07-05 01:12:08.655676141 +0000 UTC m=+103.904413349, 111.484584ms after stability
2025-07-05T01:12:08.655Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:08.761Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:08.65576615 +0000 UTC m=+103.904503358 (1 nodes), 1.583µs from start of poll, 174.201999ms from cluster start. Timeout at 2025-07-05 01:12:08.761562828 +0000 UTC m=+104.010300036, 105.796678ms after stability
2025-07-05T01:12:08.771Z [ERROR] server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: peer has newer term, stopping replication: peer="{Voter server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b 20d7d117-e1a1-67a7-dc9c-60c3b836d82b}"
2025-07-05T01:12:08.771Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: aborting pipeline replication: peer="{Voter server-20d7d117-e1a1-67a7-dc9c-60c3b836d82b 20d7d117-e1a1-67a7-dc9c-60c3b836d82b}"
2025-07-05T01:12:08.771Z [INFO]  server-06f30e1f-1f5d-e1c4-a415-0cdd8ab35031: entering follower state: follower="Node at 06f30e1f-1f5d-e1c4-a415-0cdd8ab35031 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.29s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-07-05T01:12:08.772Z [INFO]  server-823bd7aa-6100-29e9-4949-226a6047c621: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823bd7aa-6100-29e9-4949-226a6047c621 Address:823bd7aa-6100-29e9-4949-226a6047c621} {Suffrage:Voter ID:server-33f730e0-1c13-3b0a-797e-af8c85aa50e8 Address:33f730e0-1c13-3b0a-797e-af8c85aa50e8} {Suffrage:Voter ID:server-650c2b31-683c-f904-86d6-7f18f351f82e Address:650c2b31-683c-f904-86d6-7f18f351f82e}]"
2025-07-05T01:12:08.772Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823bd7aa-6100-29e9-4949-226a6047c621 Address:823bd7aa-6100-29e9-4949-226a6047c621} {Suffrage:Voter ID:server-33f730e0-1c13-3b0a-797e-af8c85aa50e8 Address:33f730e0-1c13-3b0a-797e-af8c85aa50e8} {Suffrage:Voter ID:server-650c2b31-683c-f904-86d6-7f18f351f82e Address:650c2b31-683c-f904-86d6-7f18f351f82e}]"
2025-07-05T01:12:08.772Z [INFO]  server-650c2b31-683c-f904-86d6-7f18f351f82e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-823bd7aa-6100-29e9-4949-226a6047c621 Address:823bd7aa-6100-29e9-4949-226a6047c621} {Suffrage:Voter ID:server-33f730e0-1c13-3b0a-797e-af8c85aa50e8 Address:33f730e0-1c13-3b0a-797e-af8c85aa50e8} {Suffrage:Voter ID:server-650c2b31-683c-f904-86d6-7f18f351f82e Address:650c2b31-683c-f904-86d6-7f18f351f82e}]"
2025-07-05T01:12:08.772Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:08.772Z [INFO]  server-823bd7aa-6100-29e9-4949-226a6047c621: entering follower state: follower="Node at 823bd7aa-6100-29e9-4949-226a6047c621 [Follower]" leader-address= leader-id=
2025-07-05T01:12:08.772Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: entering follower state: follower="Node at 33f730e0-1c13-3b0a-797e-af8c85aa50e8 [Follower]" leader-address= leader-id=
2025-07-05T01:12:08.772Z [INFO]  server-650c2b31-683c-f904-86d6-7f18f351f82e: entering follower state: follower="Node at 650c2b31-683c-f904-86d6-7f18f351f82e [Follower]" leader-address= leader-id=
2025-07-05T01:12:08.830Z [WARN]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:08.830Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: entering candidate state: node="Node at 33f730e0-1c13-3b0a-797e-af8c85aa50e8 [Candidate]" term=2
2025-07-05T01:12:08.831Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: election won: term=2 tally=2
2025-07-05T01:12:08.831Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: entering leader state: leader="Node at 33f730e0-1c13-3b0a-797e-af8c85aa50e8 [Leader]"
2025-07-05T01:12:08.831Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: added peer, starting replication: peer=server-823bd7aa-6100-29e9-4949-226a6047c621
2025-07-05T01:12:08.831Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: added peer, starting replication: peer=server-650c2b31-683c-f904-86d6-7f18f351f82e
2025-07-05T01:12:08.831Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: pipelining replication: peer="{Voter server-650c2b31-683c-f904-86d6-7f18f351f82e 650c2b31-683c-f904-86d6-7f18f351f82e}"
2025-07-05T01:12:08.831Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: pipelining replication: peer="{Voter server-823bd7aa-6100-29e9-4949-226a6047c621 823bd7aa-6100-29e9-4949-226a6047c621}"
2025-07-05T01:12:08.937Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:08.831550886 +0000 UTC m=+104.080288094 (1 nodes), 59.143684ms from start of poll, 59.353418ms from cluster start. Timeout at 2025-07-05 01:12:08.937699845 +0000 UTC m=+104.186437053, 106.148959ms after stability
2025-07-05T01:12:08.943Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:09.048Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:08.943100672 +0000 UTC m=+104.191837880 (2 nodes), 1.754µs from start of poll, 170.903204ms from cluster start. Timeout at 2025-07-05 01:12:09.04864116 +0000 UTC m=+104.297378368, 105.540488ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 823bd7aa-6100-29e9-4949-226a6047c621 [Follower]
2025-07-05T01:12:09.048Z [ERROR] server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: failed to heartbeat to: peer=823bd7aa-6100-29e9-4949-226a6047c621 backoff time=10ms error="failed to connect to peer: 823bd7aa-6100-29e9-4949-226a6047c621"
2025-07-05T01:12:09.049Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: aborting pipeline replication: peer="{Voter server-650c2b31-683c-f904-86d6-7f18f351f82e 650c2b31-683c-f904-86d6-7f18f351f82e}"
2025-07-05T01:12:09.049Z [INFO]  server-33f730e0-1c13-3b0a-797e-af8c85aa50e8: aborting pipeline replication: peer="{Voter server-823bd7aa-6100-29e9-4949-226a6047c621 823bd7aa-6100-29e9-4949-226a6047c621}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s)
=== RUN   TestRaft_NotifyCh
2025-07-05T01:12:09.055Z [INFO]  server-9be9c2e8-a41d-b9af-9ca9-064d2410d07a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9be9c2e8-a41d-b9af-9ca9-064d2410d07a Address:9be9c2e8-a41d-b9af-9ca9-064d2410d07a}]"
2025-07-05T01:12:09.055Z [INFO]  server-9be9c2e8-a41d-b9af-9ca9-064d2410d07a: entering follower state: follower="Node at 9be9c2e8-a41d-b9af-9ca9-064d2410d07a [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.107Z [WARN]  server-9be9c2e8-a41d-b9af-9ca9-064d2410d07a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:09.108Z [INFO]  server-9be9c2e8-a41d-b9af-9ca9-064d2410d07a: entering candidate state: node="Node at 9be9c2e8-a41d-b9af-9ca9-064d2410d07a [Candidate]" term=2
2025-07-05T01:12:09.108Z [INFO]  server-9be9c2e8-a41d-b9af-9ca9-064d2410d07a: election won: term=2 tally=1
2025-07-05T01:12:09.108Z [INFO]  server-9be9c2e8-a41d-b9af-9ca9-064d2410d07a: entering leader state: leader="Node at 9be9c2e8-a41d-b9af-9ca9-064d2410d07a [Leader]"
--- PASS: TestRaft_NotifyCh (0.05s)
=== RUN   TestRaft_AppendEntry
2025-07-05T01:12:09.109Z [INFO]  server-c60ed44b-c243-d929-f936-252e801722c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60ed44b-c243-d929-f936-252e801722c6 Address:c60ed44b-c243-d929-f936-252e801722c6} {Suffrage:Voter ID:server-64ecd529-7d88-c50f-c6fe-26baee998d88 Address:64ecd529-7d88-c50f-c6fe-26baee998d88} {Suffrage:Voter ID:server-584f237a-165f-1b2b-4748-f6b851717b9d Address:584f237a-165f-1b2b-4748-f6b851717b9d}]"
2025-07-05T01:12:09.109Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60ed44b-c243-d929-f936-252e801722c6 Address:c60ed44b-c243-d929-f936-252e801722c6} {Suffrage:Voter ID:server-64ecd529-7d88-c50f-c6fe-26baee998d88 Address:64ecd529-7d88-c50f-c6fe-26baee998d88} {Suffrage:Voter ID:server-584f237a-165f-1b2b-4748-f6b851717b9d Address:584f237a-165f-1b2b-4748-f6b851717b9d}]"
2025-07-05T01:12:09.109Z [INFO]  server-584f237a-165f-1b2b-4748-f6b851717b9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60ed44b-c243-d929-f936-252e801722c6 Address:c60ed44b-c243-d929-f936-252e801722c6} {Suffrage:Voter ID:server-64ecd529-7d88-c50f-c6fe-26baee998d88 Address:64ecd529-7d88-c50f-c6fe-26baee998d88} {Suffrage:Voter ID:server-584f237a-165f-1b2b-4748-f6b851717b9d Address:584f237a-165f-1b2b-4748-f6b851717b9d}]"
2025-07-05T01:12:09.109Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:09.109Z [INFO]  server-c60ed44b-c243-d929-f936-252e801722c6: entering follower state: follower="Node at c60ed44b-c243-d929-f936-252e801722c6 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.109Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: entering follower state: follower="Node at 64ecd529-7d88-c50f-c6fe-26baee998d88 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.109Z [INFO]  server-584f237a-165f-1b2b-4748-f6b851717b9d: entering follower state: follower="Node at 584f237a-165f-1b2b-4748-f6b851717b9d [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.162Z [WARN]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:09.162Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: entering candidate state: node="Node at 64ecd529-7d88-c50f-c6fe-26baee998d88 [Candidate]" term=2
2025-07-05T01:12:09.162Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: election won: term=2 tally=2
2025-07-05T01:12:09.162Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: entering leader state: leader="Node at 64ecd529-7d88-c50f-c6fe-26baee998d88 [Leader]"
2025-07-05T01:12:09.162Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: added peer, starting replication: peer=server-c60ed44b-c243-d929-f936-252e801722c6
2025-07-05T01:12:09.162Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: added peer, starting replication: peer=server-584f237a-165f-1b2b-4748-f6b851717b9d
2025-07-05T01:12:09.162Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: pipelining replication: peer="{Voter server-584f237a-165f-1b2b-4748-f6b851717b9d 584f237a-165f-1b2b-4748-f6b851717b9d}"
2025-07-05T01:12:09.162Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: pipelining replication: peer="{Voter server-c60ed44b-c243-d929-f936-252e801722c6 c60ed44b-c243-d929-f936-252e801722c6}"
2025-07-05T01:12:09.268Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:09.162732311 +0000 UTC m=+104.411469519 (2 nodes), 53.615168ms from start of poll, 53.821295ms from cluster start. Timeout at 2025-07-05 01:12:09.268097911 +0000 UTC m=+104.516835119, 105.3656ms after stability
2025-07-05T01:12:09.268Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:09.373Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:09.2681561 +0000 UTC m=+104.516893309 (1 nodes), 1.694µs from start of poll, 159.245085ms from cluster start. Timeout at 2025-07-05 01:12:09.373602232 +0000 UTC m=+104.622339450, 105.446141ms after stability
2025-07-05T01:12:09.373Z [WARN]  server-c60ed44b-c243-d929-f936-252e801722c6: clearing log suffix: from=1 to=2
2025-07-05T01:12:09.374Z [WARN]  server-c60ed44b-c243-d929-f936-252e801722c6: skipping application of old log: index=1
2025-07-05T01:12:09.374Z [WARN]  server-c60ed44b-c243-d929-f936-252e801722c6: skipping application of old log: index=1
2025-07-05T01:12:09.375Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: aborting pipeline replication: peer="{Voter server-584f237a-165f-1b2b-4748-f6b851717b9d 584f237a-165f-1b2b-4748-f6b851717b9d}"
2025-07-05T01:12:09.375Z [INFO]  server-64ecd529-7d88-c50f-c6fe-26baee998d88: aborting pipeline replication: peer="{Voter server-c60ed44b-c243-d929-f936-252e801722c6 c60ed44b-c243-d929-f936-252e801722c6}"
--- PASS: TestRaft_AppendEntry (0.27s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-07-05T01:12:09.376Z [INFO]  server-36d8283c-1cbf-04ff-9195-5e282ff56a87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36d8283c-1cbf-04ff-9195-5e282ff56a87 Address:36d8283c-1cbf-04ff-9195-5e282ff56a87} {Suffrage:Voter ID:server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93 Address:e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93} {Suffrage:Voter ID:server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3 Address:4d968ac8-c6a4-fc88-cab0-4af8b4436bd3}]"
2025-07-05T01:12:09.377Z [INFO]  server-36d8283c-1cbf-04ff-9195-5e282ff56a87: entering follower state: follower="Node at 36d8283c-1cbf-04ff-9195-5e282ff56a87 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.377Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36d8283c-1cbf-04ff-9195-5e282ff56a87 Address:36d8283c-1cbf-04ff-9195-5e282ff56a87} {Suffrage:Voter ID:server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93 Address:e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93} {Suffrage:Voter ID:server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3 Address:4d968ac8-c6a4-fc88-cab0-4af8b4436bd3}]"
2025-07-05T01:12:09.377Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: entering follower state: follower="Node at e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.378Z [INFO]  server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36d8283c-1cbf-04ff-9195-5e282ff56a87 Address:36d8283c-1cbf-04ff-9195-5e282ff56a87} {Suffrage:Voter ID:server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93 Address:e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93} {Suffrage:Voter ID:server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3 Address:4d968ac8-c6a4-fc88-cab0-4af8b4436bd3}]"
2025-07-05T01:12:09.378Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:09.378Z [INFO]  server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3: entering follower state: follower="Node at 4d968ac8-c6a4-fc88-cab0-4af8b4436bd3 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.439Z [WARN]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:09.439Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: entering candidate state: node="Node at e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93 [Candidate]" term=2
2025-07-05T01:12:09.439Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: election won: term=2 tally=2
2025-07-05T01:12:09.439Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: entering leader state: leader="Node at e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93 [Leader]"
2025-07-05T01:12:09.439Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: added peer, starting replication: peer=server-36d8283c-1cbf-04ff-9195-5e282ff56a87
2025-07-05T01:12:09.439Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: added peer, starting replication: peer=server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3
2025-07-05T01:12:09.439Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: pipelining replication: peer="{Voter server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3 4d968ac8-c6a4-fc88-cab0-4af8b4436bd3}"
2025-07-05T01:12:09.439Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: pipelining replication: peer="{Voter server-36d8283c-1cbf-04ff-9195-5e282ff56a87 36d8283c-1cbf-04ff-9195-5e282ff56a87}"
2025-07-05T01:12:09.544Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:09.439716206 +0000 UTC m=+104.688453414 (2 nodes), 61.319575ms from start of poll, 63.243422ms from cluster start. Timeout at 2025-07-05 01:12:09.544803574 +0000 UTC m=+104.793540782, 105.087368ms after stability
2025-07-05T01:12:09.544Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:09.650Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:09.544858067 +0000 UTC m=+104.793595275 (1 nodes), 1.593µs from start of poll, 168.385283ms from cluster start. Timeout at 2025-07-05 01:12:09.650338252 +0000 UTC m=+104.899075460, 105.480185ms after stability
2025-07-05T01:12:09.650Z [WARN]  server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3: rejecting vote request since we have a leader: from=36d8283c-1cbf-04ff-9195-5e282ff56a87 leader=e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93 leader-id=server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93
2025-07-05T01:12:09.650Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: aborting pipeline replication: peer="{Voter server-36d8283c-1cbf-04ff-9195-5e282ff56a87 36d8283c-1cbf-04ff-9195-5e282ff56a87}"
2025-07-05T01:12:09.650Z [INFO]  server-e5bac6db-dde7-f8e9-3820-f9a2b8ba8f93: aborting pipeline replication: peer="{Voter server-4d968ac8-c6a4-fc88-cab0-4af8b4436bd3 4d968ac8-c6a4-fc88-cab0-4af8b4436bd3}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-07-05T01:12:09.651Z [INFO]  server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d Address:f5b82b3b-dd0f-ea27-4c48-653db1f8576d} {Suffrage:Voter ID:server-e775af9e-943b-9629-e695-9081024f0a29 Address:e775af9e-943b-9629-e695-9081024f0a29} {Suffrage:Voter ID:server-6895f148-7fe8-0106-98a5-43a6ba47c236 Address:6895f148-7fe8-0106-98a5-43a6ba47c236}]"
2025-07-05T01:12:09.651Z [INFO]  server-e775af9e-943b-9629-e695-9081024f0a29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d Address:f5b82b3b-dd0f-ea27-4c48-653db1f8576d} {Suffrage:Voter ID:server-e775af9e-943b-9629-e695-9081024f0a29 Address:e775af9e-943b-9629-e695-9081024f0a29} {Suffrage:Voter ID:server-6895f148-7fe8-0106-98a5-43a6ba47c236 Address:6895f148-7fe8-0106-98a5-43a6ba47c236}]"
2025-07-05T01:12:09.651Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d Address:f5b82b3b-dd0f-ea27-4c48-653db1f8576d} {Suffrage:Voter ID:server-e775af9e-943b-9629-e695-9081024f0a29 Address:e775af9e-943b-9629-e695-9081024f0a29} {Suffrage:Voter ID:server-6895f148-7fe8-0106-98a5-43a6ba47c236 Address:6895f148-7fe8-0106-98a5-43a6ba47c236}]"
2025-07-05T01:12:09.651Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:09.651Z [INFO]  server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d: entering follower state: follower="Node at f5b82b3b-dd0f-ea27-4c48-653db1f8576d [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.651Z [INFO]  server-e775af9e-943b-9629-e695-9081024f0a29: entering follower state: follower="Node at e775af9e-943b-9629-e695-9081024f0a29 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.651Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: entering follower state: follower="Node at 6895f148-7fe8-0106-98a5-43a6ba47c236 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.722Z [WARN]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:09.723Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: entering candidate state: node="Node at 6895f148-7fe8-0106-98a5-43a6ba47c236 [Candidate]" term=2
2025-07-05T01:12:09.723Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: election won: term=2 tally=2
2025-07-05T01:12:09.723Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: entering leader state: leader="Node at 6895f148-7fe8-0106-98a5-43a6ba47c236 [Leader]"
2025-07-05T01:12:09.723Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: added peer, starting replication: peer=server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d
2025-07-05T01:12:09.723Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: added peer, starting replication: peer=server-e775af9e-943b-9629-e695-9081024f0a29
2025-07-05T01:12:09.723Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: pipelining replication: peer="{Voter server-e775af9e-943b-9629-e695-9081024f0a29 e775af9e-943b-9629-e695-9081024f0a29}"
2025-07-05T01:12:09.723Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: pipelining replication: peer="{Voter server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d f5b82b3b-dd0f-ea27-4c48-653db1f8576d}"
2025-07-05T01:12:09.829Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:09.723475324 +0000 UTC m=+104.972212532 (2 nodes), 71.596654ms from start of poll, 71.803472ms from cluster start. Timeout at 2025-07-05 01:12:09.82922244 +0000 UTC m=+105.077959648, 105.747116ms after stability
2025-07-05T01:12:09.829Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:09.934Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:09.829277573 +0000 UTC m=+105.078014781 (1 nodes), 1.463µs from start of poll, 177.605721ms from cluster start. Timeout at 2025-07-05 01:12:09.934286014 +0000 UTC m=+105.183023222, 105.008441ms after stability
2025-07-05T01:12:09.934Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: aborting pipeline replication: peer="{Voter server-e775af9e-943b-9629-e695-9081024f0a29 e775af9e-943b-9629-e695-9081024f0a29}"
2025-07-05T01:12:09.934Z [INFO]  server-6895f148-7fe8-0106-98a5-43a6ba47c236: aborting pipeline replication: peer="{Voter server-f5b82b3b-dd0f-ea27-4c48-653db1f8576d f5b82b3b-dd0f-ea27-4c48-653db1f8576d}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-07-05T01:12:09.941Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e86096bb-4072-8f3e-fafd-840f4f1da0e5 Address:e86096bb-4072-8f3e-fafd-840f4f1da0e5} {Suffrage:Voter ID:server-4c59c101-9052-41ee-7dc9-6aec761691f8 Address:4c59c101-9052-41ee-7dc9-6aec761691f8} {Suffrage:Voter ID:server-e8315d1a-1cbe-fa93-f431-5b8018591c4a Address:e8315d1a-1cbe-fa93-f431-5b8018591c4a}]"
2025-07-05T01:12:09.947Z [INFO]  server-4c59c101-9052-41ee-7dc9-6aec761691f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e86096bb-4072-8f3e-fafd-840f4f1da0e5 Address:e86096bb-4072-8f3e-fafd-840f4f1da0e5} {Suffrage:Voter ID:server-4c59c101-9052-41ee-7dc9-6aec761691f8 Address:4c59c101-9052-41ee-7dc9-6aec761691f8} {Suffrage:Voter ID:server-e8315d1a-1cbe-fa93-f431-5b8018591c4a Address:e8315d1a-1cbe-fa93-f431-5b8018591c4a}]"
2025-07-05T01:12:09.947Z [INFO]  server-e8315d1a-1cbe-fa93-f431-5b8018591c4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e86096bb-4072-8f3e-fafd-840f4f1da0e5 Address:e86096bb-4072-8f3e-fafd-840f4f1da0e5} {Suffrage:Voter ID:server-4c59c101-9052-41ee-7dc9-6aec761691f8 Address:4c59c101-9052-41ee-7dc9-6aec761691f8} {Suffrage:Voter ID:server-e8315d1a-1cbe-fa93-f431-5b8018591c4a Address:e8315d1a-1cbe-fa93-f431-5b8018591c4a}]"
2025-07-05T01:12:09.947Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:09.947Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: entering follower state: follower="Node at e86096bb-4072-8f3e-fafd-840f4f1da0e5 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.947Z [INFO]  server-4c59c101-9052-41ee-7dc9-6aec761691f8: entering follower state: follower="Node at 4c59c101-9052-41ee-7dc9-6aec761691f8 [Follower]" leader-address= leader-id=
2025-07-05T01:12:09.947Z [INFO]  server-e8315d1a-1cbe-fa93-f431-5b8018591c4a: entering follower state: follower="Node at e8315d1a-1cbe-fa93-f431-5b8018591c4a [Follower]" leader-address= leader-id=
2025-07-05T01:12:10.011Z [WARN]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:10.011Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: entering candidate state: node="Node at e86096bb-4072-8f3e-fafd-840f4f1da0e5 [Candidate]" term=2
2025-07-05T01:12:10.011Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: election won: term=2 tally=2
2025-07-05T01:12:10.011Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: entering leader state: leader="Node at e86096bb-4072-8f3e-fafd-840f4f1da0e5 [Leader]"
2025-07-05T01:12:10.011Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: added peer, starting replication: peer=server-4c59c101-9052-41ee-7dc9-6aec761691f8
2025-07-05T01:12:10.011Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: added peer, starting replication: peer=server-e8315d1a-1cbe-fa93-f431-5b8018591c4a
2025-07-05T01:12:10.011Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: pipelining replication: peer="{Voter server-e8315d1a-1cbe-fa93-f431-5b8018591c4a e8315d1a-1cbe-fa93-f431-5b8018591c4a}"
2025-07-05T01:12:10.011Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: pipelining replication: peer="{Voter server-4c59c101-9052-41ee-7dc9-6aec761691f8 4c59c101-9052-41ee-7dc9-6aec761691f8}"
2025-07-05T01:12:10.117Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:10.011737946 +0000 UTC m=+105.260475154 (1 nodes), 64.638266ms from start of poll, 70.552235ms from cluster start. Timeout at 2025-07-05 01:12:10.117578547 +0000 UTC m=+105.366315755, 105.840601ms after stability
2025-07-05T01:12:10.117Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:10.223Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:10.117651073 +0000 UTC m=+105.366388281 (1 nodes), 611ns from start of poll, 176.465362ms from cluster start. Timeout at 2025-07-05 01:12:10.222976659 +0000 UTC m=+105.471713867, 105.325586ms after stability
2025-07-05T01:12:10.223Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:10.328Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:10.22330246 +0000 UTC m=+105.472039668 (1 nodes), 561ns from start of poll, 282.116749ms from cluster start. Timeout at 2025-07-05 01:12:10.328568493 +0000 UTC m=+105.577305711, 105.266043ms after stability
2025-07-05T01:12:10.328Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:10.439Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:10.328694219 +0000 UTC m=+105.577431427 (1 nodes), 461ns from start of poll, 387.508508ms from cluster start. Timeout at 2025-07-05 01:12:10.439220335 +0000 UTC m=+105.687957554, 110.526127ms after stability
2025-07-05T01:12:10.447Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:10.553Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:10.447395494 +0000 UTC m=+105.696132702 (1 nodes), 1.493µs from start of poll, 506.209783ms from cluster start. Timeout at 2025-07-05 01:12:10.553042412 +0000 UTC m=+105.801779620, 105.646918ms after stability
2025-07-05T01:12:10.553Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:10.658Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:10.553175602 +0000 UTC m=+105.801912810 (1 nodes), 541ns from start of poll, 611.989891ms from cluster start. Timeout at 2025-07-05 01:12:10.65865213 +0000 UTC m=+105.907389338, 105.476528ms after stability
2025-07-05T01:12:10.764Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:10.870Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:10.764283108 +0000 UTC m=+106.013020316 (1 nodes), 1.242µs from start of poll, 823.097397ms from cluster start. Timeout at 2025-07-05 01:12:10.87016122 +0000 UTC m=+106.118898428, 105.878112ms after stability
2025-07-05T01:12:10.870Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: aborting pipeline replication: peer="{Voter server-e8315d1a-1cbe-fa93-f431-5b8018591c4a e8315d1a-1cbe-fa93-f431-5b8018591c4a}"
2025-07-05T01:12:10.870Z [INFO]  server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: aborting pipeline replication: peer="{Voter server-4c59c101-9052-41ee-7dc9-6aec761691f8 4c59c101-9052-41ee-7dc9-6aec761691f8}"
2025-07-05T01:12:11.371Z [ERROR] server-e86096bb-4072-8f3e-fafd-840f4f1da0e5: failed to heartbeat to: peer=4c59c101-9052-41ee-7dc9-6aec761691f8 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeaderID_Propagated (1.43s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-07-05T01:12:11.372Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa Address:a9bcd379-1443-0fbf-f44c-e456ad91c7fa} {Suffrage:Voter ID:server-9b83d3a6-cdec-9c1b-7642-30ce09944859 Address:9b83d3a6-cdec-9c1b-7642-30ce09944859} {Suffrage:Voter ID:server-eea99390-53fd-f8f8-9761-993247da343a Address:eea99390-53fd-f8f8-9761-993247da343a}]"
2025-07-05T01:12:11.372Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa Address:a9bcd379-1443-0fbf-f44c-e456ad91c7fa} {Suffrage:Voter ID:server-9b83d3a6-cdec-9c1b-7642-30ce09944859 Address:9b83d3a6-cdec-9c1b-7642-30ce09944859} {Suffrage:Voter ID:server-eea99390-53fd-f8f8-9761-993247da343a Address:eea99390-53fd-f8f8-9761-993247da343a}]"
2025-07-05T01:12:11.372Z [INFO]  server-eea99390-53fd-f8f8-9761-993247da343a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa Address:a9bcd379-1443-0fbf-f44c-e456ad91c7fa} {Suffrage:Voter ID:server-9b83d3a6-cdec-9c1b-7642-30ce09944859 Address:9b83d3a6-cdec-9c1b-7642-30ce09944859} {Suffrage:Voter ID:server-eea99390-53fd-f8f8-9761-993247da343a Address:eea99390-53fd-f8f8-9761-993247da343a}]"
2025-07-05T01:12:11.372Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:11.372Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: entering follower state: follower="Node at a9bcd379-1443-0fbf-f44c-e456ad91c7fa [Follower]" leader-address= leader-id=
2025-07-05T01:12:11.372Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: entering follower state: follower="Node at 9b83d3a6-cdec-9c1b-7642-30ce09944859 [Follower]" leader-address= leader-id=
2025-07-05T01:12:11.372Z [INFO]  server-eea99390-53fd-f8f8-9761-993247da343a: entering follower state: follower="Node at eea99390-53fd-f8f8-9761-993247da343a [Follower]" leader-address= leader-id=
2025-07-05T01:12:11.439Z [WARN]  server-eea99390-53fd-f8f8-9761-993247da343a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:11.439Z [INFO]  server-eea99390-53fd-f8f8-9761-993247da343a: entering candidate state: node="Node at eea99390-53fd-f8f8-9761-993247da343a [Candidate]" term=2
2025-07-05T01:12:11.439Z [WARN]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:11.439Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: entering candidate state: node="Node at a9bcd379-1443-0fbf-f44c-e456ad91c7fa [Candidate]" term=2
2025-07-05T01:12:11.439Z [INFO]  server-eea99390-53fd-f8f8-9761-993247da343a: duplicate requestVote for same term: term=2
2025-07-05T01:12:11.439Z [WARN]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:11.439Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: entering candidate state: node="Node at 9b83d3a6-cdec-9c1b-7642-30ce09944859 [Candidate]" term=2
2025-07-05T01:12:11.439Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: duplicate requestVote for same term: term=2
2025-07-05T01:12:11.439Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: duplicate requestVote for same term: term=2
2025-07-05T01:12:11.439Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: duplicate requestVote for same term: term=2
2025-07-05T01:12:11.439Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: duplicate requestVote for same term: term=2
2025-07-05T01:12:11.439Z [INFO]  server-eea99390-53fd-f8f8-9761-993247da343a: duplicate requestVote for same term: term=2
2025-07-05T01:12:11.501Z [WARN]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: Election timeout reached, restarting election
2025-07-05T01:12:11.501Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: entering candidate state: node="Node at a9bcd379-1443-0fbf-f44c-e456ad91c7fa [Candidate]" term=3
2025-07-05T01:12:11.501Z [INFO]  server-eea99390-53fd-f8f8-9761-993247da343a: entering follower state: follower="Node at eea99390-53fd-f8f8-9761-993247da343a [Follower]" leader-address= leader-id=
2025-07-05T01:12:11.501Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: election won: term=3 tally=2
2025-07-05T01:12:11.501Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: entering leader state: leader="Node at a9bcd379-1443-0fbf-f44c-e456ad91c7fa [Leader]"
2025-07-05T01:12:11.501Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: added peer, starting replication: peer=server-9b83d3a6-cdec-9c1b-7642-30ce09944859
2025-07-05T01:12:11.501Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: added peer, starting replication: peer=server-eea99390-53fd-f8f8-9761-993247da343a
2025-07-05T01:12:11.501Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: entering follower state: follower="Node at 9b83d3a6-cdec-9c1b-7642-30ce09944859 [Follower]" leader-address= leader-id=
2025-07-05T01:12:11.501Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: pipelining replication: peer="{Voter server-eea99390-53fd-f8f8-9761-993247da343a eea99390-53fd-f8f8-9761-993247da343a}"
2025-07-05T01:12:11.501Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: pipelining replication: peer="{Voter server-9b83d3a6-cdec-9c1b-7642-30ce09944859 9b83d3a6-cdec-9c1b-7642-30ce09944859}"
2025-07-05T01:12:11.607Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:11.501371128 +0000 UTC m=+106.750108336 (1 nodes), 128.679414ms from start of poll, 128.953077ms from cluster start. Timeout at 2025-07-05 01:12:11.606986086 +0000 UTC m=+106.855723294, 105.614958ms after stability
2025-07-05T01:12:11.607Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:11.715Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:11.607048092 +0000 UTC m=+106.855785300 (1 nodes), 952ns from start of poll, 234.630041ms from cluster start. Timeout at 2025-07-05 01:12:11.714926163 +0000 UTC m=+106.963663372, 107.878072ms after stability
2025-07-05T01:12:11.715Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: entering candidate state: node="Node at 9b83d3a6-cdec-9c1b-7642-30ce09944859 [Candidate]" term=4
2025-07-05T01:12:11.715Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: election won: term=4 tally=2
2025-07-05T01:12:11.715Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: entering leader state: leader="Node at 9b83d3a6-cdec-9c1b-7642-30ce09944859 [Leader]"
2025-07-05T01:12:11.715Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: added peer, starting replication: peer=server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa
2025-07-05T01:12:11.715Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: added peer, starting replication: peer=server-eea99390-53fd-f8f8-9761-993247da343a
2025-07-05T01:12:11.715Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:11.715Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: pipelining replication: peer="{Voter server-eea99390-53fd-f8f8-9761-993247da343a eea99390-53fd-f8f8-9761-993247da343a}"
2025-07-05T01:12:11.715Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: entering follower state: follower="Node at a9bcd379-1443-0fbf-f44c-e456ad91c7fa [Follower]" leader-address=9b83d3a6-cdec-9c1b-7642-30ce09944859 leader-id=server-9b83d3a6-cdec-9c1b-7642-30ce09944859
2025-07-05T01:12:11.715Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: aborting pipeline replication: peer="{Voter server-eea99390-53fd-f8f8-9761-993247da343a eea99390-53fd-f8f8-9761-993247da343a}"
2025-07-05T01:12:11.715Z [WARN]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: rejecting vote request since our last term is greater: candidate=9b83d3a6-cdec-9c1b-7642-30ce09944859 last-term=4 last-candidate-term=3
2025-07-05T01:12:11.715Z [INFO]  server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa: aborting pipeline replication: peer="{Voter server-9b83d3a6-cdec-9c1b-7642-30ce09944859 9b83d3a6-cdec-9c1b-7642-30ce09944859}"
2025-07-05T01:12:11.715Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: pipelining replication: peer="{Voter server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa a9bcd379-1443-0fbf-f44c-e456ad91c7fa}"
2025-07-05T01:12:11.821Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:11.715310434 +0000 UTC m=+106.964047642 (1 nodes), 91.091µs from start of poll, 342.892383ms from cluster start. Timeout at 2025-07-05 01:12:11.821333838 +0000 UTC m=+107.070071046, 106.023404ms after stability
2025-07-05T01:12:11.821Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: aborting pipeline replication: peer="{Voter server-a9bcd379-1443-0fbf-f44c-e456ad91c7fa a9bcd379-1443-0fbf-f44c-e456ad91c7fa}"
2025-07-05T01:12:11.821Z [INFO]  server-9b83d3a6-cdec-9c1b-7642-30ce09944859: aborting pipeline replication: peer="{Voter server-eea99390-53fd-f8f8-9761-993247da343a eea99390-53fd-f8f8-9761-993247da343a}"
--- PASS: TestRaft_LeadershipTransfer (0.45s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-07-05T01:12:11.822Z [INFO]  server-21fc211e-4e12-5d30-4a71-83bc049487a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21fc211e-4e12-5d30-4a71-83bc049487a6 Address:21fc211e-4e12-5d30-4a71-83bc049487a6}]"
2025-07-05T01:12:11.822Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:11.822Z [INFO]  server-21fc211e-4e12-5d30-4a71-83bc049487a6: entering follower state: follower="Node at 21fc211e-4e12-5d30-4a71-83bc049487a6 [Follower]" leader-address= leader-id=
2025-07-05T01:12:11.888Z [WARN]  server-21fc211e-4e12-5d30-4a71-83bc049487a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:11.888Z [INFO]  server-21fc211e-4e12-5d30-4a71-83bc049487a6: entering candidate state: node="Node at 21fc211e-4e12-5d30-4a71-83bc049487a6 [Candidate]" term=2
2025-07-05T01:12:11.888Z [INFO]  server-21fc211e-4e12-5d30-4a71-83bc049487a6: election won: term=2 tally=1
2025-07-05T01:12:11.888Z [INFO]  server-21fc211e-4e12-5d30-4a71-83bc049487a6: entering leader state: leader="Node at 21fc211e-4e12-5d30-4a71-83bc049487a6 [Leader]"
2025-07-05T01:12:11.999Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:11.888815296 +0000 UTC m=+107.137552504 (1 nodes), 66.431509ms from start of poll, 66.543049ms from cluster start. Timeout at 2025-07-05 01:12:11.994086038 +0000 UTC m=+107.242823256, 105.270752ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-07-05T01:12:12.000Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ed3828-510b-88c5-607d-4c376928b3de Address:05ed3828-510b-88c5-607d-4c376928b3de} {Suffrage:Voter ID:server-bb1915d4-d582-9268-3b01-f83c9dd94e86 Address:bb1915d4-d582-9268-3b01-f83c9dd94e86} {Suffrage:Voter ID:server-97834e6b-06e4-5f26-2477-27ec5895e7a8 Address:97834e6b-06e4-5f26-2477-27ec5895e7a8} {Suffrage:Voter ID:server-4a71b554-b66f-39c5-e907-01687c6d9abf Address:4a71b554-b66f-39c5-e907-01687c6d9abf} {Suffrage:Voter ID:server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed Address:dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed} {Suffrage:Voter ID:server-339b8ce0-53ec-bee1-0e22-c218cbec31bd Address:339b8ce0-53ec-bee1-0e22-c218cbec31bd} {Suffrage:Voter ID:server-843d3dc5-8915-fc39-5085-fd219beab8d2 Address:843d3dc5-8915-fc39-5085-fd219beab8d2}]"
2025-07-05T01:12:12.001Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ed3828-510b-88c5-607d-4c376928b3de Address:05ed3828-510b-88c5-607d-4c376928b3de} {Suffrage:Voter ID:server-bb1915d4-d582-9268-3b01-f83c9dd94e86 Address:bb1915d4-d582-9268-3b01-f83c9dd94e86} {Suffrage:Voter ID:server-97834e6b-06e4-5f26-2477-27ec5895e7a8 Address:97834e6b-06e4-5f26-2477-27ec5895e7a8} {Suffrage:Voter ID:server-4a71b554-b66f-39c5-e907-01687c6d9abf Address:4a71b554-b66f-39c5-e907-01687c6d9abf} {Suffrage:Voter ID:server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed Address:dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed} {Suffrage:Voter ID:server-339b8ce0-53ec-bee1-0e22-c218cbec31bd Address:339b8ce0-53ec-bee1-0e22-c218cbec31bd} {Suffrage:Voter ID:server-843d3dc5-8915-fc39-5085-fd219beab8d2 Address:843d3dc5-8915-fc39-5085-fd219beab8d2}]"
2025-07-05T01:12:12.001Z [INFO]  server-97834e6b-06e4-5f26-2477-27ec5895e7a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ed3828-510b-88c5-607d-4c376928b3de Address:05ed3828-510b-88c5-607d-4c376928b3de} {Suffrage:Voter ID:server-bb1915d4-d582-9268-3b01-f83c9dd94e86 Address:bb1915d4-d582-9268-3b01-f83c9dd94e86} {Suffrage:Voter ID:server-97834e6b-06e4-5f26-2477-27ec5895e7a8 Address:97834e6b-06e4-5f26-2477-27ec5895e7a8} {Suffrage:Voter ID:server-4a71b554-b66f-39c5-e907-01687c6d9abf Address:4a71b554-b66f-39c5-e907-01687c6d9abf} {Suffrage:Voter ID:server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed Address:dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed} {Suffrage:Voter ID:server-339b8ce0-53ec-bee1-0e22-c218cbec31bd Address:339b8ce0-53ec-bee1-0e22-c218cbec31bd} {Suffrage:Voter ID:server-843d3dc5-8915-fc39-5085-fd219beab8d2 Address:843d3dc5-8915-fc39-5085-fd219beab8d2}]"
2025-07-05T01:12:12.001Z [INFO]  server-4a71b554-b66f-39c5-e907-01687c6d9abf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ed3828-510b-88c5-607d-4c376928b3de Address:05ed3828-510b-88c5-607d-4c376928b3de} {Suffrage:Voter ID:server-bb1915d4-d582-9268-3b01-f83c9dd94e86 Address:bb1915d4-d582-9268-3b01-f83c9dd94e86} {Suffrage:Voter ID:server-97834e6b-06e4-5f26-2477-27ec5895e7a8 Address:97834e6b-06e4-5f26-2477-27ec5895e7a8} {Suffrage:Voter ID:server-4a71b554-b66f-39c5-e907-01687c6d9abf Address:4a71b554-b66f-39c5-e907-01687c6d9abf} {Suffrage:Voter ID:server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed Address:dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed} {Suffrage:Voter ID:server-339b8ce0-53ec-bee1-0e22-c218cbec31bd Address:339b8ce0-53ec-bee1-0e22-c218cbec31bd} {Suffrage:Voter ID:server-843d3dc5-8915-fc39-5085-fd219beab8d2 Address:843d3dc5-8915-fc39-5085-fd219beab8d2}]"
2025-07-05T01:12:12.001Z [INFO]  server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ed3828-510b-88c5-607d-4c376928b3de Address:05ed3828-510b-88c5-607d-4c376928b3de} {Suffrage:Voter ID:server-bb1915d4-d582-9268-3b01-f83c9dd94e86 Address:bb1915d4-d582-9268-3b01-f83c9dd94e86} {Suffrage:Voter ID:server-97834e6b-06e4-5f26-2477-27ec5895e7a8 Address:97834e6b-06e4-5f26-2477-27ec5895e7a8} {Suffrage:Voter ID:server-4a71b554-b66f-39c5-e907-01687c6d9abf Address:4a71b554-b66f-39c5-e907-01687c6d9abf} {Suffrage:Voter ID:server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed Address:dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed} {Suffrage:Voter ID:server-339b8ce0-53ec-bee1-0e22-c218cbec31bd Address:339b8ce0-53ec-bee1-0e22-c218cbec31bd} {Suffrage:Voter ID:server-843d3dc5-8915-fc39-5085-fd219beab8d2 Address:843d3dc5-8915-fc39-5085-fd219beab8d2}]"
2025-07-05T01:12:12.001Z [INFO]  server-339b8ce0-53ec-bee1-0e22-c218cbec31bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ed3828-510b-88c5-607d-4c376928b3de Address:05ed3828-510b-88c5-607d-4c376928b3de} {Suffrage:Voter ID:server-bb1915d4-d582-9268-3b01-f83c9dd94e86 Address:bb1915d4-d582-9268-3b01-f83c9dd94e86} {Suffrage:Voter ID:server-97834e6b-06e4-5f26-2477-27ec5895e7a8 Address:97834e6b-06e4-5f26-2477-27ec5895e7a8} {Suffrage:Voter ID:server-4a71b554-b66f-39c5-e907-01687c6d9abf Address:4a71b554-b66f-39c5-e907-01687c6d9abf} {Suffrage:Voter ID:server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed Address:dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed} {Suffrage:Voter ID:server-339b8ce0-53ec-bee1-0e22-c218cbec31bd Address:339b8ce0-53ec-bee1-0e22-c218cbec31bd} {Suffrage:Voter ID:server-843d3dc5-8915-fc39-5085-fd219beab8d2 Address:843d3dc5-8915-fc39-5085-fd219beab8d2}]"
2025-07-05T01:12:12.001Z [INFO]  server-843d3dc5-8915-fc39-5085-fd219beab8d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ed3828-510b-88c5-607d-4c376928b3de Address:05ed3828-510b-88c5-607d-4c376928b3de} {Suffrage:Voter ID:server-bb1915d4-d582-9268-3b01-f83c9dd94e86 Address:bb1915d4-d582-9268-3b01-f83c9dd94e86} {Suffrage:Voter ID:server-97834e6b-06e4-5f26-2477-27ec5895e7a8 Address:97834e6b-06e4-5f26-2477-27ec5895e7a8} {Suffrage:Voter ID:server-4a71b554-b66f-39c5-e907-01687c6d9abf Address:4a71b554-b66f-39c5-e907-01687c6d9abf} {Suffrage:Voter ID:server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed Address:dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed} {Suffrage:Voter ID:server-339b8ce0-53ec-bee1-0e22-c218cbec31bd Address:339b8ce0-53ec-bee1-0e22-c218cbec31bd} {Suffrage:Voter ID:server-843d3dc5-8915-fc39-5085-fd219beab8d2 Address:843d3dc5-8915-fc39-5085-fd219beab8d2}]"
2025-07-05T01:12:12.001Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:12.001Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: entering follower state: follower="Node at 05ed3828-510b-88c5-607d-4c376928b3de [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.001Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: entering follower state: follower="Node at bb1915d4-d582-9268-3b01-f83c9dd94e86 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.001Z [INFO]  server-97834e6b-06e4-5f26-2477-27ec5895e7a8: entering follower state: follower="Node at 97834e6b-06e4-5f26-2477-27ec5895e7a8 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.001Z [INFO]  server-4a71b554-b66f-39c5-e907-01687c6d9abf: entering follower state: follower="Node at 4a71b554-b66f-39c5-e907-01687c6d9abf [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.001Z [INFO]  server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed: entering follower state: follower="Node at dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.001Z [INFO]  server-339b8ce0-53ec-bee1-0e22-c218cbec31bd: entering follower state: follower="Node at 339b8ce0-53ec-bee1-0e22-c218cbec31bd [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.001Z [INFO]  server-843d3dc5-8915-fc39-5085-fd219beab8d2: entering follower state: follower="Node at 843d3dc5-8915-fc39-5085-fd219beab8d2 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.059Z [WARN]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: entering candidate state: node="Node at bb1915d4-d582-9268-3b01-f83c9dd94e86 [Candidate]" term=2
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: election won: term=2 tally=4
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: entering leader state: leader="Node at bb1915d4-d582-9268-3b01-f83c9dd94e86 [Leader]"
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: added peer, starting replication: peer=server-05ed3828-510b-88c5-607d-4c376928b3de
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: added peer, starting replication: peer=server-97834e6b-06e4-5f26-2477-27ec5895e7a8
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: added peer, starting replication: peer=server-4a71b554-b66f-39c5-e907-01687c6d9abf
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: added peer, starting replication: peer=server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: added peer, starting replication: peer=server-339b8ce0-53ec-bee1-0e22-c218cbec31bd
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: added peer, starting replication: peer=server-843d3dc5-8915-fc39-5085-fd219beab8d2
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: pipelining replication: peer="{Voter server-843d3dc5-8915-fc39-5085-fd219beab8d2 843d3dc5-8915-fc39-5085-fd219beab8d2}"
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: pipelining replication: peer="{Voter server-05ed3828-510b-88c5-607d-4c376928b3de 05ed3828-510b-88c5-607d-4c376928b3de}"
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: pipelining replication: peer="{Voter server-97834e6b-06e4-5f26-2477-27ec5895e7a8 97834e6b-06e4-5f26-2477-27ec5895e7a8}"
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: pipelining replication: peer="{Voter server-4a71b554-b66f-39c5-e907-01687c6d9abf 4a71b554-b66f-39c5-e907-01687c6d9abf}"
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: pipelining replication: peer="{Voter server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed}"
2025-07-05T01:12:12.059Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: pipelining replication: peer="{Voter server-339b8ce0-53ec-bee1-0e22-c218cbec31bd 339b8ce0-53ec-bee1-0e22-c218cbec31bd}"
2025-07-05T01:12:12.165Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:12.059412265 +0000 UTC m=+107.308149473 (1 nodes), 58.125736ms from start of poll, 58.580408ms from cluster start. Timeout at 2025-07-05 01:12:12.16535615 +0000 UTC m=+107.414093368, 105.943895ms after stability
2025-07-05T01:12:12.165Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:12.271Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:12.165412415 +0000 UTC m=+107.414149623 (6 nodes), 1.913µs from start of poll, 164.580558ms from cluster start. Timeout at 2025-07-05 01:12:12.270887801 +0000 UTC m=+107.519625009, 105.475386ms after stability
2025-07-05T01:12:12.271Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:12.376Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:12.271039907 +0000 UTC m=+107.519777115 (1 nodes), 1.523µs from start of poll, 270.20805ms from cluster start. Timeout at 2025-07-05 01:12:12.376253983 +0000 UTC m=+107.624991201, 105.214086ms after stability
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: entering candidate state: node="Node at 05ed3828-510b-88c5-607d-4c376928b3de [Candidate]" term=3
2025-07-05T01:12:12.376Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:12.376Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: entering follower state: follower="Node at bb1915d4-d582-9268-3b01-f83c9dd94e86 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.376Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: aborting pipeline replication: peer="{Voter server-843d3dc5-8915-fc39-5085-fd219beab8d2 843d3dc5-8915-fc39-5085-fd219beab8d2}"
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: election won: term=3 tally=4
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: entering leader state: leader="Node at 05ed3828-510b-88c5-607d-4c376928b3de [Leader]"
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: added peer, starting replication: peer=server-bb1915d4-d582-9268-3b01-f83c9dd94e86
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: added peer, starting replication: peer=server-97834e6b-06e4-5f26-2477-27ec5895e7a8
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: added peer, starting replication: peer=server-4a71b554-b66f-39c5-e907-01687c6d9abf
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: added peer, starting replication: peer=server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: added peer, starting replication: peer=server-339b8ce0-53ec-bee1-0e22-c218cbec31bd
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: added peer, starting replication: peer=server-843d3dc5-8915-fc39-5085-fd219beab8d2
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: pipelining replication: peer="{Voter server-843d3dc5-8915-fc39-5085-fd219beab8d2 843d3dc5-8915-fc39-5085-fd219beab8d2}"
2025-07-05T01:12:12.376Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: aborting pipeline replication: peer="{Voter server-05ed3828-510b-88c5-607d-4c376928b3de 05ed3828-510b-88c5-607d-4c376928b3de}"
2025-07-05T01:12:12.376Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: aborting pipeline replication: peer="{Voter server-97834e6b-06e4-5f26-2477-27ec5895e7a8 97834e6b-06e4-5f26-2477-27ec5895e7a8}"
2025-07-05T01:12:12.376Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: aborting pipeline replication: peer="{Voter server-4a71b554-b66f-39c5-e907-01687c6d9abf 4a71b554-b66f-39c5-e907-01687c6d9abf}"
2025-07-05T01:12:12.376Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: aborting pipeline replication: peer="{Voter server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed}"
2025-07-05T01:12:12.376Z [INFO]  server-bb1915d4-d582-9268-3b01-f83c9dd94e86: aborting pipeline replication: peer="{Voter server-339b8ce0-53ec-bee1-0e22-c218cbec31bd 339b8ce0-53ec-bee1-0e22-c218cbec31bd}"
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: pipelining replication: peer="{Voter server-bb1915d4-d582-9268-3b01-f83c9dd94e86 bb1915d4-d582-9268-3b01-f83c9dd94e86}"
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: pipelining replication: peer="{Voter server-97834e6b-06e4-5f26-2477-27ec5895e7a8 97834e6b-06e4-5f26-2477-27ec5895e7a8}"
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: pipelining replication: peer="{Voter server-4a71b554-b66f-39c5-e907-01687c6d9abf 4a71b554-b66f-39c5-e907-01687c6d9abf}"
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: pipelining replication: peer="{Voter server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed}"
2025-07-05T01:12:12.376Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: pipelining replication: peer="{Voter server-339b8ce0-53ec-bee1-0e22-c218cbec31bd 339b8ce0-53ec-bee1-0e22-c218cbec31bd}"
2025-07-05T01:12:12.483Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:12.376659072 +0000 UTC m=+107.625396280 (1 nodes), 203.02µs from start of poll, 375.827215ms from cluster start. Timeout at 2025-07-05 01:12:12.482966178 +0000 UTC m=+107.731703386, 106.307106ms after stability
2025-07-05T01:12:12.483Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: aborting pipeline replication: peer="{Voter server-4a71b554-b66f-39c5-e907-01687c6d9abf 4a71b554-b66f-39c5-e907-01687c6d9abf}"
2025-07-05T01:12:12.483Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: aborting pipeline replication: peer="{Voter server-dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed dce06f3a-5f1c-4bb3-94e9-c5d78df0a8ed}"
2025-07-05T01:12:12.483Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: aborting pipeline replication: peer="{Voter server-97834e6b-06e4-5f26-2477-27ec5895e7a8 97834e6b-06e4-5f26-2477-27ec5895e7a8}"
2025-07-05T01:12:12.483Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: aborting pipeline replication: peer="{Voter server-bb1915d4-d582-9268-3b01-f83c9dd94e86 bb1915d4-d582-9268-3b01-f83c9dd94e86}"
2025-07-05T01:12:12.483Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: aborting pipeline replication: peer="{Voter server-843d3dc5-8915-fc39-5085-fd219beab8d2 843d3dc5-8915-fc39-5085-fd219beab8d2}"
2025-07-05T01:12:12.483Z [INFO]  server-05ed3828-510b-88c5-607d-4c376928b3de: aborting pipeline replication: peer="{Voter server-339b8ce0-53ec-bee1-0e22-c218cbec31bd 339b8ce0-53ec-bee1-0e22-c218cbec31bd}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-07-05T01:12:12.484Z [INFO]  server-4d62044a-8817-c5aa-c631-0a7faec0c82c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d62044a-8817-c5aa-c631-0a7faec0c82c Address:4d62044a-8817-c5aa-c631-0a7faec0c82c} {Suffrage:Voter ID:server-7f29cac3-166e-6c75-dc80-e40d0e2b5246 Address:7f29cac3-166e-6c75-dc80-e40d0e2b5246} {Suffrage:Voter ID:server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c Address:8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c}]"
2025-07-05T01:12:12.484Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d62044a-8817-c5aa-c631-0a7faec0c82c Address:4d62044a-8817-c5aa-c631-0a7faec0c82c} {Suffrage:Voter ID:server-7f29cac3-166e-6c75-dc80-e40d0e2b5246 Address:7f29cac3-166e-6c75-dc80-e40d0e2b5246} {Suffrage:Voter ID:server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c Address:8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c}]"
2025-07-05T01:12:12.484Z [INFO]  server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d62044a-8817-c5aa-c631-0a7faec0c82c Address:4d62044a-8817-c5aa-c631-0a7faec0c82c} {Suffrage:Voter ID:server-7f29cac3-166e-6c75-dc80-e40d0e2b5246 Address:7f29cac3-166e-6c75-dc80-e40d0e2b5246} {Suffrage:Voter ID:server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c Address:8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c}]"
2025-07-05T01:12:12.484Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:12.484Z [INFO]  server-4d62044a-8817-c5aa-c631-0a7faec0c82c: entering follower state: follower="Node at 4d62044a-8817-c5aa-c631-0a7faec0c82c [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.484Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: entering follower state: follower="Node at 7f29cac3-166e-6c75-dc80-e40d0e2b5246 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.484Z [INFO]  server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c: entering follower state: follower="Node at 8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.552Z [WARN]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:12.552Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: entering candidate state: node="Node at 7f29cac3-166e-6c75-dc80-e40d0e2b5246 [Candidate]" term=2
2025-07-05T01:12:12.552Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: election won: term=2 tally=2
2025-07-05T01:12:12.552Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: entering leader state: leader="Node at 7f29cac3-166e-6c75-dc80-e40d0e2b5246 [Leader]"
2025-07-05T01:12:12.552Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: added peer, starting replication: peer=server-4d62044a-8817-c5aa-c631-0a7faec0c82c
2025-07-05T01:12:12.552Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: added peer, starting replication: peer=server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c
2025-07-05T01:12:12.552Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: pipelining replication: peer="{Voter server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c 8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c}"
2025-07-05T01:12:12.552Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: pipelining replication: peer="{Voter server-4d62044a-8817-c5aa-c631-0a7faec0c82c 4d62044a-8817-c5aa-c631-0a7faec0c82c}"
2025-07-05T01:12:12.657Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:12.552603288 +0000 UTC m=+107.801340496 (1 nodes), 67.668308ms from start of poll, 67.95214ms from cluster start. Timeout at 2025-07-05 01:12:12.657640833 +0000 UTC m=+107.906378041, 105.037545ms after stability
2025-07-05T01:12:12.657Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: aborting pipeline replication: peer="{Voter server-8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c 8b03312b-7b4d-97b1-b6ec-57dbd4f21e7c}"
2025-07-05T01:12:12.657Z [INFO]  server-7f29cac3-166e-6c75-dc80-e40d0e2b5246: aborting pipeline replication: peer="{Voter server-4d62044a-8817-c5aa-c631-0a7faec0c82c 4d62044a-8817-c5aa-c631-0a7faec0c82c}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-07-05T01:12:12.658Z [INFO]  server-2230a96f-bcdf-5842-4222-9edb4650a987: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2230a96f-bcdf-5842-4222-9edb4650a987 Address:2230a96f-bcdf-5842-4222-9edb4650a987} {Suffrage:Voter ID:server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8 Address:10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8} {Suffrage:Voter ID:server-78a9aff2-5e14-593e-79a6-781530457136 Address:78a9aff2-5e14-593e-79a6-781530457136}]"
2025-07-05T01:12:12.658Z [INFO]  server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2230a96f-bcdf-5842-4222-9edb4650a987 Address:2230a96f-bcdf-5842-4222-9edb4650a987} {Suffrage:Voter ID:server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8 Address:10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8} {Suffrage:Voter ID:server-78a9aff2-5e14-593e-79a6-781530457136 Address:78a9aff2-5e14-593e-79a6-781530457136}]"
2025-07-05T01:12:12.658Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2230a96f-bcdf-5842-4222-9edb4650a987 Address:2230a96f-bcdf-5842-4222-9edb4650a987} {Suffrage:Voter ID:server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8 Address:10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8} {Suffrage:Voter ID:server-78a9aff2-5e14-593e-79a6-781530457136 Address:78a9aff2-5e14-593e-79a6-781530457136}]"
2025-07-05T01:12:12.658Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:12.658Z [INFO]  server-2230a96f-bcdf-5842-4222-9edb4650a987: entering follower state: follower="Node at 2230a96f-bcdf-5842-4222-9edb4650a987 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.658Z [INFO]  server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8: entering follower state: follower="Node at 10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.658Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: entering follower state: follower="Node at 78a9aff2-5e14-593e-79a6-781530457136 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.733Z [WARN]  server-78a9aff2-5e14-593e-79a6-781530457136: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:12.733Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: entering candidate state: node="Node at 78a9aff2-5e14-593e-79a6-781530457136 [Candidate]" term=2
2025-07-05T01:12:12.733Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: election won: term=2 tally=2
2025-07-05T01:12:12.733Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: entering leader state: leader="Node at 78a9aff2-5e14-593e-79a6-781530457136 [Leader]"
2025-07-05T01:12:12.733Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: added peer, starting replication: peer=server-2230a96f-bcdf-5842-4222-9edb4650a987
2025-07-05T01:12:12.733Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: added peer, starting replication: peer=server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8
2025-07-05T01:12:12.733Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: pipelining replication: peer="{Voter server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8 10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8}"
2025-07-05T01:12:12.733Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: pipelining replication: peer="{Voter server-2230a96f-bcdf-5842-4222-9edb4650a987 2230a96f-bcdf-5842-4222-9edb4650a987}"
2025-07-05T01:12:12.839Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:12.733555233 +0000 UTC m=+107.982292441 (2 nodes), 74.630232ms from start of poll, 74.862568ms from cluster start. Timeout at 2025-07-05 01:12:12.839344718 +0000 UTC m=+108.088081926, 105.789485ms after stability
2025-07-05T01:12:12.839Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:12.945Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:12.839629672 +0000 UTC m=+108.088366880 (1 nodes), 831ns from start of poll, 180.937007ms from cluster start. Timeout at 2025-07-05 01:12:12.945446098 +0000 UTC m=+108.194183316, 105.816436ms after stability
2025-07-05T01:12:12.945Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: aborting pipeline replication: peer="{Voter server-10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8 10e5e53e-2662-ca37-fbdb-0b3cd1fb3df8}"
2025-07-05T01:12:12.945Z [INFO]  server-78a9aff2-5e14-593e-79a6-781530457136: aborting pipeline replication: peer="{Voter server-2230a96f-bcdf-5842-4222-9edb4650a987 2230a96f-bcdf-5842-4222-9edb4650a987}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-07-05T01:12:12.946Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a218718e-0587-74a2-0b37-690026dd9ecd Address:a218718e-0587-74a2-0b37-690026dd9ecd} {Suffrage:Voter ID:server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222 Address:76e7030d-f8ce-6ee6-28d1-5b5e1e251222} {Suffrage:Voter ID:server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 Address:5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}]"
2025-07-05T01:12:12.946Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a218718e-0587-74a2-0b37-690026dd9ecd Address:a218718e-0587-74a2-0b37-690026dd9ecd} {Suffrage:Voter ID:server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222 Address:76e7030d-f8ce-6ee6-28d1-5b5e1e251222} {Suffrage:Voter ID:server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 Address:5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}]"
2025-07-05T01:12:12.946Z [INFO]  server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a218718e-0587-74a2-0b37-690026dd9ecd Address:a218718e-0587-74a2-0b37-690026dd9ecd} {Suffrage:Voter ID:server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222 Address:76e7030d-f8ce-6ee6-28d1-5b5e1e251222} {Suffrage:Voter ID:server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 Address:5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}]"
2025-07-05T01:12:12.946Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:12.946Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: entering follower state: follower="Node at a218718e-0587-74a2-0b37-690026dd9ecd [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.946Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: entering follower state: follower="Node at 76e7030d-f8ce-6ee6-28d1-5b5e1e251222 [Follower]" leader-address= leader-id=
2025-07-05T01:12:12.946Z [INFO]  server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3: entering follower state: follower="Node at 5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 [Follower]" leader-address= leader-id=
2025-07-05T01:12:13.013Z [WARN]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:13.013Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: entering candidate state: node="Node at 76e7030d-f8ce-6ee6-28d1-5b5e1e251222 [Candidate]" term=2
2025-07-05T01:12:13.013Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: election won: term=2 tally=2
2025-07-05T01:12:13.013Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: entering leader state: leader="Node at 76e7030d-f8ce-6ee6-28d1-5b5e1e251222 [Leader]"
2025-07-05T01:12:13.013Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: added peer, starting replication: peer=server-a218718e-0587-74a2-0b37-690026dd9ecd
2025-07-05T01:12:13.013Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: added peer, starting replication: peer=server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3
2025-07-05T01:12:13.013Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: pipelining replication: peer="{Voter server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}"
2025-07-05T01:12:13.013Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: pipelining replication: peer="{Voter server-a218718e-0587-74a2-0b37-690026dd9ecd a218718e-0587-74a2-0b37-690026dd9ecd}"
2025-07-05T01:12:13.014Z [WARN]  server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3: rejecting vote request since our last term is greater: candidate=76e7030d-f8ce-6ee6-28d1-5b5e1e251222 last-term=2 last-candidate-term=1
2025-07-05T01:12:13.119Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:13.014028951 +0000 UTC m=+108.262766169 (1 nodes), 67.307181ms from start of poll, 67.524719ms from cluster start. Timeout at 2025-07-05 01:12:13.119196209 +0000 UTC m=+108.367933417, 105.167248ms after stability
2025-07-05T01:12:13.119Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:13.224Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:13.119240663 +0000 UTC m=+108.367977881 (2 nodes), 1.824µs from start of poll, 172.736431ms from cluster start. Timeout at 2025-07-05 01:12:13.224500234 +0000 UTC m=+108.473237452, 105.259571ms after stability
2025-07-05T01:12:13.245Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: entering candidate state: node="Node at a218718e-0587-74a2-0b37-690026dd9ecd [Candidate]" term=3
2025-07-05T01:12:13.245Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: election won: term=3 tally=2
2025-07-05T01:12:13.245Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: entering leader state: leader="Node at a218718e-0587-74a2-0b37-690026dd9ecd [Leader]"
2025-07-05T01:12:13.245Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: added peer, starting replication: peer=server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222
2025-07-05T01:12:13.245Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: added peer, starting replication: peer=server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3
2025-07-05T01:12:13.245Z [WARN]  server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3: failed to get previous log: previous-index=1002 last-index=785 error="log not found"
2025-07-05T01:12:13.245Z [WARN]  server-a218718e-0587-74a2-0b37-690026dd9ecd: appendEntries rejected, sending older logs: peer="{Voter server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}" next=786
2025-07-05T01:12:13.245Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: pipelining replication: peer="{Voter server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}"
2025-07-05T01:12:13.245Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:13.245Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: entering follower state: follower="Node at 76e7030d-f8ce-6ee6-28d1-5b5e1e251222 [Follower]" leader-address= leader-id=
2025-07-05T01:12:13.245Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: aborting pipeline replication: peer="{Voter server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}"
2025-07-05T01:12:13.245Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: pipelining replication: peer="{Voter server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222 76e7030d-f8ce-6ee6-28d1-5b5e1e251222}"
2025-07-05T01:12:13.245Z [INFO]  server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222: aborting pipeline replication: peer="{Voter server-a218718e-0587-74a2-0b37-690026dd9ecd a218718e-0587-74a2-0b37-690026dd9ecd}"
2025-07-05T01:12:13.350Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:13.245803137 +0000 UTC m=+108.494540345 (1 nodes), 412.062µs from start of poll, 299.298895ms from cluster start. Timeout at 2025-07-05 01:12:13.350929138 +0000 UTC m=+108.599666347, 105.126002ms after stability
2025-07-05T01:12:13.351Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: aborting pipeline replication: peer="{Voter server-76e7030d-f8ce-6ee6-28d1-5b5e1e251222 76e7030d-f8ce-6ee6-28d1-5b5e1e251222}"
2025-07-05T01:12:13.351Z [INFO]  server-a218718e-0587-74a2-0b37-690026dd9ecd: aborting pipeline replication: peer="{Voter server-5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 5c8f400f-ecad-35e1-5dc5-d2ef467a58f3}"
2025-07-05T01:12:13.851Z [ERROR] server-a218718e-0587-74a2-0b37-690026dd9ecd: failed to heartbeat to: peer=5c8f400f-ecad-35e1-5dc5-d2ef467a58f3 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.91s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-07-05T01:12:13.853Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb Address:68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb} {Suffrage:Voter ID:server-6dd68625-fdd5-6c89-897e-810e15340f35 Address:6dd68625-fdd5-6c89-897e-810e15340f35} {Suffrage:Voter ID:server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e Address:fbb97038-5edc-d5be-f1fd-0253e7d0f43e}]"
2025-07-05T01:12:13.853Z [INFO]  server-6dd68625-fdd5-6c89-897e-810e15340f35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb Address:68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb} {Suffrage:Voter ID:server-6dd68625-fdd5-6c89-897e-810e15340f35 Address:6dd68625-fdd5-6c89-897e-810e15340f35} {Suffrage:Voter ID:server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e Address:fbb97038-5edc-d5be-f1fd-0253e7d0f43e}]"
2025-07-05T01:12:13.853Z [INFO]  server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb Address:68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb} {Suffrage:Voter ID:server-6dd68625-fdd5-6c89-897e-810e15340f35 Address:6dd68625-fdd5-6c89-897e-810e15340f35} {Suffrage:Voter ID:server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e Address:fbb97038-5edc-d5be-f1fd-0253e7d0f43e}]"
2025-07-05T01:12:13.853Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:13.853Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: entering follower state: follower="Node at 68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb [Follower]" leader-address= leader-id=
2025-07-05T01:12:13.853Z [INFO]  server-6dd68625-fdd5-6c89-897e-810e15340f35: entering follower state: follower="Node at 6dd68625-fdd5-6c89-897e-810e15340f35 [Follower]" leader-address= leader-id=
2025-07-05T01:12:13.853Z [INFO]  server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e: entering follower state: follower="Node at fbb97038-5edc-d5be-f1fd-0253e7d0f43e [Follower]" leader-address= leader-id=
2025-07-05T01:12:13.934Z [WARN]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:13.934Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: entering candidate state: node="Node at 68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb [Candidate]" term=2
2025-07-05T01:12:13.934Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: election won: term=2 tally=2
2025-07-05T01:12:13.934Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: entering leader state: leader="Node at 68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb [Leader]"
2025-07-05T01:12:13.934Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: added peer, starting replication: peer=server-6dd68625-fdd5-6c89-897e-810e15340f35
2025-07-05T01:12:13.934Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: added peer, starting replication: peer=server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e
2025-07-05T01:12:13.934Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: pipelining replication: peer="{Voter server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e fbb97038-5edc-d5be-f1fd-0253e7d0f43e}"
2025-07-05T01:12:13.934Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: pipelining replication: peer="{Voter server-6dd68625-fdd5-6c89-897e-810e15340f35 6dd68625-fdd5-6c89-897e-810e15340f35}"
2025-07-05T01:12:14.046Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:13.934637592 +0000 UTC m=+109.183374800 (1 nodes), 81.506866ms from start of poll, 81.714234ms from cluster start. Timeout at 2025-07-05 01:12:14.046399615 +0000 UTC m=+109.295136823, 111.762023ms after stability
2025-07-05T01:12:14.046Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: cannot transfer leadership to itself
2025-07-05T01:12:14.046Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: aborting pipeline replication: peer="{Voter server-6dd68625-fdd5-6c89-897e-810e15340f35 6dd68625-fdd5-6c89-897e-810e15340f35}"
2025-07-05T01:12:14.046Z [INFO]  server-68ea29d7-10bb-3aaa-c3c8-f83d47f36bbb: aborting pipeline replication: peer="{Voter server-fbb97038-5edc-d5be-f1fd-0253e7d0f43e fbb97038-5edc-d5be-f1fd-0253e7d0f43e}"
--- PASS: TestRaft_LeadershipTransferToItself (0.19s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-07-05T01:12:14.048Z [INFO]  server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a Address:40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a} {Suffrage:Voter ID:server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789 Address:e4df3433-b9d1-f3e4-2406-b0bd1e5bc789} {Suffrage:Voter ID:server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f Address:b2cedc34-820e-5c1b-fc8f-8f1254f82a4f}]"
2025-07-05T01:12:14.048Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a Address:40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a} {Suffrage:Voter ID:server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789 Address:e4df3433-b9d1-f3e4-2406-b0bd1e5bc789} {Suffrage:Voter ID:server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f Address:b2cedc34-820e-5c1b-fc8f-8f1254f82a4f}]"
2025-07-05T01:12:14.048Z [INFO]  server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a Address:40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a} {Suffrage:Voter ID:server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789 Address:e4df3433-b9d1-f3e4-2406-b0bd1e5bc789} {Suffrage:Voter ID:server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f Address:b2cedc34-820e-5c1b-fc8f-8f1254f82a4f}]"
2025-07-05T01:12:14.048Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:14.048Z [INFO]  server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a: entering follower state: follower="Node at 40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a [Follower]" leader-address= leader-id=
2025-07-05T01:12:14.048Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: entering follower state: follower="Node at e4df3433-b9d1-f3e4-2406-b0bd1e5bc789 [Follower]" leader-address= leader-id=
2025-07-05T01:12:14.049Z [INFO]  server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f: entering follower state: follower="Node at b2cedc34-820e-5c1b-fc8f-8f1254f82a4f [Follower]" leader-address= leader-id=
2025-07-05T01:12:14.102Z [WARN]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:14.102Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: entering candidate state: node="Node at e4df3433-b9d1-f3e4-2406-b0bd1e5bc789 [Candidate]" term=2
2025-07-05T01:12:14.103Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: election won: term=2 tally=2
2025-07-05T01:12:14.103Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: entering leader state: leader="Node at e4df3433-b9d1-f3e4-2406-b0bd1e5bc789 [Leader]"
2025-07-05T01:12:14.103Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: added peer, starting replication: peer=server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a
2025-07-05T01:12:14.103Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: added peer, starting replication: peer=server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f
2025-07-05T01:12:14.103Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: pipelining replication: peer="{Voter server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f b2cedc34-820e-5c1b-fc8f-8f1254f82a4f}"
2025-07-05T01:12:14.103Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: pipelining replication: peer="{Voter server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a 40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a}"
2025-07-05T01:12:14.209Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:14.10322334 +0000 UTC m=+109.351960548 (1 nodes), 55.041603ms from start of poll, 55.379186ms from cluster start. Timeout at 2025-07-05 01:12:14.209005822 +0000 UTC m=+109.457743030, 105.782482ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-07-05T01:12:14.209Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: aborting pipeline replication: peer="{Voter server-b2cedc34-820e-5c1b-fc8f-8f1254f82a4f b2cedc34-820e-5c1b-fc8f-8f1254f82a4f}"
2025-07-05T01:12:14.209Z [INFO]  server-e4df3433-b9d1-f3e4-2406-b0bd1e5bc789: aborting pipeline replication: peer="{Voter server-40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a 40d36d3a-5b99-e3d1-206f-9aa2ed97ca5a}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-07-05T01:12:14.210Z [INFO]  server-aebc05a5-1748-64fc-fab4-40d563d35cbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aebc05a5-1748-64fc-fab4-40d563d35cbf Address:aebc05a5-1748-64fc-fab4-40d563d35cbf} {Suffrage:Voter ID:server-087e4386-1d37-04ae-27ac-8d0fa4b7107b Address:087e4386-1d37-04ae-27ac-8d0fa4b7107b} {Suffrage:Voter ID:server-654ff9ba-c539-30ca-0b7a-dee5212a25b6 Address:654ff9ba-c539-30ca-0b7a-dee5212a25b6}]"
2025-07-05T01:12:14.210Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aebc05a5-1748-64fc-fab4-40d563d35cbf Address:aebc05a5-1748-64fc-fab4-40d563d35cbf} {Suffrage:Voter ID:server-087e4386-1d37-04ae-27ac-8d0fa4b7107b Address:087e4386-1d37-04ae-27ac-8d0fa4b7107b} {Suffrage:Voter ID:server-654ff9ba-c539-30ca-0b7a-dee5212a25b6 Address:654ff9ba-c539-30ca-0b7a-dee5212a25b6}]"
2025-07-05T01:12:14.210Z [INFO]  server-654ff9ba-c539-30ca-0b7a-dee5212a25b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aebc05a5-1748-64fc-fab4-40d563d35cbf Address:aebc05a5-1748-64fc-fab4-40d563d35cbf} {Suffrage:Voter ID:server-087e4386-1d37-04ae-27ac-8d0fa4b7107b Address:087e4386-1d37-04ae-27ac-8d0fa4b7107b} {Suffrage:Voter ID:server-654ff9ba-c539-30ca-0b7a-dee5212a25b6 Address:654ff9ba-c539-30ca-0b7a-dee5212a25b6}]"
2025-07-05T01:12:14.210Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:14.210Z [INFO]  server-aebc05a5-1748-64fc-fab4-40d563d35cbf: entering follower state: follower="Node at aebc05a5-1748-64fc-fab4-40d563d35cbf [Follower]" leader-address= leader-id=
2025-07-05T01:12:14.210Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: entering follower state: follower="Node at 087e4386-1d37-04ae-27ac-8d0fa4b7107b [Follower]" leader-address= leader-id=
2025-07-05T01:12:14.210Z [INFO]  server-654ff9ba-c539-30ca-0b7a-dee5212a25b6: entering follower state: follower="Node at 654ff9ba-c539-30ca-0b7a-dee5212a25b6 [Follower]" leader-address= leader-id=
2025-07-05T01:12:14.266Z [WARN]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:14.266Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: entering candidate state: node="Node at 087e4386-1d37-04ae-27ac-8d0fa4b7107b [Candidate]" term=2
2025-07-05T01:12:14.266Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: election won: term=2 tally=2
2025-07-05T01:12:14.266Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: entering leader state: leader="Node at 087e4386-1d37-04ae-27ac-8d0fa4b7107b [Leader]"
2025-07-05T01:12:14.266Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: added peer, starting replication: peer=server-aebc05a5-1748-64fc-fab4-40d563d35cbf
2025-07-05T01:12:14.266Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: added peer, starting replication: peer=server-654ff9ba-c539-30ca-0b7a-dee5212a25b6
2025-07-05T01:12:14.266Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: pipelining replication: peer="{Voter server-654ff9ba-c539-30ca-0b7a-dee5212a25b6 654ff9ba-c539-30ca-0b7a-dee5212a25b6}"
2025-07-05T01:12:14.266Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: pipelining replication: peer="{Voter server-aebc05a5-1748-64fc-fab4-40d563d35cbf aebc05a5-1748-64fc-fab4-40d563d35cbf}"
2025-07-05T01:12:14.371Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:14.266437677 +0000 UTC m=+109.515174885 (1 nodes), 55.928546ms from start of poll, 56.162465ms from cluster start. Timeout at 2025-07-05 01:12:14.371500169 +0000 UTC m=+109.620237387, 105.062502ms after stability
2025-07-05T01:12:14.371Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:14.476Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:14.371557627 +0000 UTC m=+109.620294835 (2 nodes), 1.513µs from start of poll, 161.282415ms from cluster start. Timeout at 2025-07-05 01:12:14.47688204 +0000 UTC m=+109.725619248, 105.324413ms after stability
2025-07-05T01:12:14.782Z [INFO]  server-aebc05a5-1748-64fc-fab4-40d563d35cbf: entering candidate state: node="Node at aebc05a5-1748-64fc-fab4-40d563d35cbf [Candidate]" term=3
2025-07-05T01:12:14.782Z [INFO]  server-aebc05a5-1748-64fc-fab4-40d563d35cbf: election won: term=3 tally=2
2025-07-05T01:12:14.782Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: aborting pipeline replication: peer="{Voter server-aebc05a5-1748-64fc-fab4-40d563d35cbf aebc05a5-1748-64fc-fab4-40d563d35cbf}"
2025-07-05T01:12:14.783Z [INFO]  server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: aborting pipeline replication: peer="{Voter server-654ff9ba-c539-30ca-0b7a-dee5212a25b6 654ff9ba-c539-30ca-0b7a-dee5212a25b6}"
2025-07-05T01:12:15.283Z [ERROR] server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: failed to heartbeat to: peer=654ff9ba-c539-30ca-0b7a-dee5212a25b6 backoff time=10ms error="command timed out"
2025-07-05T01:12:15.283Z [ERROR] server-aebc05a5-1748-64fc-fab4-40d563d35cbf: failed to make requestVote RPC: target="{Voter server-087e4386-1d37-04ae-27ac-8d0fa4b7107b 087e4386-1d37-04ae-27ac-8d0fa4b7107b}" error="command timed out" term=3
2025-07-05T01:12:15.285Z [ERROR] server-087e4386-1d37-04ae-27ac-8d0fa4b7107b: failed to heartbeat to: peer=aebc05a5-1748-64fc-fab4-40d563d35cbf backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (1.08s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-07-05T01:12:15.286Z [INFO]  server-79470ecd-4d14-29f3-7417-25680a2ea788: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79470ecd-4d14-29f3-7417-25680a2ea788 Address:79470ecd-4d14-29f3-7417-25680a2ea788} {Suffrage:Voter ID:server-ddf853c1-1e31-545a-becf-5bb86a441932 Address:ddf853c1-1e31-545a-becf-5bb86a441932}]"
2025-07-05T01:12:15.286Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79470ecd-4d14-29f3-7417-25680a2ea788 Address:79470ecd-4d14-29f3-7417-25680a2ea788} {Suffrage:Voter ID:server-ddf853c1-1e31-545a-becf-5bb86a441932 Address:ddf853c1-1e31-545a-becf-5bb86a441932}]"
2025-07-05T01:12:15.286Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:15.286Z [INFO]  server-79470ecd-4d14-29f3-7417-25680a2ea788: entering follower state: follower="Node at 79470ecd-4d14-29f3-7417-25680a2ea788 [Follower]" leader-address= leader-id=
2025-07-05T01:12:15.286Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: entering follower state: follower="Node at ddf853c1-1e31-545a-becf-5bb86a441932 [Follower]" leader-address= leader-id=
2025-07-05T01:12:15.359Z [WARN]  server-ddf853c1-1e31-545a-becf-5bb86a441932: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:15.359Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: entering candidate state: node="Node at ddf853c1-1e31-545a-becf-5bb86a441932 [Candidate]" term=2
2025-07-05T01:12:15.359Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: election won: term=2 tally=2
2025-07-05T01:12:15.359Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: entering leader state: leader="Node at ddf853c1-1e31-545a-becf-5bb86a441932 [Leader]"
2025-07-05T01:12:15.359Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: added peer, starting replication: peer=server-79470ecd-4d14-29f3-7417-25680a2ea788
2025-07-05T01:12:15.359Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: pipelining replication: peer="{Voter server-79470ecd-4d14-29f3-7417-25680a2ea788 79470ecd-4d14-29f3-7417-25680a2ea788}"
2025-07-05T01:12:15.464Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:15.359828955 +0000 UTC m=+110.608566173 (1 nodes), 73.49881ms from start of poll, 73.654242ms from cluster start. Timeout at 2025-07-05 01:12:15.464917506 +0000 UTC m=+110.713654714, 105.088541ms after stability
2025-07-05T01:12:15.464Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:15.570Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:15.464970545 +0000 UTC m=+110.713707753 (1 nodes), 661ns from start of poll, 178.795822ms from cluster start. Timeout at 2025-07-05 01:12:15.570581255 +0000 UTC m=+110.819318463, 105.61071ms after stability
2025-07-05T01:12:15.570Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: updating configuration: command=DemoteVoter server-id=server-79470ecd-4d14-29f3-7417-25680a2ea788 server-addr= servers="[{Suffrage:Nonvoter ID:server-79470ecd-4d14-29f3-7417-25680a2ea788 Address:79470ecd-4d14-29f3-7417-25680a2ea788} {Suffrage:Voter ID:server-ddf853c1-1e31-545a-becf-5bb86a441932 Address:ddf853c1-1e31-545a-becf-5bb86a441932}]"
2025-07-05T01:12:15.570Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:15.676Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:15.570696291 +0000 UTC m=+110.819433499 (1 nodes), 501ns from start of poll, 284.521568ms from cluster start. Timeout at 2025-07-05 01:12:15.676605461 +0000 UTC m=+110.925342679, 105.90918ms after stability
2025-07-05T01:12:15.676Z [INFO]  server-ddf853c1-1e31-545a-becf-5bb86a441932: aborting pipeline replication: peer="{Voter server-79470ecd-4d14-29f3-7417-25680a2ea788 79470ecd-4d14-29f3-7417-25680a2ea788}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-07-05T01:12:15.677Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8b3e7c9-2391-030f-1505-f531fd1cc738 Address:e8b3e7c9-2391-030f-1505-f531fd1cc738} {Suffrage:Voter ID:server-a08e40d7-12bf-da5b-08c5-dcfec47adafb Address:a08e40d7-12bf-da5b-08c5-dcfec47adafb}]"
2025-07-05T01:12:15.677Z [INFO]  server-a08e40d7-12bf-da5b-08c5-dcfec47adafb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8b3e7c9-2391-030f-1505-f531fd1cc738 Address:e8b3e7c9-2391-030f-1505-f531fd1cc738} {Suffrage:Voter ID:server-a08e40d7-12bf-da5b-08c5-dcfec47adafb Address:a08e40d7-12bf-da5b-08c5-dcfec47adafb}]"
2025-07-05T01:12:15.677Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:15.677Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: entering follower state: follower="Node at e8b3e7c9-2391-030f-1505-f531fd1cc738 [Follower]" leader-address= leader-id=
2025-07-05T01:12:15.677Z [INFO]  server-a08e40d7-12bf-da5b-08c5-dcfec47adafb: entering follower state: follower="Node at a08e40d7-12bf-da5b-08c5-dcfec47adafb [Follower]" leader-address= leader-id=
2025-07-05T01:12:15.730Z [WARN]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:15.730Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: entering candidate state: node="Node at e8b3e7c9-2391-030f-1505-f531fd1cc738 [Candidate]" term=2
2025-07-05T01:12:15.731Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: election won: term=2 tally=2
2025-07-05T01:12:15.731Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: entering leader state: leader="Node at e8b3e7c9-2391-030f-1505-f531fd1cc738 [Leader]"
2025-07-05T01:12:15.731Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: added peer, starting replication: peer=server-a08e40d7-12bf-da5b-08c5-dcfec47adafb
2025-07-05T01:12:15.731Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: pipelining replication: peer="{Voter server-a08e40d7-12bf-da5b-08c5-dcfec47adafb a08e40d7-12bf-da5b-08c5-dcfec47adafb}"
2025-07-05T01:12:15.836Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:15.731155282 +0000 UTC m=+110.979892490 (1 nodes), 53.609358ms from start of poll, 53.755853ms from cluster start. Timeout at 2025-07-05 01:12:15.836628304 +0000 UTC m=+111.085365512, 105.473022ms after stability
2025-07-05T01:12:15.836Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:15.942Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:15.836734292 +0000 UTC m=+111.085471500 (1 nodes), 1.242µs from start of poll, 159.334863ms from cluster start. Timeout at 2025-07-05 01:12:15.942438037 +0000 UTC m=+111.191175245, 105.703745ms after stability
2025-07-05T01:12:15.953Z [INFO]  server-a08e40d7-12bf-da5b-08c5-dcfec47adafb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8b3e7c9-2391-030f-1505-f531fd1cc738 Address:e8b3e7c9-2391-030f-1505-f531fd1cc738} {Suffrage:Voter ID:server-a08e40d7-12bf-da5b-08c5-dcfec47adafb Address:a08e40d7-12bf-da5b-08c5-dcfec47adafb}]"
2025-07-05T01:12:15.953Z [INFO]  server-e8b3e7c9-2391-030f-1505-f531fd1cc738: aborting pipeline replication: peer="{Voter server-a08e40d7-12bf-da5b-08c5-dcfec47adafb a08e40d7-12bf-da5b-08c5-dcfec47adafb}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-07-05T01:12:15.954Z [INFO]  server-143d6bf2-38f4-c576-45ec-26acac58d847: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-143d6bf2-38f4-c576-45ec-26acac58d847 Address:143d6bf2-38f4-c576-45ec-26acac58d847}]"
2025-07-05T01:12:15.954Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:15.954Z [INFO]  server-143d6bf2-38f4-c576-45ec-26acac58d847: entering follower state: follower="Node at 143d6bf2-38f4-c576-45ec-26acac58d847 [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.016Z [WARN]  server-143d6bf2-38f4-c576-45ec-26acac58d847: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:16.016Z [INFO]  server-143d6bf2-38f4-c576-45ec-26acac58d847: entering candidate state: node="Node at 143d6bf2-38f4-c576-45ec-26acac58d847 [Candidate]" term=2
2025-07-05T01:12:16.017Z [INFO]  server-143d6bf2-38f4-c576-45ec-26acac58d847: election won: term=2 tally=1
2025-07-05T01:12:16.017Z [INFO]  server-143d6bf2-38f4-c576-45ec-26acac58d847: entering leader state: leader="Node at 143d6bf2-38f4-c576-45ec-26acac58d847 [Leader]"
2025-07-05T01:12:16.122Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:16.017083747 +0000 UTC m=+111.265820965 (1 nodes), 62.906038ms from start of poll, 62.99766ms from cluster start. Timeout at 2025-07-05 01:12:16.122178209 +0000 UTC m=+111.370915418, 105.094453ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.17s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-07-05T01:12:16.122Z [INFO]  server-357b6d66-6750-2156-cd43-f956a0b1f13f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-357b6d66-6750-2156-cd43-f956a0b1f13f Address:357b6d66-6750-2156-cd43-f956a0b1f13f} {Suffrage:Voter ID:server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51 Address:f791e051-fb1c-d1f5-2a3f-fc44a1457d51}]"
2025-07-05T01:12:16.123Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-357b6d66-6750-2156-cd43-f956a0b1f13f Address:357b6d66-6750-2156-cd43-f956a0b1f13f} {Suffrage:Voter ID:server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51 Address:f791e051-fb1c-d1f5-2a3f-fc44a1457d51}]"
2025-07-05T01:12:16.123Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:16.123Z [INFO]  server-357b6d66-6750-2156-cd43-f956a0b1f13f: entering follower state: follower="Node at 357b6d66-6750-2156-cd43-f956a0b1f13f [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.123Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: entering follower state: follower="Node at f791e051-fb1c-d1f5-2a3f-fc44a1457d51 [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.185Z [WARN]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:16.185Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: entering candidate state: node="Node at f791e051-fb1c-d1f5-2a3f-fc44a1457d51 [Candidate]" term=2
2025-07-05T01:12:16.185Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: election won: term=2 tally=2
2025-07-05T01:12:16.185Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: entering leader state: leader="Node at f791e051-fb1c-d1f5-2a3f-fc44a1457d51 [Leader]"
2025-07-05T01:12:16.185Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: added peer, starting replication: peer=server-357b6d66-6750-2156-cd43-f956a0b1f13f
2025-07-05T01:12:16.186Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: pipelining replication: peer="{Voter server-357b6d66-6750-2156-cd43-f956a0b1f13f 357b6d66-6750-2156-cd43-f956a0b1f13f}"
2025-07-05T01:12:16.290Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:16.18587431 +0000 UTC m=+111.434611528 (1 nodes), 62.813896ms from start of poll, 62.995337ms from cluster start. Timeout at 2025-07-05 01:12:16.290937974 +0000 UTC m=+111.539675182, 105.063654ms after stability
2025-07-05T01:12:16.290Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:16.396Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:16.290995352 +0000 UTC m=+111.539732560 (1 nodes), 1.393µs from start of poll, 168.116369ms from cluster start. Timeout at 2025-07-05 01:12:16.396166047 +0000 UTC m=+111.644903255, 105.170695ms after stability
2025-07-05T01:12:16.397Z [ERROR] server-357b6d66-6750-2156-cd43-f956a0b1f13f: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-07-05T01:12:16.397Z [WARN]  server-357b6d66-6750-2156-cd43-f956a0b1f13f: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-07-05T01:12:16.397Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: aborting pipeline replication: peer="{Voter server-357b6d66-6750-2156-cd43-f956a0b1f13f 357b6d66-6750-2156-cd43-f956a0b1f13f}"
2025-07-05T01:12:16.404Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: pipelining replication: peer="{Voter server-357b6d66-6750-2156-cd43-f956a0b1f13f 357b6d66-6750-2156-cd43-f956a0b1f13f}"
2025-07-05T01:12:16.404Z [INFO]  server-357b6d66-6750-2156-cd43-f956a0b1f13f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-357b6d66-6750-2156-cd43-f956a0b1f13f Address:357b6d66-6750-2156-cd43-f956a0b1f13f} {Suffrage:Voter ID:server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51 Address:f791e051-fb1c-d1f5-2a3f-fc44a1457d51}]"
2025-07-05T01:12:16.404Z [INFO]  server-f791e051-fb1c-d1f5-2a3f-fc44a1457d51: aborting pipeline replication: peer="{Voter server-357b6d66-6750-2156-cd43-f956a0b1f13f 357b6d66-6750-2156-cd43-f956a0b1f13f}"
--- PASS: TestRaft_CacheLogWithStoreError (0.28s)
=== RUN   TestRaft_ReloadConfig
2025-07-05T01:12:16.405Z [INFO]  server-f0b739ee-7eb3-340b-cdf4-c5543d39e016: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0b739ee-7eb3-340b-cdf4-c5543d39e016 Address:f0b739ee-7eb3-340b-cdf4-c5543d39e016}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-07-05T01:12:16.405Z [INFO]  server-b75b30f5-919e-fac0-35cd-75133e4968ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b75b30f5-919e-fac0-35cd-75133e4968ca Address:b75b30f5-919e-fac0-35cd-75133e4968ca}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-07-05T01:12:16.405Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-07-05T01:12:16.406Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe Address:9e190a9f-c619-2a6e-23c1-79b55cdc13fe} {Suffrage:Voter ID:server-78fbf3d1-2de1-974b-ce74-390d6d488b1e Address:78fbf3d1-2de1-974b-ce74-390d6d488b1e} {Suffrage:Voter ID:server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 Address:bd3ee4e2-9835-0a47-9e0f-b17875461a57}]"
2025-07-05T01:12:16.406Z [INFO]  server-78fbf3d1-2de1-974b-ce74-390d6d488b1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe Address:9e190a9f-c619-2a6e-23c1-79b55cdc13fe} {Suffrage:Voter ID:server-78fbf3d1-2de1-974b-ce74-390d6d488b1e Address:78fbf3d1-2de1-974b-ce74-390d6d488b1e} {Suffrage:Voter ID:server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 Address:bd3ee4e2-9835-0a47-9e0f-b17875461a57}]"
2025-07-05T01:12:16.406Z [INFO]  server-bd3ee4e2-9835-0a47-9e0f-b17875461a57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe Address:9e190a9f-c619-2a6e-23c1-79b55cdc13fe} {Suffrage:Voter ID:server-78fbf3d1-2de1-974b-ce74-390d6d488b1e Address:78fbf3d1-2de1-974b-ce74-390d6d488b1e} {Suffrage:Voter ID:server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 Address:bd3ee4e2-9835-0a47-9e0f-b17875461a57}]"
2025-07-05T01:12:16.406Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:16.406Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: entering follower state: follower="Node at 9e190a9f-c619-2a6e-23c1-79b55cdc13fe [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.406Z [INFO]  server-78fbf3d1-2de1-974b-ce74-390d6d488b1e: entering follower state: follower="Node at 78fbf3d1-2de1-974b-ce74-390d6d488b1e [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.406Z [INFO]  server-bd3ee4e2-9835-0a47-9e0f-b17875461a57: entering follower state: follower="Node at bd3ee4e2-9835-0a47-9e0f-b17875461a57 [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.460Z [WARN]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:16.460Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: entering candidate state: node="Node at 9e190a9f-c619-2a6e-23c1-79b55cdc13fe [Candidate]" term=2
2025-07-05T01:12:16.460Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: election won: term=2 tally=2
2025-07-05T01:12:16.460Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: entering leader state: leader="Node at 9e190a9f-c619-2a6e-23c1-79b55cdc13fe [Leader]"
2025-07-05T01:12:16.460Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: added peer, starting replication: peer=server-78fbf3d1-2de1-974b-ce74-390d6d488b1e
2025-07-05T01:12:16.460Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: added peer, starting replication: peer=server-bd3ee4e2-9835-0a47-9e0f-b17875461a57
2025-07-05T01:12:16.460Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: pipelining replication: peer="{Voter server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 bd3ee4e2-9835-0a47-9e0f-b17875461a57}"
2025-07-05T01:12:16.460Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: pipelining replication: peer="{Voter server-78fbf3d1-2de1-974b-ce74-390d6d488b1e 78fbf3d1-2de1-974b-ce74-390d6d488b1e}"
2025-07-05T01:12:16.565Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:16.46042906 +0000 UTC m=+111.709166268 (1 nodes), 53.993087ms from start of poll, 54.157325ms from cluster start. Timeout at 2025-07-05 01:12:16.565468829 +0000 UTC m=+111.814206037, 105.039769ms after stability
2025-07-05T01:12:16.571Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:16.677Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:16.57153303 +0000 UTC m=+111.820270238 (2 nodes), 2.204µs from start of poll, 165.261295ms from cluster start. Timeout at 2025-07-05 01:12:16.677347572 +0000 UTC m=+111.926084780, 105.814542ms after stability
2025-07-05T01:12:16.677Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: updating configuration: command=RemoveServer server-id=server-78fbf3d1-2de1-974b-ce74-390d6d488b1e server-addr= servers="[{Suffrage:Voter ID:server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe Address:9e190a9f-c619-2a6e-23c1-79b55cdc13fe} {Suffrage:Voter ID:server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 Address:bd3ee4e2-9835-0a47-9e0f-b17875461a57}]"
2025-07-05T01:12:16.677Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: removed peer, stopping replication: peer=server-78fbf3d1-2de1-974b-ce74-390d6d488b1e last-index=3
2025-07-05T01:12:16.677Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: aborting pipeline replication: peer="{Voter server-78fbf3d1-2de1-974b-ce74-390d6d488b1e 78fbf3d1-2de1-974b-ce74-390d6d488b1e}"
2025-07-05T01:12:16.783Z [ERROR] server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to heartbeat to: peer=bd3ee4e2-9835-0a47-9e0f-b17875461a57 backoff time=10ms error="failed to connect to peer: bd3ee4e2-9835-0a47-9e0f-b17875461a57"
2025-07-05T01:12:16.784Z [ERROR] server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to pipeline appendEntries: peer="{Voter server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 bd3ee4e2-9835-0a47-9e0f-b17875461a57}" error="append pipeline closed"
2025-07-05T01:12:16.784Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: aborting pipeline replication: peer="{Voter server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 bd3ee4e2-9835-0a47-9e0f-b17875461a57}"
2025-07-05T01:12:16.791Z [WARN]  server-78fbf3d1-2de1-974b-ce74-390d6d488b1e: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-07-05T01:12:16.795Z [ERROR] server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to appendEntries to: peer="{Voter server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 bd3ee4e2-9835-0a47-9e0f-b17875461a57}" error="failed to connect to peer: bd3ee4e2-9835-0a47-9e0f-b17875461a57"
2025-07-05T01:12:16.802Z [ERROR] server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to heartbeat to: peer=bd3ee4e2-9835-0a47-9e0f-b17875461a57 backoff time=10ms error="failed to connect to peer: bd3ee4e2-9835-0a47-9e0f-b17875461a57"
2025-07-05T01:12:16.814Z [ERROR] server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to appendEntries to: peer="{Voter server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 bd3ee4e2-9835-0a47-9e0f-b17875461a57}" error="failed to connect to peer: bd3ee4e2-9835-0a47-9e0f-b17875461a57"
2025-07-05T01:12:16.823Z [ERROR] server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to heartbeat to: peer=bd3ee4e2-9835-0a47-9e0f-b17875461a57 backoff time=10ms error="failed to connect to peer: bd3ee4e2-9835-0a47-9e0f-b17875461a57"
2025-07-05T01:12:16.825Z [WARN]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to contact: server-id=server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 time=50.747612ms
2025-07-05T01:12:16.825Z [WARN]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to contact quorum of nodes, stepping down
2025-07-05T01:12:16.825Z [INFO]  server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: entering follower state: follower="Node at 9e190a9f-c619-2a6e-23c1-79b55cdc13fe [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.834Z [ERROR] server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe: failed to appendEntries to: peer="{Voter server-bd3ee4e2-9835-0a47-9e0f-b17875461a57 bd3ee4e2-9835-0a47-9e0f-b17875461a57}" error="failed to connect to peer: bd3ee4e2-9835-0a47-9e0f-b17875461a57"
2025-07-05T01:12:16.841Z [WARN]  server-bd3ee4e2-9835-0a47-9e0f-b17875461a57: heartbeat timeout reached, starting election: last-leader-addr=9e190a9f-c619-2a6e-23c1-79b55cdc13fe last-leader-id=server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe
2025-07-05T01:12:16.841Z [INFO]  server-bd3ee4e2-9835-0a47-9e0f-b17875461a57: entering candidate state: node="Node at bd3ee4e2-9835-0a47-9e0f-b17875461a57 [Candidate]" term=3
2025-07-05T01:12:16.841Z [ERROR] server-bd3ee4e2-9835-0a47-9e0f-b17875461a57: failed to make requestVote RPC: target="{Voter server-9e190a9f-c619-2a6e-23c1-79b55cdc13fe 9e190a9f-c619-2a6e-23c1-79b55cdc13fe}" error="failed to connect to peer: 9e190a9f-c619-2a6e-23c1-79b55cdc13fe" term=3
2025-07-05T01:12:16.889Z [WARN]  server-bd3ee4e2-9835-0a47-9e0f-b17875461a57: rejecting vote request since node is not in configuration: from=78fbf3d1-2de1-974b-ce74-390d6d488b1e
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.48s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-07-05T01:12:16.890Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-462a26f3-7eb1-3fef-e63a-e68896c18422 Address:462a26f3-7eb1-3fef-e63a-e68896c18422} {Suffrage:Voter ID:server-50dcb41c-166f-d93b-1bdb-c1246994bb9f Address:50dcb41c-166f-d93b-1bdb-c1246994bb9f} {Suffrage:Voter ID:server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 Address:21f75e0e-ad55-1e2c-2ff1-52f386fd1562}]"
2025-07-05T01:12:16.890Z [INFO]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-462a26f3-7eb1-3fef-e63a-e68896c18422 Address:462a26f3-7eb1-3fef-e63a-e68896c18422} {Suffrage:Voter ID:server-50dcb41c-166f-d93b-1bdb-c1246994bb9f Address:50dcb41c-166f-d93b-1bdb-c1246994bb9f} {Suffrage:Voter ID:server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 Address:21f75e0e-ad55-1e2c-2ff1-52f386fd1562}]"
2025-07-05T01:12:16.890Z [INFO]  server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-462a26f3-7eb1-3fef-e63a-e68896c18422 Address:462a26f3-7eb1-3fef-e63a-e68896c18422} {Suffrage:Voter ID:server-50dcb41c-166f-d93b-1bdb-c1246994bb9f Address:50dcb41c-166f-d93b-1bdb-c1246994bb9f} {Suffrage:Voter ID:server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 Address:21f75e0e-ad55-1e2c-2ff1-52f386fd1562}]"
2025-07-05T01:12:16.890Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:16.890Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering follower state: follower="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.890Z [INFO]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: entering follower state: follower="Node at 50dcb41c-166f-d93b-1bdb-c1246994bb9f [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.890Z [INFO]  server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562: entering follower state: follower="Node at 21f75e0e-ad55-1e2c-2ff1-52f386fd1562 [Follower]" leader-address= leader-id=
2025-07-05T01:12:16.949Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:16.949Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering candidate state: node="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Candidate]" term=2
2025-07-05T01:12:16.949Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: election won: term=2 tally=2
2025-07-05T01:12:16.949Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering leader state: leader="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Leader]"
2025-07-05T01:12:16.949Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:16.949Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562
2025-07-05T01:12:16.949Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: pipelining replication: peer="{Voter server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 21f75e0e-ad55-1e2c-2ff1-52f386fd1562}"
2025-07-05T01:12:16.949Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: pipelining replication: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:17.055Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:16.949761057 +0000 UTC m=+112.198498265 (1 nodes), 59.104501ms from start of poll, 59.326728ms from cluster start. Timeout at 2025-07-05 01:12:17.054995211 +0000 UTC m=+112.303732429, 105.234164ms after stability
2025-07-05T01:12:17.061Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:17.166Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:17.061188293 +0000 UTC m=+112.309925501 (2 nodes), 3.236µs from start of poll, 170.753964ms from cluster start. Timeout at 2025-07-05 01:12:17.16638153 +0000 UTC m=+112.415118748, 105.193247ms after stability
2025-07-05T01:12:17.170Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to heartbeat to: peer=50dcb41c-166f-d93b-1bdb-c1246994bb9f backoff time=10ms error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.170Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to pipeline appendEntries: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" error="append pipeline closed"
2025-07-05T01:12:17.171Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: aborting pipeline replication: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:17.179Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to appendEntries to: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.190Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to heartbeat to: peer=50dcb41c-166f-d93b-1bdb-c1246994bb9f backoff time=10ms error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.203Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to appendEntries to: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.211Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to heartbeat to: peer=50dcb41c-166f-d93b-1bdb-c1246994bb9f backoff time=10ms error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.215Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to contact: server-id=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f time=50.371316ms
2025-07-05T01:12:17.218Z [WARN]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: heartbeat timeout reached, starting election: last-leader-addr=462a26f3-7eb1-3fef-e63a-e68896c18422 last-leader-id=server-462a26f3-7eb1-3fef-e63a-e68896c18422
2025-07-05T01:12:17.218Z [INFO]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: entering candidate state: node="Node at 50dcb41c-166f-d93b-1bdb-c1246994bb9f [Candidate]" term=3
2025-07-05T01:12:17.218Z [ERROR] server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: failed to make requestVote RPC: target="{Voter server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 21f75e0e-ad55-1e2c-2ff1-52f386fd1562}" error="failed to connect to peer: 21f75e0e-ad55-1e2c-2ff1-52f386fd1562" term=3
2025-07-05T01:12:17.218Z [ERROR] server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: failed to make requestVote RPC: target="{Voter server-462a26f3-7eb1-3fef-e63a-e68896c18422 462a26f3-7eb1-3fef-e63a-e68896c18422}" error="failed to connect to peer: 462a26f3-7eb1-3fef-e63a-e68896c18422" term=3
2025-07-05T01:12:17.222Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to appendEntries to: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.230Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to heartbeat to: peer=50dcb41c-166f-d93b-1bdb-c1246994bb9f backoff time=20ms error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.255Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to appendEntries to: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.258Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to heartbeat to: peer=50dcb41c-166f-d93b-1bdb-c1246994bb9f backoff time=25ms error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.263Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to contact: server-id=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f time=98.123011ms
2025-07-05T01:12:17.271Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: updating configuration: command=RemoveServer server-id=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f server-addr= servers="[{Suffrage:Voter ID:server-462a26f3-7eb1-3fef-e63a-e68896c18422 Address:462a26f3-7eb1-3fef-e63a-e68896c18422} {Suffrage:Voter ID:server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 Address:21f75e0e-ad55-1e2c-2ff1-52f386fd1562}]"
2025-07-05T01:12:17.271Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: removed peer, stopping replication: peer=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f last-index=3
2025-07-05T01:12:17.271Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: updating configuration: command=AddNonvoter server-id=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f server-addr=50dcb41c-166f-d93b-1bdb-c1246994bb9f servers="[{Suffrage:Voter ID:server-462a26f3-7eb1-3fef-e63a-e68896c18422 Address:462a26f3-7eb1-3fef-e63a-e68896c18422} {Suffrage:Voter ID:server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 Address:21f75e0e-ad55-1e2c-2ff1-52f386fd1562} {Suffrage:Nonvoter ID:server-50dcb41c-166f-d93b-1bdb-c1246994bb9f Address:50dcb41c-166f-d93b-1bdb-c1246994bb9f}]"
2025-07-05T01:12:17.271Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.271Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: failed to appendEntries to: peer="{Nonvoter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" error="failed to connect to peer: 50dcb41c-166f-d93b-1bdb-c1246994bb9f"
2025-07-05T01:12:17.290Z [WARN]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: Election timeout reached, restarting election
2025-07-05T01:12:17.290Z [INFO]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: entering candidate state: node="Node at 50dcb41c-166f-d93b-1bdb-c1246994bb9f [Candidate]" term=23
2025-07-05T01:12:17.290Z [WARN]  server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562: rejecting vote request since we have a leader: from=50dcb41c-166f-d93b-1bdb-c1246994bb9f leader=462a26f3-7eb1-3fef-e63a-e68896c18422 leader-id=server-462a26f3-7eb1-3fef-e63a-e68896c18422
2025-07-05T01:12:17.290Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: rejecting vote request since we have a leader: from=50dcb41c-166f-d93b-1bdb-c1246994bb9f leader=462a26f3-7eb1-3fef-e63a-e68896c18422 leader-id=server-462a26f3-7eb1-3fef-e63a-e68896c18422
2025-07-05T01:12:17.291Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: peer has newer term, stopping replication: peer="{Nonvoter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:17.291Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering follower state: follower="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Follower]" leader-address= leader-id=
2025-07-05T01:12:17.291Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: aborting pipeline replication: peer="{Voter server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 21f75e0e-ad55-1e2c-2ff1-52f386fd1562}"
2025-07-05T01:12:17.304Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: peer has newer term, stopping replication: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:17.351Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:17.351Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering candidate state: node="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Candidate]" term=3
2025-07-05T01:12:17.351Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: election won: term=3 tally=2
2025-07-05T01:12:17.351Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering leader state: leader="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Leader]"
2025-07-05T01:12:17.351Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562
2025-07-05T01:12:17.351Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.351Z [ERROR] server-462a26f3-7eb1-3fef-e63a-e68896c18422: peer has newer term, stopping replication: peer="{Nonvoter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:17.351Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering follower state: follower="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Follower]" leader-address= leader-id=
2025-07-05T01:12:17.358Z [WARN]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: Election timeout reached, restarting election
2025-07-05T01:12:17.358Z [INFO]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: entering candidate state: node="Node at 50dcb41c-166f-d93b-1bdb-c1246994bb9f [Candidate]" term=24
2025-07-05T01:12:17.358Z [WARN]  server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562: rejecting vote request since node is not a voter: from=50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.358Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: rejecting vote request since node is not a voter: from=50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.377Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:17.419Z [WARN]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: Election timeout reached, restarting election
2025-07-05T01:12:17.419Z [INFO]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: entering candidate state: node="Node at 50dcb41c-166f-d93b-1bdb-c1246994bb9f [Candidate]" term=25
2025-07-05T01:12:17.419Z [WARN]  server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562: rejecting vote request since node is not a voter: from=50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.419Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: rejecting vote request since node is not a voter: from=50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.451Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:17.451Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering candidate state: node="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Candidate]" term=26
2025-07-05T01:12:17.451Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: election won: term=26 tally=2
2025-07-05T01:12:17.451Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: entering leader state: leader="Node at 462a26f3-7eb1-3fef-e63a-e68896c18422 [Leader]"
2025-07-05T01:12:17.451Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562
2025-07-05T01:12:17.451Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.451Z [WARN]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: failed to get previous log: previous-index=5 last-index=2 error="log not found"
2025-07-05T01:12:17.451Z [INFO]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: entering follower state: follower="Node at 50dcb41c-166f-d93b-1bdb-c1246994bb9f [Follower]" leader-address=462a26f3-7eb1-3fef-e63a-e68896c18422 leader-id=server-462a26f3-7eb1-3fef-e63a-e68896c18422
2025-07-05T01:12:17.451Z [WARN]  server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562: failed to get previous log: previous-index=5 last-index=4 error="log not found"
2025-07-05T01:12:17.451Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: appendEntries rejected, sending older logs: peer="{Voter server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 21f75e0e-ad55-1e2c-2ff1-52f386fd1562}" next=5
2025-07-05T01:12:17.451Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: pipelining replication: peer="{Voter server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 21f75e0e-ad55-1e2c-2ff1-52f386fd1562}"
2025-07-05T01:12:17.451Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: appendEntries rejected, sending older logs: peer="{Nonvoter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" next=3
2025-07-05T01:12:17.451Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: pipelining replication: peer="{Nonvoter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:17.557Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:17.451790993 +0000 UTC m=+112.700528201 (1 nodes), 73.913137ms from start of poll, 561.356664ms from cluster start. Timeout at 2025-07-05 01:12:17.557335168 +0000 UTC m=+112.806072376, 105.544175ms after stability
2025-07-05T01:12:17.577Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: updating configuration: command=RemoveServer server-id=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f server-addr= servers="[{Suffrage:Voter ID:server-462a26f3-7eb1-3fef-e63a-e68896c18422 Address:462a26f3-7eb1-3fef-e63a-e68896c18422} {Suffrage:Voter ID:server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 Address:21f75e0e-ad55-1e2c-2ff1-52f386fd1562}]"
2025-07-05T01:12:17.577Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: removed peer, stopping replication: peer=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f last-index=107
2025-07-05T01:12:17.577Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: aborting pipeline replication: peer="{Nonvoter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:17.577Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: updating configuration: command=AddVoter server-id=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f server-addr=50dcb41c-166f-d93b-1bdb-c1246994bb9f servers="[{Suffrage:Voter ID:server-462a26f3-7eb1-3fef-e63a-e68896c18422 Address:462a26f3-7eb1-3fef-e63a-e68896c18422} {Suffrage:Voter ID:server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 Address:21f75e0e-ad55-1e2c-2ff1-52f386fd1562} {Suffrage:Voter ID:server-50dcb41c-166f-d93b-1bdb-c1246994bb9f Address:50dcb41c-166f-d93b-1bdb-c1246994bb9f}]"
2025-07-05T01:12:17.577Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: added peer, starting replication: peer=server-50dcb41c-166f-d93b-1bdb-c1246994bb9f
2025-07-05T01:12:17.577Z [WARN]  server-50dcb41c-166f-d93b-1bdb-c1246994bb9f: failed to get previous log: previous-index=108 last-index=107 error="log not found"
2025-07-05T01:12:17.577Z [WARN]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: appendEntries rejected, sending older logs: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}" next=108
2025-07-05T01:12:17.577Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: pipelining replication: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:18.653Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:18.758Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:18.653166041 +0000 UTC m=+113.901903259 (1 nodes), 1.283µs from start of poll, 1.762731722s from cluster start. Timeout at 2025-07-05 01:12:18.758571015 +0000 UTC m=+114.007308233, 105.404974ms after stability
2025-07-05T01:12:18.758Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: aborting pipeline replication: peer="{Voter server-50dcb41c-166f-d93b-1bdb-c1246994bb9f 50dcb41c-166f-d93b-1bdb-c1246994bb9f}"
2025-07-05T01:12:18.758Z [INFO]  server-462a26f3-7eb1-3fef-e63a-e68896c18422: aborting pipeline replication: peer="{Voter server-21f75e0e-ad55-1e2c-2ff1-52f386fd1562 21f75e0e-ad55-1e2c-2ff1-52f386fd1562}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.92s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-07-05T01:12:18.815Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79f03579-375f-f3d1-435c-bfae27546b5d Address:79f03579-375f-f3d1-435c-bfae27546b5d} {Suffrage:Voter ID:server-9b6f91fa-c6af-2889-a654-324b35beea19 Address:9b6f91fa-c6af-2889-a654-324b35beea19} {Suffrage:Voter ID:server-4fc579a0-b7e7-f0ef-2712-86dc99739d08 Address:4fc579a0-b7e7-f0ef-2712-86dc99739d08}]"
2025-07-05T01:12:18.815Z [INFO]  server-9b6f91fa-c6af-2889-a654-324b35beea19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79f03579-375f-f3d1-435c-bfae27546b5d Address:79f03579-375f-f3d1-435c-bfae27546b5d} {Suffrage:Voter ID:server-9b6f91fa-c6af-2889-a654-324b35beea19 Address:9b6f91fa-c6af-2889-a654-324b35beea19} {Suffrage:Voter ID:server-4fc579a0-b7e7-f0ef-2712-86dc99739d08 Address:4fc579a0-b7e7-f0ef-2712-86dc99739d08}]"
2025-07-05T01:12:18.815Z [INFO]  server-4fc579a0-b7e7-f0ef-2712-86dc99739d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79f03579-375f-f3d1-435c-bfae27546b5d Address:79f03579-375f-f3d1-435c-bfae27546b5d} {Suffrage:Voter ID:server-9b6f91fa-c6af-2889-a654-324b35beea19 Address:9b6f91fa-c6af-2889-a654-324b35beea19} {Suffrage:Voter ID:server-4fc579a0-b7e7-f0ef-2712-86dc99739d08 Address:4fc579a0-b7e7-f0ef-2712-86dc99739d08}]"
2025-07-05T01:12:18.816Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:18.816Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: entering follower state: follower="Node at 79f03579-375f-f3d1-435c-bfae27546b5d [Follower]" leader-address= leader-id=
2025-07-05T01:12:18.816Z [INFO]  server-9b6f91fa-c6af-2889-a654-324b35beea19: entering follower state: follower="Node at 9b6f91fa-c6af-2889-a654-324b35beea19 [Follower]" leader-address= leader-id=
2025-07-05T01:12:18.816Z [INFO]  server-4fc579a0-b7e7-f0ef-2712-86dc99739d08: entering follower state: follower="Node at 4fc579a0-b7e7-f0ef-2712-86dc99739d08 [Follower]" leader-address= leader-id=
2025-07-05T01:12:18.953Z [WARN]  server-79f03579-375f-f3d1-435c-bfae27546b5d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:18.953Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: entering candidate state: node="Node at 79f03579-375f-f3d1-435c-bfae27546b5d [Candidate]" term=2
2025-07-05T01:12:18.953Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: election won: term=2 tally=2
2025-07-05T01:12:18.953Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: entering leader state: leader="Node at 79f03579-375f-f3d1-435c-bfae27546b5d [Leader]"
2025-07-05T01:12:18.953Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: added peer, starting replication: peer=server-9b6f91fa-c6af-2889-a654-324b35beea19
2025-07-05T01:12:18.953Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: added peer, starting replication: peer=server-4fc579a0-b7e7-f0ef-2712-86dc99739d08
2025-07-05T01:12:18.953Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: pipelining replication: peer="{Voter server-4fc579a0-b7e7-f0ef-2712-86dc99739d08 4fc579a0-b7e7-f0ef-2712-86dc99739d08}"
2025-07-05T01:12:18.953Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: pipelining replication: peer="{Voter server-9b6f91fa-c6af-2889-a654-324b35beea19 9b6f91fa-c6af-2889-a654-324b35beea19}"
2025-07-05T01:12:19.159Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:18.954097858 +0000 UTC m=+114.202835076 (1 nodes), 138.082124ms from start of poll, 138.549831ms from cluster start. Timeout at 2025-07-05 01:12:19.15910698 +0000 UTC m=+114.407844188, 205.009112ms after stability
2025-07-05T01:12:19.159Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:19.364Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:19.159188392 +0000 UTC m=+114.407925601 (1 nodes), 1.463µs from start of poll, 343.640356ms from cluster start. Timeout at 2025-07-05 01:12:19.364859536 +0000 UTC m=+114.613596754, 205.671153ms after stability
2025-07-05T01:12:19.371Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:19.579Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:19.371381595 +0000 UTC m=+114.620118803 (2 nodes), 2.385µs from start of poll, 555.833558ms from cluster start. Timeout at 2025-07-05 01:12:19.579127017 +0000 UTC m=+114.827864235, 207.745432ms after stability
2025-07-05T01:12:19.579Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:19.784Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:19.57925097 +0000 UTC m=+114.827988538 (1 nodes), 2.625µs from start of poll, 763.703293ms from cluster start. Timeout at 2025-07-05 01:12:19.784751884 +0000 UTC m=+115.033489102, 205.500564ms after stability
    raft_test.go:2961: [INFO] restarting Node at 9b6f91fa-c6af-2889-a654-324b35beea19 [Follower]
2025-07-05T01:12:19.785Z [INFO]  server-9b6f91fa-c6af-2889-a654-324b35beea19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79f03579-375f-f3d1-435c-bfae27546b5d Address:79f03579-375f-f3d1-435c-bfae27546b5d} {Suffrage:Voter ID:server-9b6f91fa-c6af-2889-a654-324b35beea19 Address:9b6f91fa-c6af-2889-a654-324b35beea19} {Suffrage:Voter ID:server-4fc579a0-b7e7-f0ef-2712-86dc99739d08 Address:4fc579a0-b7e7-f0ef-2712-86dc99739d08}]"
2025-07-05T01:12:19.785Z [INFO]  server-9b6f91fa-c6af-2889-a654-324b35beea19: entering follower state: follower="Node at 9b6f91fa-c6af-2889-a654-324b35beea19 [Follower]" leader-address= leader-id=
2025-07-05T01:12:20.035Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: aborting pipeline replication: peer="{Voter server-4fc579a0-b7e7-f0ef-2712-86dc99739d08 4fc579a0-b7e7-f0ef-2712-86dc99739d08}"
2025-07-05T01:12:20.424Z [ERROR] server-79f03579-375f-f3d1-435c-bfae27546b5d: failed to pipeline appendEntries: peer="{Voter server-9b6f91fa-c6af-2889-a654-324b35beea19 9b6f91fa-c6af-2889-a654-324b35beea19}" error="command enqueue timeout"
2025-07-05T01:12:20.424Z [INFO]  server-79f03579-375f-f3d1-435c-bfae27546b5d: aborting pipeline replication: peer="{Voter server-9b6f91fa-c6af-2889-a654-324b35beea19 9b6f91fa-c6af-2889-a654-324b35beea19}"
2025-07-05T01:12:20.535Z [ERROR] server-79f03579-375f-f3d1-435c-bfae27546b5d: failed to heartbeat to: peer=4fc579a0-b7e7-f0ef-2712-86dc99739d08 backoff time=10ms error="command timed out"
2025-07-05T01:12:20.547Z [ERROR] server-79f03579-375f-f3d1-435c-bfae27546b5d: failed to heartbeat to: peer=9b6f91fa-c6af-2889-a654-324b35beea19 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.73s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-07-05T01:12:20.549Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291 Address:c4a5a28c-f4ff-8d16-6dba-070a90fdd291} {Suffrage:Voter ID:server-680194cb-d075-343f-ceff-cc084646d23c Address:680194cb-d075-343f-ceff-cc084646d23c} {Suffrage:Voter ID:server-2f8dba74-10b1-89c8-2ab4-312f399761eb Address:2f8dba74-10b1-89c8-2ab4-312f399761eb}]"
2025-07-05T01:12:20.549Z [INFO]  server-680194cb-d075-343f-ceff-cc084646d23c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291 Address:c4a5a28c-f4ff-8d16-6dba-070a90fdd291} {Suffrage:Voter ID:server-680194cb-d075-343f-ceff-cc084646d23c Address:680194cb-d075-343f-ceff-cc084646d23c} {Suffrage:Voter ID:server-2f8dba74-10b1-89c8-2ab4-312f399761eb Address:2f8dba74-10b1-89c8-2ab4-312f399761eb}]"
2025-07-05T01:12:20.549Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: entering follower state: follower="Node at c4a5a28c-f4ff-8d16-6dba-070a90fdd291 [Follower]" leader-address= leader-id=
2025-07-05T01:12:20.549Z [INFO]  server-2f8dba74-10b1-89c8-2ab4-312f399761eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291 Address:c4a5a28c-f4ff-8d16-6dba-070a90fdd291} {Suffrage:Voter ID:server-680194cb-d075-343f-ceff-cc084646d23c Address:680194cb-d075-343f-ceff-cc084646d23c} {Suffrage:Voter ID:server-2f8dba74-10b1-89c8-2ab4-312f399761eb Address:2f8dba74-10b1-89c8-2ab4-312f399761eb}]"
2025-07-05T01:12:20.549Z [INFO]  server-680194cb-d075-343f-ceff-cc084646d23c: entering follower state: follower="Node at 680194cb-d075-343f-ceff-cc084646d23c [Follower]" leader-address= leader-id=
2025-07-05T01:12:20.549Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:20.549Z [INFO]  server-2f8dba74-10b1-89c8-2ab4-312f399761eb: entering follower state: follower="Node at 2f8dba74-10b1-89c8-2ab4-312f399761eb [Follower]" leader-address= leader-id=
2025-07-05T01:12:20.609Z [WARN]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-05T01:12:20.609Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: entering candidate state: node="Node at c4a5a28c-f4ff-8d16-6dba-070a90fdd291 [Candidate]" term=2
2025-07-05T01:12:20.609Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: election won: term=2 tally=2
2025-07-05T01:12:20.609Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: entering leader state: leader="Node at c4a5a28c-f4ff-8d16-6dba-070a90fdd291 [Leader]"
2025-07-05T01:12:20.609Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: added peer, starting replication: peer=server-680194cb-d075-343f-ceff-cc084646d23c
2025-07-05T01:12:20.609Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: added peer, starting replication: peer=server-2f8dba74-10b1-89c8-2ab4-312f399761eb
2025-07-05T01:12:20.610Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: pipelining replication: peer="{Voter server-2f8dba74-10b1-89c8-2ab4-312f399761eb 2f8dba74-10b1-89c8-2ab4-312f399761eb}"
2025-07-05T01:12:20.610Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: pipelining replication: peer="{Voter server-680194cb-d075-343f-ceff-cc084646d23c 680194cb-d075-343f-ceff-cc084646d23c}"
2025-07-05T01:12:20.715Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:20.610201692 +0000 UTC m=+115.858938900 (1 nodes), 60.678222ms from start of poll, 60.972574ms from cluster start. Timeout at 2025-07-05 01:12:20.715296054 +0000 UTC m=+115.964033272, 105.094372ms after stability
2025-07-05T01:12:20.715Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-05T01:12:20.821Z [INFO]  cluster: stable state for Leader reached at 2025-07-05 01:12:20.715403936 +0000 UTC m=+115.964141154 (1 nodes), 1.713µs from start of poll, 166.174828ms from cluster start. Timeout at 2025-07-05 01:12:20.821010478 +0000 UTC m=+116.069747696, 105.606542ms after stability
2025-07-05T01:12:20.826Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-05T01:12:20.931Z [INFO]  cluster: stable state for Follower reached at 2025-07-05 01:12:20.826225026 +0000 UTC m=+116.074962234 (2 nodes), 2.215µs from start of poll, 276.995908ms from cluster start. Timeout at 2025-07-05 01:12:20.93176339 +0000 UTC m=+116.180500608, 105.538374ms after stability
2025-07-05T01:12:20.931Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to heartbeat to: peer=2f8dba74-10b1-89c8-2ab4-312f399761eb backoff time=10ms error="failed to connect to peer: 2f8dba74-10b1-89c8-2ab4-312f399761eb"
2025-07-05T01:12:20.932Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to pipeline appendEntries: peer="{Voter server-680194cb-d075-343f-ceff-cc084646d23c 680194cb-d075-343f-ceff-cc084646d23c}" error="append pipeline closed"
2025-07-05T01:12:20.932Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: aborting pipeline replication: peer="{Voter server-680194cb-d075-343f-ceff-cc084646d23c 680194cb-d075-343f-ceff-cc084646d23c}"
2025-07-05T01:12:20.934Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to pipeline appendEntries: peer="{Voter server-2f8dba74-10b1-89c8-2ab4-312f399761eb 2f8dba74-10b1-89c8-2ab4-312f399761eb}" error="append pipeline closed"
2025-07-05T01:12:20.934Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: aborting pipeline replication: peer="{Voter server-2f8dba74-10b1-89c8-2ab4-312f399761eb 2f8dba74-10b1-89c8-2ab4-312f399761eb}"
2025-07-05T01:12:20.936Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to heartbeat to: peer=680194cb-d075-343f-ceff-cc084646d23c backoff time=10ms error="failed to connect to peer: 680194cb-d075-343f-ceff-cc084646d23c"
2025-07-05T01:12:20.941Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to appendEntries to: peer="{Voter server-2f8dba74-10b1-89c8-2ab4-312f399761eb 2f8dba74-10b1-89c8-2ab4-312f399761eb}" error="failed to connect to peer: 2f8dba74-10b1-89c8-2ab4-312f399761eb"
2025-07-05T01:12:20.942Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to appendEntries to: peer="{Voter server-680194cb-d075-343f-ceff-cc084646d23c 680194cb-d075-343f-ceff-cc084646d23c}" error="failed to connect to peer: 680194cb-d075-343f-ceff-cc084646d23c"
2025-07-05T01:12:20.948Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to heartbeat to: peer=2f8dba74-10b1-89c8-2ab4-312f399761eb backoff time=10ms error="failed to connect to peer: 2f8dba74-10b1-89c8-2ab4-312f399761eb"
2025-07-05T01:12:20.953Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to heartbeat to: peer=680194cb-d075-343f-ceff-cc084646d23c backoff time=10ms error="failed to connect to peer: 680194cb-d075-343f-ceff-cc084646d23c"
2025-07-05T01:12:20.963Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to appendEntries to: peer="{Voter server-2f8dba74-10b1-89c8-2ab4-312f399761eb 2f8dba74-10b1-89c8-2ab4-312f399761eb}" error="failed to connect to peer: 2f8dba74-10b1-89c8-2ab4-312f399761eb"
2025-07-05T01:12:20.967Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to appendEntries to: peer="{Voter server-680194cb-d075-343f-ceff-cc084646d23c 680194cb-d075-343f-ceff-cc084646d23c}" error="failed to connect to peer: 680194cb-d075-343f-ceff-cc084646d23c"
2025-07-05T01:12:20.967Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to heartbeat to: peer=2f8dba74-10b1-89c8-2ab4-312f399761eb backoff time=10ms error="failed to connect to peer: 2f8dba74-10b1-89c8-2ab4-312f399761eb"
2025-07-05T01:12:20.976Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to heartbeat to: peer=680194cb-d075-343f-ceff-cc084646d23c backoff time=10ms error="failed to connect to peer: 680194cb-d075-343f-ceff-cc084646d23c"
2025-07-05T01:12:20.977Z [WARN]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to contact: server-id=server-680194cb-d075-343f-ceff-cc084646d23c time=50.851937ms
2025-07-05T01:12:20.980Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to appendEntries to: peer="{Voter server-2f8dba74-10b1-89c8-2ab4-312f399761eb 2f8dba74-10b1-89c8-2ab4-312f399761eb}" error="failed to connect to peer: 2f8dba74-10b1-89c8-2ab4-312f399761eb"
2025-07-05T01:12:20.986Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to appendEntries to: peer="{Voter server-680194cb-d075-343f-ceff-cc084646d23c 680194cb-d075-343f-ceff-cc084646d23c}" error="failed to connect to peer: 680194cb-d075-343f-ceff-cc084646d23c"
2025-07-05T01:12:20.986Z [ERROR] server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to heartbeat to: peer=2f8dba74-10b1-89c8-2ab4-312f399761eb backoff time=20ms error="failed to connect to peer: 2f8dba74-10b1-89c8-2ab4-312f399761eb"
2025-07-05T01:12:20.987Z [WARN]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to contact: server-id=server-680194cb-d075-343f-ceff-cc084646d23c time=60.966933ms
2025-07-05T01:12:20.987Z [WARN]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to contact: server-id=server-2f8dba74-10b1-89c8-2ab4-312f399761eb time=58.558898ms
2025-07-05T01:12:20.987Z [WARN]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: failed to contact quorum of nodes, stepping down
2025-07-05T01:12:20.987Z [INFO]  server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291: entering follower state: follower="Node at c4a5a28c-f4ff-8d16-6dba-070a90fdd291 [Follower]" leader-address= leader-id=
2025-07-05T01:12:20.992Z [WARN]  server-680194cb-d075-343f-ceff-cc084646d23c: heartbeat timeout reached, starting election: last-leader-addr=c4a5a28c-f4ff-8d16-6dba-070a90fdd291 last-leader-id=server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291
2025-07-05T01:12:20.992Z [INFO]  server-680194cb-d075-343f-ceff-cc084646d23c: entering candidate state: node="Node at 680194cb-d075-343f-ceff-cc084646d23c [Candidate]" term=3
2025-07-05T01:12:20.992Z [WARN]  server-2f8dba74-10b1-89c8-2ab4-312f399761eb: rejecting vote request since we have a leader: from=680194cb-d075-343f-ceff-cc084646d23c leader=c4a5a28c-f4ff-8d16-6dba-070a90fdd291 leader-id=server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291
2025-07-05T01:12:20.992Z [ERROR] server-680194cb-d075-343f-ceff-cc084646d23c: failed to make requestVote RPC: target="{Voter server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291 c4a5a28c-f4ff-8d16-6dba-070a90fdd291}" error="failed to connect to peer: c4a5a28c-f4ff-8d16-6dba-070a90fdd291" term=3
2025-07-05T01:12:20.997Z [WARN]  server-2f8dba74-10b1-89c8-2ab4-312f399761eb: heartbeat timeout reached, starting election: last-leader-addr=c4a5a28c-f4ff-8d16-6dba-070a90fdd291 last-leader-id=server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291
2025-07-05T01:12:20.997Z [INFO]  server-2f8dba74-10b1-89c8-2ab4-312f399761eb: entering candidate state: node="Node at 2f8dba74-10b1-89c8-2ab4-312f399761eb [Candidate]" term=3
2025-07-05T01:12:20.997Z [INFO]  server-680194cb-d075-343f-ceff-cc084646d23c: duplicate requestVote for same term: term=3
2025-07-05T01:12:20.997Z [ERROR] server-2f8dba74-10b1-89c8-2ab4-312f399761eb: failed to make requestVote RPC: target="{Voter server-c4a5a28c-f4ff-8d16-6dba-070a90fdd291 c4a5a28c-f4ff-8d16-6dba-070a90fdd291}" error="failed to connect to peer: c4a5a28c-f4ff-8d16-6dba-070a90fdd291" term=3
2025-07-05T01:12:21.037Z [INFO]  server-680194cb-d075-343f-ceff-cc084646d23c: entering follower state: follower="Node at 680194cb-d075-343f-ceff-cc084646d23c [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.49s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-07-05T01:12:21.038Z [INFO]  127.0.0.1:33907 :: starting node: addr=127.0.0.1:33907
2025-07-05T01:12:21.038Z [INFO]  127.0.0.1:33907 :: initial configuration: index=0 servers=[]
2025-07-05T01:12:21.038Z [INFO]  127.0.0.1:33907 :: entering follower state: follower="Node at 127.0.0.1:33907 [Follower]" leader-address= leader-id=
2025-07-05T01:12:21.112Z [WARN]  127.0.0.1:33907 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-07-05T01:12:21.189Z [INFO]  127.0.0.1:46275 :: starting node: addr=127.0.0.1:46275
2025-07-05T01:12:21.189Z [INFO]  127.0.0.1:46275 :: initial configuration: index=0 servers=[]
2025-07-05T01:12:21.189Z [INFO]  127.0.0.1:46275 :: entering follower state: follower="Node at 127.0.0.1:46275 [Follower]" leader-address= leader-id=
2025-07-05T01:12:21.289Z [WARN]  127.0.0.1:46275 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-07-05T01:12:21.340Z [INFO]  127.0.0.1:40391 :: starting node: addr=127.0.0.1:40391
2025-07-05T01:12:21.340Z [INFO]  127.0.0.1:40391 :: initial configuration: index=0 servers=[]
2025-07-05T01:12:21.340Z [INFO]  127.0.0.1:40391 :: entering follower state: follower="Node at 127.0.0.1:40391 [Follower]" leader-address= leader-id=
2025-07-05T01:12:21.421Z [WARN]  127.0.0.1:40391 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-07-05T01:12:21.491Z [INFO]  127.0.0.1:45329 :: starting node: addr=127.0.0.1:45329
2025-07-05T01:12:21.491Z [INFO]  127.0.0.1:45329 :: initial configuration: index=0 servers=[]
2025-07-05T01:12:21.491Z [INFO]  127.0.0.1:45329 :: entering follower state: follower="Node at 127.0.0.1:45329 [Follower]" leader-address= leader-id=
2025-07-05T01:12:21.492Z [WARN]  127.0.0.1:45329 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.01s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	116.930s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/B01_cleanup starting
I: user script /srv/workspace/pbuilder/827298/tmp/hooks/B01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/827298 and its subdirectories
I: Current time: Sat Jul  5 15:12:29 +14 2025
I: pbuilder-time-stamp: 1751677949