I: pbuilder: network access will be disabled during build
I: Current time: Mon Jul 14 19:37:34 +14 2025
I: pbuilder-time-stamp: 1752471454
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: using eatmydata during job
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
gpgv: Signature made Thu Jul 27 15:15:46 2023 
gpgv:                using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29
gpgv:                issuer "dswarbrick@debian.org"
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/D01_modify_environment starting
debug: Running on infom08-i386.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 Jul 14 05:37 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="21" [3]="1" [4]="release" [5]="i686-pc-linux-gnu")
  BASH_VERSION='5.2.21(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=i386
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=5 '
  DIRSTACK=()
  DISTRIBUTION=unstable
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=i686
  HOST_ARCH=i386
  IFS=' 	
  '
  INVOCATION_ID=d6b3436bd57e495cac2c5a3293f06e0c
  LANG=C
  LANGUAGE=de_CH:de
  LC_ALL=C
  LD_LIBRARY_PATH=/usr/lib/libeatmydata
  LD_PRELOAD=libeatmydata.so
  MACHTYPE=i686-pc-linux-gnu
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnu
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=20195
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.IQFZDLkI/pbuilderrc_KfLI --distribution unstable --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.IQFZDLkI/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID=111
  SUDO_UID=104
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
I: uname -a
  Linux i-capture-the-hostname 6.7.12+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.12-1~bpo12+1 (2024-05-06) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Jun  5  2024 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: i386
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19700 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.22-go{a} golang-1.22-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 
0 packages upgraded, 102 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.6 MB of archives. After unpacking 619 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian unstable/main i386 sensible-utils all 0.0.22 [22.4 kB]
Get: 2 http://deb.debian.org/debian unstable/main i386 libmagic-mgc i386 1:5.45-3 [314 kB]
Get: 3 http://deb.debian.org/debian unstable/main i386 libmagic1t64 i386 1:5.45-3 [114 kB]
Get: 4 http://deb.debian.org/debian unstable/main i386 file i386 1:5.45-3 [42.9 kB]
Get: 5 http://deb.debian.org/debian unstable/main i386 gettext-base i386 0.21-14+b1 [162 kB]
Get: 6 http://deb.debian.org/debian unstable/main i386 libuchardet0 i386 0.0.8-1+b1 [69.1 kB]
Get: 7 http://deb.debian.org/debian unstable/main i386 groff-base i386 1.23.0-4 [1194 kB]
Get: 8 http://deb.debian.org/debian unstable/main i386 bsdextrautils i386 2.40.1-8.1 [100 kB]
Get: 9 http://deb.debian.org/debian unstable/main i386 libpipeline1 i386 1.5.7-2 [39.7 kB]
Get: 10 http://deb.debian.org/debian unstable/main i386 man-db i386 2.12.1-2 [1421 kB]
Get: 11 http://deb.debian.org/debian unstable/main i386 m4 i386 1.4.19-4 [293 kB]
Get: 12 http://deb.debian.org/debian unstable/main i386 autoconf all 2.71-3 [332 kB]
Get: 13 http://deb.debian.org/debian unstable/main i386 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian unstable/main i386 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian unstable/main i386 autopoint all 0.21-14 [496 kB]
Get: 16 http://deb.debian.org/debian unstable/main i386 libdebhelper-perl all 13.15.3 [88.0 kB]
Get: 17 http://deb.debian.org/debian unstable/main i386 libtool all 2.4.7-7 [517 kB]
Get: 18 http://deb.debian.org/debian unstable/main i386 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian unstable/main i386 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian unstable/main i386 libfile-stripnondeterminism-perl all 1.14.0-1 [19.5 kB]
Get: 21 http://deb.debian.org/debian unstable/main i386 dh-strip-nondeterminism all 1.14.0-1 [8448 B]
Get: 22 http://deb.debian.org/debian unstable/main i386 libelf1t64 i386 0.191-1+b1 [194 kB]
Get: 23 http://deb.debian.org/debian unstable/main i386 dwz i386 0.15-1+b1 [116 kB]
Get: 24 http://deb.debian.org/debian unstable/main i386 libicu72 i386 72.1-4+b1 [9549 kB]
Get: 25 http://deb.debian.org/debian unstable/main i386 libxml2 i386 2.12.7+dfsg-3 [704 kB]
Get: 26 http://deb.debian.org/debian unstable/main i386 gettext i386 0.21-14+b1 [1311 kB]
Get: 27 http://deb.debian.org/debian unstable/main i386 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 28 http://deb.debian.org/debian unstable/main i386 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 29 http://deb.debian.org/debian unstable/main i386 debhelper all 13.15.3 [901 kB]
Get: 30 http://deb.debian.org/debian unstable/main i386 dh-golang all 1.62 [24.7 kB]
Get: 31 http://deb.debian.org/debian unstable/main i386 golang-1.22-src all 1.22.4-1 [18.8 MB]
Get: 32 http://deb.debian.org/debian unstable/main i386 golang-1.22-go i386 1.22.4-1 [22.8 MB]
Get: 33 http://deb.debian.org/debian unstable/main i386 golang-src all 2:1.22~3 [5020 B]
Get: 34 http://deb.debian.org/debian unstable/main i386 golang-go i386 2:1.22~3 [44.2 kB]
Get: 35 http://deb.debian.org/debian unstable/main i386 golang-any i386 2:1.22~3 [5104 B]
Get: 36 http://deb.debian.org/debian unstable/main i386 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B]
Get: 37 http://deb.debian.org/debian unstable/main i386 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 38 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 39 http://deb.debian.org/debian unstable/main i386 golang-golang-x-sys-dev all 0.20.0-1 [430 kB]
Get: 40 http://deb.debian.org/debian unstable/main i386 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 41 http://deb.debian.org/debian unstable/main i386 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 42 http://deb.debian.org/debian unstable/main i386 golang-github-fatih-color-dev all 1.16.0-1 [13.8 kB]
Get: 43 http://deb.debian.org/debian unstable/main i386 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 44 http://deb.debian.org/debian unstable/main i386 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 45 http://deb.debian.org/debian unstable/main i386 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 46 http://deb.debian.org/debian unstable/main i386 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 47 http://deb.debian.org/debian unstable/main i386 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 48 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 49 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 50 http://deb.debian.org/debian unstable/main i386 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 51 http://deb.debian.org/debian unstable/main i386 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 52 http://deb.debian.org/debian unstable/main i386 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 53 http://deb.debian.org/debian unstable/main i386 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 54 http://deb.debian.org/debian unstable/main i386 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 55 http://deb.debian.org/debian unstable/main i386 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 56 http://deb.debian.org/debian unstable/main i386 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 57 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 58 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 59 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 60 http://deb.debian.org/debian unstable/main i386 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 61 http://deb.debian.org/debian unstable/main i386 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 62 http://deb.debian.org/debian unstable/main i386 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B]
Get: 63 http://deb.debian.org/debian unstable/main i386 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 64 http://deb.debian.org/debian unstable/main i386 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 65 http://deb.debian.org/debian unstable/main i386 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 66 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 67 http://deb.debian.org/debian unstable/main i386 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 68 http://deb.debian.org/debian unstable/main i386 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 69 http://deb.debian.org/debian unstable/main i386 golang-golang-x-term-dev all 0.20.0-1 [14.8 kB]
Get: 70 http://deb.debian.org/debian unstable/main i386 golang-golang-x-text-dev all 0.15.0-1 [4006 kB]
Get: 71 http://deb.debian.org/debian unstable/main i386 golang-golang-x-crypto-dev all 1:0.23.0-1 [1689 kB]
Get: 72 http://deb.debian.org/debian unstable/main i386 golang-golang-x-net-dev all 1:0.25.0+dfsg-1 [895 kB]
Get: 73 http://deb.debian.org/debian unstable/main i386 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 74 http://deb.debian.org/debian unstable/main i386 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 75 http://deb.debian.org/debian unstable/main i386 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 76 http://deb.debian.org/debian unstable/main i386 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 77 http://deb.debian.org/debian unstable/main i386 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 78 http://deb.debian.org/debian unstable/main i386 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 79 http://deb.debian.org/debian unstable/main i386 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 80 http://deb.debian.org/debian unstable/main i386 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 81 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-common-dev all 0.53.0-1 [114 kB]
Get: 82 http://deb.debian.org/debian unstable/main i386 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 83 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 84 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-client-golang-dev all 1.19.0-1 [179 kB]
Get: 85 http://deb.debian.org/debian unstable/main i386 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 86 http://deb.debian.org/debian unstable/main i386 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 87 http://deb.debian.org/debian unstable/main i386 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 88 http://deb.debian.org/debian unstable/main i386 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 89 http://deb.debian.org/debian unstable/main i386 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 90 http://deb.debian.org/debian unstable/main i386 libsasl2-modules-db i386 2.1.28+dfsg1-6 [20.5 kB]
Get: 91 http://deb.debian.org/debian unstable/main i386 libsasl2-2 i386 2.1.28+dfsg1-6 [60.6 kB]
Get: 92 http://deb.debian.org/debian unstable/main i386 libssl-dev i386 3.2.2-1 [2748 kB]
Get: 93 http://deb.debian.org/debian unstable/main i386 libsasl2-dev i386 2.1.28+dfsg1-6 [244 kB]
Get: 94 http://deb.debian.org/debian unstable/main i386 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 95 http://deb.debian.org/debian unstable/main i386 golang-golang-x-mod-dev all 0.17.0-1 [103 kB]
Get: 96 http://deb.debian.org/debian unstable/main i386 golang-github-yuin-goldmark-dev all 1.7.1-1 [113 kB]
Get: 97 http://deb.debian.org/debian unstable/main i386 golang-golang-x-tools-dev all 1:0.20.0+ds-1 [1571 kB]
Get: 98 http://deb.debian.org/debian unstable/main i386 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 99 http://deb.debian.org/debian unstable/main i386 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 100 http://deb.debian.org/debian unstable/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 101 http://deb.debian.org/debian unstable/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 102 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 88.6 MB in 1s (96.4 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19700 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.22_all.deb ...
Unpacking sensible-utils (0.0.22) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3_i386.deb ...
Unpacking libmagic-mgc (1:5.45-3) ...
Selecting previously unselected package libmagic1t64:i386.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3_i386.deb ...
Unpacking libmagic1t64:i386 (1:5.45-3) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3_i386.deb ...
Unpacking file (1:5.45-3) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.21-14+b1_i386.deb ...
Unpacking gettext-base (0.21-14+b1) ...
Selecting previously unselected package libuchardet0:i386.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b1_i386.deb ...
Unpacking libuchardet0:i386 (0.0.8-1+b1) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-4_i386.deb ...
Unpacking groff-base (1.23.0-4) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.1-8.1_i386.deb ...
Unpacking bsdextrautils (2.40.1-8.1) ...
Selecting previously unselected package libpipeline1:i386.
Preparing to unpack .../008-libpipeline1_1.5.7-2_i386.deb ...
Unpacking libpipeline1:i386 (1.5.7-2) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.12.1-2_i386.deb ...
Unpacking man-db (2.12.1-2) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-4_i386.deb ...
Unpacking m4 (1.4.19-4) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.71-3_all.deb ...
Unpacking autoconf (2.71-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.21-14_all.deb ...
Unpacking autopoint (0.21-14) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.15.3_all.deb ...
Unpacking libdebhelper-perl (13.15.3) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.4.7-7_all.deb ...
Unpacking libtool (2.4.7-7) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.0-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.0-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.0-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.0-1) ...
Selecting previously unselected package libelf1t64:i386.
Preparing to unpack .../021-libelf1t64_0.191-1+b1_i386.deb ...
Unpacking libelf1t64:i386 (0.191-1+b1) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_i386.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libicu72:i386.
Preparing to unpack .../023-libicu72_72.1-4+b1_i386.deb ...
Unpacking libicu72:i386 (72.1-4+b1) ...
Selecting previously unselected package libxml2:i386.
Preparing to unpack .../024-libxml2_2.12.7+dfsg-3_i386.deb ...
Unpacking libxml2:i386 (2.12.7+dfsg-3) ...
Selecting previously unselected package gettext.
Preparing to unpack .../025-gettext_0.21-14+b1_i386.deb ...
Unpacking gettext (0.21-14+b1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../028-debhelper_13.15.3_all.deb ...
Unpacking debhelper (13.15.3) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../029-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.22-src.
Preparing to unpack .../030-golang-1.22-src_1.22.4-1_all.deb ...
Unpacking golang-1.22-src (1.22.4-1) ...
Selecting previously unselected package golang-1.22-go.
Preparing to unpack .../031-golang-1.22-go_1.22.4-1_i386.deb ...
Unpacking golang-1.22-go (1.22.4-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../032-golang-src_2%3a1.22~3_all.deb ...
Unpacking golang-src (2:1.22~3) ...
Selecting previously unselected package golang-go:i386.
Preparing to unpack .../033-golang-go_2%3a1.22~3_i386.deb ...
Unpacking golang-go:i386 (2:1.22~3) ...
Selecting previously unselected package golang-any:i386.
Preparing to unpack .../034-golang-any_2%3a1.22~3_i386.deb ...
Unpacking golang-any:i386 (2:1.22~3) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../038-golang-golang-x-sys-dev_0.20.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.20.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../041-golang-github-fatih-color-dev_1.16.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.16.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../054-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../065-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../066-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../067-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../068-golang-golang-x-term-dev_0.20.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.20.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../069-golang-golang-x-text-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.15.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../070-golang-golang-x-crypto-dev_1%3a0.23.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.23.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../071-golang-golang-x-net-dev_1%3a0.25.0+dfsg-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.25.0+dfsg-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../072-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../073-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../074-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../075-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../076-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../078-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../079-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../080-golang-github-prometheus-common-dev_0.53.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.53.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../081-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../082-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../083-golang-github-prometheus-client-golang-dev_1.19.0-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../084-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../085-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../086-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../087-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../088-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:i386.
Preparing to unpack .../089-libsasl2-modules-db_2.1.28+dfsg1-6_i386.deb ...
Unpacking libsasl2-modules-db:i386 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libsasl2-2:i386.
Preparing to unpack .../090-libsasl2-2_2.1.28+dfsg1-6_i386.deb ...
Unpacking libsasl2-2:i386 (2.1.28+dfsg1-6) ...
Selecting previously unselected package libssl-dev:i386.
Preparing to unpack .../091-libssl-dev_3.2.2-1_i386.deb ...
Unpacking libssl-dev:i386 (3.2.2-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../092-libsasl2-dev_2.1.28+dfsg1-6_i386.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-6) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../093-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../094-golang-golang-x-mod-dev_0.17.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.17.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../095-golang-github-yuin-goldmark-dev_1.7.1-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.1-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../096-golang-golang-x-tools-dev_1%3a0.20.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.20.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../097-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../098-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../099-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../100-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../101-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:i386 (1.5.7-2) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:i386 (72.1-4+b1) ...
Setting up bsdextrautils (2.40.1-8.1) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.15.3) ...
Setting up golang-golang-x-sys-dev (0.20.0-1) ...
Setting up libmagic1t64:i386 (1:5.45-3) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.21-14+b1) ...
Setting up m4 (1.4.19-4) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.20.0-1) ...
Setting up file (1:5.45-3) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:i386 (0.191-1+b1) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:i386 (2.1.28+dfsg1-6) ...
Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.1-1) ...
Setting up golang-1.22-src (1.22.4-1) ...
Setting up libssl-dev:i386 (3.2.2-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.17.0-1) ...
Setting up autopoint (0.21-14) ...
Setting up libsasl2-2:i386 (2.1.28+dfsg1-6) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.71-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up sensible-utils (0.0.22) ...
Setting up golang-golang-x-text-dev (0.15.0-1) ...
Setting up libuchardet0:i386 (0.0.8-1+b1) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.23.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.22~3) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:i386 (2.12.7+dfsg-3) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.0-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.21-14+b1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.4.7-7) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-1.22-go (1.22.4-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-6) ...
Setting up golang-golang-x-net-dev (1:0.25.0+dfsg-1) ...
Setting up dh-strip-nondeterminism (1.14.0-1) ...
Setting up groff-base (1.23.0-4) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.16.0-1) ...
Setting up golang-go:i386 (2:1.22~3) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.20.0+ds-1) ...
Setting up man-db (2.12.1-2) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:i386 (2:1.22~3) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.15.3) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.53.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.19.0-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.38-12.1) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for unstable
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture i386
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-i686-linux-gnu && go install -trimpath -v -p 5 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/unsafeheader
internal/goarch
internal/godebugs
internal/coverage/rtcov
internal/cpu
internal/goexperiment
internal/chacha8rand
internal/abi
internal/goos
runtime/internal/atomic
runtime/internal/math
runtime/internal/sys
internal/bytealg
runtime/internal/syscall
internal/race
sync/atomic
unicode
unicode/utf8
container/list
internal/itoa
math/bits
crypto/internal/alias
crypto/internal/boring/sig
cmp
encoding
slices
math
unicode/utf16
log/internal
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
runtime
crypto/subtle
internal/reflectlite
sync
internal/testlog
internal/bisect
internal/singleflight
runtime/cgo
errors
sort
io
internal/oserror
internal/godebug
syscall
strconv
internal/safefilepath
bytes
strings
hash
crypto/internal/randutil
bufio
path
crypto
reflect
math/rand
github.com/hashicorp/golang-lru/simplelru
github.com/hashicorp/go-immutable-radix
vendor/golang.org/x/net/dns/dnsmessage
time
internal/syscall/unix
internal/syscall/execenv
internal/intern
net/netip
hash/crc32
crypto/internal/nistec/fiat
crypto/rc4
vendor/golang.org/x/text/transform
context
internal/poll
io/fs
net/http/internal/ascii
regexp/syntax
encoding/binary
internal/fmtsort
os
internal/saferio
html
crypto/cipher
encoding/base64
crypto/internal/edwards25519/field
crypto/md5
crypto/internal/boring
crypto/aes
crypto/des
crypto/sha512
fmt
net
crypto/internal/edwards25519
crypto/hmac
crypto/sha1
crypto/internal/nistec
crypto/sha256
encoding/pem
path/filepath
vendor/golang.org/x/crypto/chacha20
math/big
encoding/json
log
compress/flate
crypto/ecdh
encoding/hex
net/url
compress/gzip
vendor/golang.org/x/crypto/internal/poly1305
vendor/golang.org/x/crypto/hkdf
vendor/golang.org/x/crypto/chacha20poly1305
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/net/http2/hpack
mime
crypto/rand
crypto/elliptic
crypto/internal/bigmod
crypto/internal/boring/bbig
encoding/asn1
crypto/ed25519
crypto/rsa
crypto/dsa
vendor/golang.org/x/text/secure/bidirule
mime/quotedprintable
net/http/internal
os/signal
vendor/golang.org/x/net/idna
golang.org/x/sys/unix
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
io/ioutil
regexp
crypto/ecdsa
encoding/gob
go/token
text/template/parse
hash/crc64
flag
internal/sysinfo
runtime/debug
text/template
runtime/trace
testing
github.com/mattn/go-isatty
html/template
github.com/mattn/go-colorable
github.com/fatih/color
github.com/hashicorp/go-hclog
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-i686-linux-gnu && go test -vet=off -v -p 5 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-07-14T05:38:20.587Z [INFO]  creating new snapshot: path=/tmp/raft1399998470/raft1124366348/snapshots/3-10-1752471500587.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.03s)
=== RUN   TestFileSS_CreateSnapshot
2025-07-14T05:38:20.615Z [INFO]  creating new snapshot: path=/tmp/raft1036997727/snapshots/3-10-1752471500615.tmp
2025-07-14T05:38:20.619Z [WARN]  found temporary snapshot: name=3-10-1752471500615.tmp
--- PASS: TestFileSS_CreateSnapshot (0.01s)
=== RUN   TestFileSS_CancelSnapshot
2025-07-14T05:38:20.629Z [INFO]  creating new snapshot: path=/tmp/raft46107243/snapshots/3-10-1752471500629.tmp
--- PASS: TestFileSS_CancelSnapshot (0.01s)
=== RUN   TestFileSS_Retention
2025-07-14T05:38:20.635Z [INFO]  creating new snapshot: path=/tmp/raft3399887281/snapshots/3-10-1752471500635.tmp
2025-07-14T05:38:20.651Z [INFO]  creating new snapshot: path=/tmp/raft3399887281/snapshots/3-11-1752471500651.tmp
2025-07-14T05:38:20.662Z [INFO]  creating new snapshot: path=/tmp/raft3399887281/snapshots/3-12-1752471500662.tmp
2025-07-14T05:38:20.674Z [INFO]  reaping snapshot: path=/tmp/raft3399887281/snapshots/3-10-1752471500635
2025-07-14T05:38:20.674Z [INFO]  creating new snapshot: path=/tmp/raft3399887281/snapshots/3-13-1752471500674.tmp
2025-07-14T05:38:20.684Z [INFO]  reaping snapshot: path=/tmp/raft3399887281/snapshots/3-11-1752471500651
2025-07-14T05:38:20.685Z [INFO]  creating new snapshot: path=/tmp/raft3399887281/snapshots/3-14-1752471500685.tmp
2025-07-14T05:38:20.697Z [INFO]  reaping snapshot: path=/tmp/raft3399887281/snapshots/3-12-1752471500662
--- PASS: TestFileSS_Retention (0.06s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-07-14T05:38:20.699Z [INFO]  creating new snapshot: path=/tmp/raft4156954490/snapshots/5-130350-1752471500699.tmp
2025-07-14T05:38:20.710Z [INFO]  creating new snapshot: path=/tmp/raft4156954490/snapshots/36-204917-1752471500710.tmp
--- PASS: TestFileSS_Ordering (0.02s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2025-07-14T05:38:20.811Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2025-07-14T05:38:20.862Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
2025-07-14T05:38:20.914Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.15s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:20.919Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:20.925Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:20.935Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:20.956Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:20.996Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:21.077Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:21.237Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:21.558Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:22.199Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:23.201Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-07-14T05:38:24.203Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-07-14T05:38:24.921Z [INFO]  server-7641c17d-1d78-8a8f-d8f6-dab2e44cfd6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7641c17d-1d78-8a8f-d8f6-dab2e44cfd6b Address:7641c17d-1d78-8a8f-d8f6-dab2e44cfd6b}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-07-14T05:38:24.922Z [INFO]  server-ef6ebe3b-7d62-9b66-3282-b51067afe866: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef6ebe3b-7d62-9b66-3282-b51067afe866 Address:ef6ebe3b-7d62-9b66-3282-b51067afe866}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-07-14T05:38:24.922Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: initial configuration: index=0 servers=[]
2025-07-14T05:38:24.922Z [INFO]  server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae: initial configuration: index=0 servers=[]
2025-07-14T05:38:24.922Z [INFO]  server-15d735b4-610d-3f61-636a-fb6285fc477f: initial configuration: index=0 servers=[]
2025-07-14T05:38:24.922Z [INFO]  server-15d735b4-610d-3f61-636a-fb6285fc477f: entering follower state: follower="Node at 15d735b4-610d-3f61-636a-fb6285fc477f [Follower]" leader-address= leader-id=
2025-07-14T05:38:24.922Z [INFO]  server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae: entering follower state: follower="Node at fb7710ec-0832-fb3a-94b9-e2abd11e99ae [Follower]" leader-address= leader-id=
2025-07-14T05:38:24.922Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: entering follower state: follower="Node at 771ed5fd-687a-1346-cd91-b1533088ab97 [Follower]" leader-address= leader-id=
2025-07-14T05:38:24.923Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:24.983Z [WARN]  server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae: no known peers, aborting election
2025-07-14T05:38:24.989Z [WARN]  server-15d735b4-610d-3f61-636a-fb6285fc477f: no known peers, aborting election
2025-07-14T05:38:24.998Z [WARN]  server-771ed5fd-687a-1346-cd91-b1533088ab97: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:24.998Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: entering candidate state: node="Node at 771ed5fd-687a-1346-cd91-b1533088ab97 [Candidate]" term=2
2025-07-14T05:38:24.998Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: election won: term=2 tally=2
2025-07-14T05:38:24.998Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: entering leader state: leader="Node at 771ed5fd-687a-1346-cd91-b1533088ab97 [Leader]"
2025-07-14T05:38:24.998Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: added peer, starting replication: peer=server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae
2025-07-14T05:38:24.998Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: added peer, starting replication: peer=server-15d735b4-610d-3f61-636a-fb6285fc477f
2025-07-14T05:38:24.998Z [WARN]  server-15d735b4-610d-3f61-636a-fb6285fc477f: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-07-14T05:38:24.998Z [WARN]  server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-07-14T05:38:24.998Z [WARN]  server-771ed5fd-687a-1346-cd91-b1533088ab97: appendEntries rejected, sending older logs: peer="{Voter server-15d735b4-610d-3f61-636a-fb6285fc477f 15d735b4-610d-3f61-636a-fb6285fc477f}" next=1
2025-07-14T05:38:24.998Z [WARN]  server-771ed5fd-687a-1346-cd91-b1533088ab97: appendEntries rejected, sending older logs: peer="{Voter server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae fb7710ec-0832-fb3a-94b9-e2abd11e99ae}" next=1
2025-07-14T05:38:24.999Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: pipelining replication: peer="{Voter server-15d735b4-610d-3f61-636a-fb6285fc477f 15d735b4-610d-3f61-636a-fb6285fc477f}"
2025-07-14T05:38:24.999Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: pipelining replication: peer="{Voter server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae fb7710ec-0832-fb3a-94b9-e2abd11e99ae}"
2025-07-14T05:38:25.104Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:24.998936558 +0000 UTC m=+4.413396956 (2 nodes), 75.868283ms from start of poll, 76.258478ms from cluster start. Timeout at 2025-07-14 05:38:25.104641645 +0000 UTC m=+4.519102053, 105.705097ms after stability
2025-07-14T05:38:25.104Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:25.210Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:25.104716347 +0000 UTC m=+4.519176725 (1 nodes), 791ns from start of poll, 182.038247ms from cluster start. Timeout at 2025-07-14 05:38:25.210599372 +0000 UTC m=+4.625059750, 105.883025ms after stability
2025-07-14T05:38:25.221Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: aborting pipeline replication: peer="{Voter server-fb7710ec-0832-fb3a-94b9-e2abd11e99ae fb7710ec-0832-fb3a-94b9-e2abd11e99ae}"
2025-07-14T05:38:25.221Z [INFO]  server-771ed5fd-687a-1346-cd91-b1533088ab97: aborting pipeline replication: peer="{Voter server-15d735b4-610d-3f61-636a-fb6285fc477f 15d735b4-610d-3f61-636a-fb6285fc477f}"
--- PASS: TestRaft_LiveBootstrap (0.30s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-07-14T05:38:25.222Z [INFO]  server-670bb13f-3171-5aa4-020f-a70a54a53829: initial configuration: index=0 servers=[]
2025-07-14T05:38:25.222Z [INFO]  server-4968b72a-cac2-2296-4679-88ef027400a6: initial configuration: index=0 servers=[]
2025-07-14T05:38:25.222Z [INFO]  server-670bb13f-3171-5aa4-020f-a70a54a53829: entering follower state: follower="Node at 670bb13f-3171-5aa4-020f-a70a54a53829 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.222Z [INFO]  server-4968b72a-cac2-2296-4679-88ef027400a6: entering follower state: follower="Node at 4968b72a-cac2-2296-4679-88ef027400a6 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-07-14T05:38:25.223Z [INFO]  server-19e9a4f3-d759-2092-70d4-34f23e17ae00: initial configuration: index=0 servers=[]
2025-07-14T05:38:25.223Z [INFO]  server-19e9a4f3-d759-2092-70d4-34f23e17ae00: entering follower state: follower="Node at 19e9a4f3-d759-2092-70d4-34f23e17ae00 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-07-14T05:38:25.223Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7402b19-355b-bcb1-67a3-108dd5072c25 Address:c7402b19-355b-bcb1-67a3-108dd5072c25} {Suffrage:Voter ID:server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f Address:7245a63e-bfc6-c0d8-499e-40b7d3581b8f} {Suffrage:Voter ID:server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 Address:f0d80198-f8f6-6fe6-07c7-c937e13005e1}]"
2025-07-14T05:38:25.223Z [INFO]  server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7402b19-355b-bcb1-67a3-108dd5072c25 Address:c7402b19-355b-bcb1-67a3-108dd5072c25} {Suffrage:Voter ID:server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f Address:7245a63e-bfc6-c0d8-499e-40b7d3581b8f} {Suffrage:Voter ID:server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 Address:f0d80198-f8f6-6fe6-07c7-c937e13005e1}]"
2025-07-14T05:38:25.223Z [INFO]  server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f: entering follower state: follower="Node at 7245a63e-bfc6-c0d8-499e-40b7d3581b8f [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.223Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: entering follower state: follower="Node at c7402b19-355b-bcb1-67a3-108dd5072c25 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.224Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7402b19-355b-bcb1-67a3-108dd5072c25 Address:c7402b19-355b-bcb1-67a3-108dd5072c25} {Suffrage:Voter ID:server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f Address:7245a63e-bfc6-c0d8-499e-40b7d3581b8f} {Suffrage:Voter ID:server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 Address:f0d80198-f8f6-6fe6-07c7-c937e13005e1}]"
2025-07-14T05:38:25.224Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:25.224Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: entering follower state: follower="Node at f0d80198-f8f6-6fe6-07c7-c937e13005e1 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.276Z [WARN]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:25.276Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: entering candidate state: node="Node at c7402b19-355b-bcb1-67a3-108dd5072c25 [Candidate]" term=2
2025-07-14T05:38:25.276Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: election won: term=2 tally=2
2025-07-14T05:38:25.276Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: entering leader state: leader="Node at c7402b19-355b-bcb1-67a3-108dd5072c25 [Leader]"
2025-07-14T05:38:25.276Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: added peer, starting replication: peer=server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f
2025-07-14T05:38:25.276Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: added peer, starting replication: peer=server-f0d80198-f8f6-6fe6-07c7-c937e13005e1
2025-07-14T05:38:25.276Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: pipelining replication: peer="{Voter server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f 7245a63e-bfc6-c0d8-499e-40b7d3581b8f}"
2025-07-14T05:38:25.276Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: pipelining replication: peer="{Voter server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 f0d80198-f8f6-6fe6-07c7-c937e13005e1}"
2025-07-14T05:38:25.382Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:25.276623919 +0000 UTC m=+4.691084297 (1 nodes), 52.565018ms from start of poll, 52.851367ms from cluster start. Timeout at 2025-07-14 05:38:25.382042847 +0000 UTC m=+4.796503245, 105.418948ms after stability
2025-07-14T05:38:25.382Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: aborting pipeline replication: peer="{Voter server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 f0d80198-f8f6-6fe6-07c7-c937e13005e1}"
2025-07-14T05:38:25.382Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: aborting pipeline replication: peer="{Voter server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f 7245a63e-bfc6-c0d8-499e-40b7d3581b8f}"
2025-07-14T05:38:25.382Z [INFO]  creating new snapshot: path=/tmp/raft2192767570/snapshots/2-2-1752471505382.tmp
2025-07-14T05:38:25.382Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: starting restore from snapshot: id=2-2-1752471505382 last-index=2 last-term=2 size-in-bytes=1
2025-07-14T05:38:25.382Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: snapshot restore progress: id=2-2-1752471505382 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-07-14T05:38:25.383Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: restored from snapshot: id=2-2-1752471505382 last-index=2 last-term=2 size-in-bytes=1
2025-07-14T05:38:25.383Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7402b19-355b-bcb1-67a3-108dd5072c25 Address:c7402b19-355b-bcb1-67a3-108dd5072c25} {Suffrage:Voter ID:server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f Address:7245a63e-bfc6-c0d8-499e-40b7d3581b8f} {Suffrage:Voter ID:server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 Address:f0d80198-f8f6-6fe6-07c7-c937e13005e1}]"
2025-07-14T05:38:25.383Z [INFO]  server-c7402b19-355b-bcb1-67a3-108dd5072c25: entering follower state: follower="Node at c7402b19-355b-bcb1-67a3-108dd5072c25 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.383Z [INFO]  creating new snapshot: path=/tmp/raft3869689989/snapshots/2-2-1752471505383.tmp
2025-07-14T05:38:25.383Z [INFO]  server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f: starting restore from snapshot: id=2-2-1752471505383 last-index=2 last-term=2 size-in-bytes=1
2025-07-14T05:38:25.383Z [INFO]  server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f: snapshot restore progress: id=2-2-1752471505383 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-07-14T05:38:25.383Z [INFO]  server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f: restored from snapshot: id=2-2-1752471505383 last-index=2 last-term=2 size-in-bytes=1
2025-07-14T05:38:25.383Z [INFO]  server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7402b19-355b-bcb1-67a3-108dd5072c25 Address:c7402b19-355b-bcb1-67a3-108dd5072c25} {Suffrage:Voter ID:server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f Address:7245a63e-bfc6-c0d8-499e-40b7d3581b8f} {Suffrage:Voter ID:server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 Address:f0d80198-f8f6-6fe6-07c7-c937e13005e1}]"
2025-07-14T05:38:25.383Z [INFO]  server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f: entering follower state: follower="Node at 7245a63e-bfc6-c0d8-499e-40b7d3581b8f [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.383Z [INFO]  creating new snapshot: path=/tmp/raft1252517658/snapshots/2-2-1752471505383.tmp
2025-07-14T05:38:25.384Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: starting restore from snapshot: id=2-2-1752471505383 last-index=2 last-term=2 size-in-bytes=1
2025-07-14T05:38:25.384Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: snapshot restore progress: id=2-2-1752471505383 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-07-14T05:38:25.384Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: restored from snapshot: id=2-2-1752471505383 last-index=2 last-term=2 size-in-bytes=1
2025-07-14T05:38:25.384Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7402b19-355b-bcb1-67a3-108dd5072c25 Address:c7402b19-355b-bcb1-67a3-108dd5072c25} {Suffrage:Voter ID:server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f Address:7245a63e-bfc6-c0d8-499e-40b7d3581b8f} {Suffrage:Voter ID:server-f0d80198-f8f6-6fe6-07c7-c937e13005e1 Address:f0d80198-f8f6-6fe6-07c7-c937e13005e1}]"
2025-07-14T05:38:25.384Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: entering follower state: follower="Node at f0d80198-f8f6-6fe6-07c7-c937e13005e1 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.458Z [WARN]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:25.458Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: entering candidate state: node="Node at f0d80198-f8f6-6fe6-07c7-c937e13005e1 [Candidate]" term=3
2025-07-14T05:38:25.458Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: election won: term=3 tally=2
2025-07-14T05:38:25.458Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: entering leader state: leader="Node at f0d80198-f8f6-6fe6-07c7-c937e13005e1 [Leader]"
2025-07-14T05:38:25.458Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: added peer, starting replication: peer=server-c7402b19-355b-bcb1-67a3-108dd5072c25
2025-07-14T05:38:25.458Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: added peer, starting replication: peer=server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f
2025-07-14T05:38:25.458Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: pipelining replication: peer="{Voter server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f 7245a63e-bfc6-c0d8-499e-40b7d3581b8f}"
2025-07-14T05:38:25.458Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: pipelining replication: peer="{Voter server-c7402b19-355b-bcb1-67a3-108dd5072c25 c7402b19-355b-bcb1-67a3-108dd5072c25}"
2025-07-14T05:38:25.699Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:25.805Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:25.699661825 +0000 UTC m=+5.114122203 (1 nodes), 22.864µs from start of poll, 475.889273ms from cluster start. Timeout at 2025-07-14 05:38:25.805495646 +0000 UTC m=+5.219956044, 105.833841ms after stability
2025-07-14T05:38:25.805Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: aborting pipeline replication: peer="{Voter server-7245a63e-bfc6-c0d8-499e-40b7d3581b8f 7245a63e-bfc6-c0d8-499e-40b7d3581b8f}"
2025-07-14T05:38:25.805Z [INFO]  server-f0d80198-f8f6-6fe6-07c7-c937e13005e1: aborting pipeline replication: peer="{Voter server-c7402b19-355b-bcb1-67a3-108dd5072c25 c7402b19-355b-bcb1-67a3-108dd5072c25}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-07-14T05:38:25.809Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 Address:5e84b327-b679-ea37-5bec-14c9e2c2fd00} {Suffrage:Voter ID:server-72a34f32-daaf-2298-2397-de69e90bd0cb Address:72a34f32-daaf-2298-2397-de69e90bd0cb} {Suffrage:Voter ID:server-cabf3eb2-5843-cfdc-5968-56a32ba10909 Address:cabf3eb2-5843-cfdc-5968-56a32ba10909}]"
2025-07-14T05:38:25.809Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: entering follower state: follower="Node at 5e84b327-b679-ea37-5bec-14c9e2c2fd00 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.809Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 Address:5e84b327-b679-ea37-5bec-14c9e2c2fd00} {Suffrage:Voter ID:server-72a34f32-daaf-2298-2397-de69e90bd0cb Address:72a34f32-daaf-2298-2397-de69e90bd0cb} {Suffrage:Voter ID:server-cabf3eb2-5843-cfdc-5968-56a32ba10909 Address:cabf3eb2-5843-cfdc-5968-56a32ba10909}]"
2025-07-14T05:38:25.809Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: entering follower state: follower="Node at 72a34f32-daaf-2298-2397-de69e90bd0cb [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.809Z [INFO]  server-cabf3eb2-5843-cfdc-5968-56a32ba10909: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 Address:5e84b327-b679-ea37-5bec-14c9e2c2fd00} {Suffrage:Voter ID:server-72a34f32-daaf-2298-2397-de69e90bd0cb Address:72a34f32-daaf-2298-2397-de69e90bd0cb} {Suffrage:Voter ID:server-cabf3eb2-5843-cfdc-5968-56a32ba10909 Address:cabf3eb2-5843-cfdc-5968-56a32ba10909}]"
2025-07-14T05:38:25.809Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:25.809Z [INFO]  server-cabf3eb2-5843-cfdc-5968-56a32ba10909: entering follower state: follower="Node at cabf3eb2-5843-cfdc-5968-56a32ba10909 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.876Z [WARN]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:25.876Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: entering candidate state: node="Node at 5e84b327-b679-ea37-5bec-14c9e2c2fd00 [Candidate]" term=2
2025-07-14T05:38:25.876Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: election won: term=2 tally=2
2025-07-14T05:38:25.876Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: entering leader state: leader="Node at 5e84b327-b679-ea37-5bec-14c9e2c2fd00 [Leader]"
2025-07-14T05:38:25.876Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: added peer, starting replication: peer=server-72a34f32-daaf-2298-2397-de69e90bd0cb
2025-07-14T05:38:25.876Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: added peer, starting replication: peer=server-cabf3eb2-5843-cfdc-5968-56a32ba10909
2025-07-14T05:38:25.876Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: pipelining replication: peer="{Voter server-cabf3eb2-5843-cfdc-5968-56a32ba10909 cabf3eb2-5843-cfdc-5968-56a32ba10909}"
2025-07-14T05:38:25.876Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: pipelining replication: peer="{Voter server-72a34f32-daaf-2298-2397-de69e90bd0cb 72a34f32-daaf-2298-2397-de69e90bd0cb}"
2025-07-14T05:38:25.982Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:25.87665161 +0000 UTC m=+5.291111988 (1 nodes), 67.328663ms from start of poll, 67.600791ms from cluster start. Timeout at 2025-07-14 05:38:25.982416998 +0000 UTC m=+5.396877366, 105.765378ms after stability
2025-07-14T05:38:25.982Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: aborting pipeline replication: peer="{Voter server-cabf3eb2-5843-cfdc-5968-56a32ba10909 cabf3eb2-5843-cfdc-5968-56a32ba10909}"
2025-07-14T05:38:25.982Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: aborting pipeline replication: peer="{Voter server-72a34f32-daaf-2298-2397-de69e90bd0cb 72a34f32-daaf-2298-2397-de69e90bd0cb}"
2025-07-14T05:38:25.983Z [INFO]  creating new snapshot: path=/tmp/raft3178296388/snapshots/2-6-1752471505983.tmp
2025-07-14T05:38:25.983Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: starting restore from snapshot: id=2-6-1752471505983 last-index=6 last-term=2 size-in-bytes=25
2025-07-14T05:38:25.983Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: snapshot restore progress: id=2-6-1752471505983 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-07-14T05:38:25.983Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: restored from snapshot: id=2-6-1752471505983 last-index=6 last-term=2 size-in-bytes=25
2025-07-14T05:38:25.983Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 Address:5e84b327-b679-ea37-5bec-14c9e2c2fd00} {Suffrage:Voter ID:server-72a34f32-daaf-2298-2397-de69e90bd0cb Address:72a34f32-daaf-2298-2397-de69e90bd0cb} {Suffrage:Voter ID:server-cabf3eb2-5843-cfdc-5968-56a32ba10909 Address:cabf3eb2-5843-cfdc-5968-56a32ba10909}]"
2025-07-14T05:38:25.983Z [INFO]  server-5e84b327-b679-ea37-5bec-14c9e2c2fd00: entering follower state: follower="Node at 5e84b327-b679-ea37-5bec-14c9e2c2fd00 [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.983Z [INFO]  creating new snapshot: path=/tmp/raft3464708275/snapshots/2-6-1752471505983.tmp
2025-07-14T05:38:25.984Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: starting restore from snapshot: id=2-6-1752471505983 last-index=6 last-term=2 size-in-bytes=25
2025-07-14T05:38:25.984Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: snapshot restore progress: id=2-6-1752471505983 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-07-14T05:38:25.984Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: restored from snapshot: id=2-6-1752471505983 last-index=6 last-term=2 size-in-bytes=25
2025-07-14T05:38:25.984Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 Address:5e84b327-b679-ea37-5bec-14c9e2c2fd00} {Suffrage:Voter ID:server-72a34f32-daaf-2298-2397-de69e90bd0cb Address:72a34f32-daaf-2298-2397-de69e90bd0cb} {Suffrage:Voter ID:server-cabf3eb2-5843-cfdc-5968-56a32ba10909 Address:cabf3eb2-5843-cfdc-5968-56a32ba10909}]"
2025-07-14T05:38:25.984Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: entering follower state: follower="Node at 72a34f32-daaf-2298-2397-de69e90bd0cb [Follower]" leader-address= leader-id=
2025-07-14T05:38:25.984Z [INFO]  creating new snapshot: path=/tmp/raft3977985156/snapshots/2-6-1752471505984.tmp
2025-07-14T05:38:25.985Z [INFO]  server-cabf3eb2-5843-cfdc-5968-56a32ba10909: starting restore from snapshot: id=2-6-1752471505984 last-index=6 last-term=2 size-in-bytes=25
2025-07-14T05:38:25.985Z [INFO]  server-cabf3eb2-5843-cfdc-5968-56a32ba10909: snapshot restore progress: id=2-6-1752471505984 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-07-14T05:38:25.985Z [INFO]  server-cabf3eb2-5843-cfdc-5968-56a32ba10909: restored from snapshot: id=2-6-1752471505984 last-index=6 last-term=2 size-in-bytes=25
2025-07-14T05:38:25.985Z [INFO]  server-cabf3eb2-5843-cfdc-5968-56a32ba10909: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 Address:5e84b327-b679-ea37-5bec-14c9e2c2fd00} {Suffrage:Voter ID:server-72a34f32-daaf-2298-2397-de69e90bd0cb Address:72a34f32-daaf-2298-2397-de69e90bd0cb} {Suffrage:Voter ID:server-cabf3eb2-5843-cfdc-5968-56a32ba10909 Address:cabf3eb2-5843-cfdc-5968-56a32ba10909}]"
2025-07-14T05:38:25.985Z [INFO]  server-cabf3eb2-5843-cfdc-5968-56a32ba10909: entering follower state: follower="Node at cabf3eb2-5843-cfdc-5968-56a32ba10909 [Follower]" leader-address= leader-id=
2025-07-14T05:38:26.040Z [WARN]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:26.040Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: entering candidate state: node="Node at 72a34f32-daaf-2298-2397-de69e90bd0cb [Candidate]" term=3
2025-07-14T05:38:26.040Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: election won: term=3 tally=2
2025-07-14T05:38:26.040Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: entering leader state: leader="Node at 72a34f32-daaf-2298-2397-de69e90bd0cb [Leader]"
2025-07-14T05:38:26.040Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: added peer, starting replication: peer=server-5e84b327-b679-ea37-5bec-14c9e2c2fd00
2025-07-14T05:38:26.040Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: added peer, starting replication: peer=server-cabf3eb2-5843-cfdc-5968-56a32ba10909
2025-07-14T05:38:26.040Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: pipelining replication: peer="{Voter server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 5e84b327-b679-ea37-5bec-14c9e2c2fd00}"
2025-07-14T05:38:26.040Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: pipelining replication: peer="{Voter server-cabf3eb2-5843-cfdc-5968-56a32ba10909 cabf3eb2-5843-cfdc-5968-56a32ba10909}"
2025-07-14T05:38:26.300Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:26.407Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:26.301009096 +0000 UTC m=+5.715469474 (1 nodes), 21.953µs from start of poll, 491.958277ms from cluster start. Timeout at 2025-07-14 05:38:26.407055015 +0000 UTC m=+5.821515383, 106.045909ms after stability
2025-07-14T05:38:26.407Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: aborting pipeline replication: peer="{Voter server-5e84b327-b679-ea37-5bec-14c9e2c2fd00 5e84b327-b679-ea37-5bec-14c9e2c2fd00}"
2025-07-14T05:38:26.407Z [INFO]  server-72a34f32-daaf-2298-2397-de69e90bd0cb: aborting pipeline replication: peer="{Voter server-cabf3eb2-5843-cfdc-5968-56a32ba10909 cabf3eb2-5843-cfdc-5968-56a32ba10909}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-07-14T05:38:26.408Z [INFO]  server-e22c746c-13aa-5696-6e78-b4dced84091e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22c746c-13aa-5696-6e78-b4dced84091e Address:e22c746c-13aa-5696-6e78-b4dced84091e} {Suffrage:Voter ID:server-898d05af-9a20-5558-0bc1-724b0c15e5a9 Address:898d05af-9a20-5558-0bc1-724b0c15e5a9} {Suffrage:Voter ID:server-cbec16bf-b81d-e19b-4549-7d03d1508249 Address:cbec16bf-b81d-e19b-4549-7d03d1508249}]"
2025-07-14T05:38:26.408Z [INFO]  server-e22c746c-13aa-5696-6e78-b4dced84091e: entering follower state: follower="Node at e22c746c-13aa-5696-6e78-b4dced84091e [Follower]" leader-address= leader-id=
2025-07-14T05:38:26.408Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22c746c-13aa-5696-6e78-b4dced84091e Address:e22c746c-13aa-5696-6e78-b4dced84091e} {Suffrage:Voter ID:server-898d05af-9a20-5558-0bc1-724b0c15e5a9 Address:898d05af-9a20-5558-0bc1-724b0c15e5a9} {Suffrage:Voter ID:server-cbec16bf-b81d-e19b-4549-7d03d1508249 Address:cbec16bf-b81d-e19b-4549-7d03d1508249}]"
2025-07-14T05:38:26.408Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: entering follower state: follower="Node at 898d05af-9a20-5558-0bc1-724b0c15e5a9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:26.409Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22c746c-13aa-5696-6e78-b4dced84091e Address:e22c746c-13aa-5696-6e78-b4dced84091e} {Suffrage:Voter ID:server-898d05af-9a20-5558-0bc1-724b0c15e5a9 Address:898d05af-9a20-5558-0bc1-724b0c15e5a9} {Suffrage:Voter ID:server-cbec16bf-b81d-e19b-4549-7d03d1508249 Address:cbec16bf-b81d-e19b-4549-7d03d1508249}]"
2025-07-14T05:38:26.409Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:26.409Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: entering follower state: follower="Node at cbec16bf-b81d-e19b-4549-7d03d1508249 [Follower]" leader-address= leader-id=
2025-07-14T05:38:26.476Z [WARN]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:26.476Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: entering candidate state: node="Node at cbec16bf-b81d-e19b-4549-7d03d1508249 [Candidate]" term=2
2025-07-14T05:38:26.476Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: election won: term=2 tally=2
2025-07-14T05:38:26.476Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: entering leader state: leader="Node at cbec16bf-b81d-e19b-4549-7d03d1508249 [Leader]"
2025-07-14T05:38:26.476Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: added peer, starting replication: peer=server-e22c746c-13aa-5696-6e78-b4dced84091e
2025-07-14T05:38:26.476Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: added peer, starting replication: peer=server-898d05af-9a20-5558-0bc1-724b0c15e5a9
2025-07-14T05:38:26.476Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: pipelining replication: peer="{Voter server-898d05af-9a20-5558-0bc1-724b0c15e5a9 898d05af-9a20-5558-0bc1-724b0c15e5a9}"
2025-07-14T05:38:26.476Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: pipelining replication: peer="{Voter server-e22c746c-13aa-5696-6e78-b4dced84091e e22c746c-13aa-5696-6e78-b4dced84091e}"
2025-07-14T05:38:26.582Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:26.476635896 +0000 UTC m=+5.891096274 (1 nodes), 67.564827ms from start of poll, 68.111267ms from cluster start. Timeout at 2025-07-14 05:38:26.582608746 +0000 UTC m=+5.997069154, 105.97288ms after stability
2025-07-14T05:38:26.585Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: aborting pipeline replication: peer="{Voter server-898d05af-9a20-5558-0bc1-724b0c15e5a9 898d05af-9a20-5558-0bc1-724b0c15e5a9}"
2025-07-14T05:38:26.585Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: aborting pipeline replication: peer="{Voter server-e22c746c-13aa-5696-6e78-b4dced84091e e22c746c-13aa-5696-6e78-b4dced84091e}"
2025-07-14T05:38:26.586Z [INFO]  creating new snapshot: path=/tmp/raft2746208691/snapshots/2-27-1752471506586.tmp
2025-07-14T05:38:26.586Z [INFO]  server-e22c746c-13aa-5696-6e78-b4dced84091e: starting restore from snapshot: id=2-27-1752471506586 last-index=27 last-term=2 size-in-bytes=168
2025-07-14T05:38:26.586Z [INFO]  server-e22c746c-13aa-5696-6e78-b4dced84091e: snapshot restore progress: id=2-27-1752471506586 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-07-14T05:38:26.586Z [INFO]  server-e22c746c-13aa-5696-6e78-b4dced84091e: restored from snapshot: id=2-27-1752471506586 last-index=27 last-term=2 size-in-bytes=168
2025-07-14T05:38:26.586Z [INFO]  server-e22c746c-13aa-5696-6e78-b4dced84091e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22c746c-13aa-5696-6e78-b4dced84091e Address:e22c746c-13aa-5696-6e78-b4dced84091e} {Suffrage:Voter ID:server-898d05af-9a20-5558-0bc1-724b0c15e5a9 Address:898d05af-9a20-5558-0bc1-724b0c15e5a9} {Suffrage:Voter ID:server-cbec16bf-b81d-e19b-4549-7d03d1508249 Address:cbec16bf-b81d-e19b-4549-7d03d1508249}]"
2025-07-14T05:38:26.586Z [INFO]  server-e22c746c-13aa-5696-6e78-b4dced84091e: entering follower state: follower="Node at e22c746c-13aa-5696-6e78-b4dced84091e [Follower]" leader-address= leader-id=
2025-07-14T05:38:26.587Z [INFO]  creating new snapshot: path=/tmp/raft3228886346/snapshots/2-27-1752471506586.tmp
2025-07-14T05:38:26.587Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: starting restore from snapshot: id=2-27-1752471506586 last-index=27 last-term=2 size-in-bytes=168
2025-07-14T05:38:26.587Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: snapshot restore progress: id=2-27-1752471506586 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-07-14T05:38:26.587Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: restored from snapshot: id=2-27-1752471506586 last-index=27 last-term=2 size-in-bytes=168
2025-07-14T05:38:26.587Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22c746c-13aa-5696-6e78-b4dced84091e Address:e22c746c-13aa-5696-6e78-b4dced84091e} {Suffrage:Voter ID:server-898d05af-9a20-5558-0bc1-724b0c15e5a9 Address:898d05af-9a20-5558-0bc1-724b0c15e5a9} {Suffrage:Voter ID:server-cbec16bf-b81d-e19b-4549-7d03d1508249 Address:cbec16bf-b81d-e19b-4549-7d03d1508249}]"
2025-07-14T05:38:26.587Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: entering follower state: follower="Node at 898d05af-9a20-5558-0bc1-724b0c15e5a9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:26.587Z [INFO]  creating new snapshot: path=/tmp/raft105739706/snapshots/2-27-1752471506587.tmp
2025-07-14T05:38:26.588Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: starting restore from snapshot: id=2-27-1752471506587 last-index=27 last-term=2 size-in-bytes=168
2025-07-14T05:38:26.588Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: snapshot restore progress: id=2-27-1752471506587 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-07-14T05:38:26.588Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: restored from snapshot: id=2-27-1752471506587 last-index=27 last-term=2 size-in-bytes=168
2025-07-14T05:38:26.588Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22c746c-13aa-5696-6e78-b4dced84091e Address:e22c746c-13aa-5696-6e78-b4dced84091e} {Suffrage:Voter ID:server-898d05af-9a20-5558-0bc1-724b0c15e5a9 Address:898d05af-9a20-5558-0bc1-724b0c15e5a9} {Suffrage:Voter ID:server-cbec16bf-b81d-e19b-4549-7d03d1508249 Address:cbec16bf-b81d-e19b-4549-7d03d1508249}]"
2025-07-14T05:38:26.588Z [INFO]  server-cbec16bf-b81d-e19b-4549-7d03d1508249: entering follower state: follower="Node at cbec16bf-b81d-e19b-4549-7d03d1508249 [Follower]" leader-address= leader-id=
2025-07-14T05:38:26.655Z [WARN]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:26.655Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: entering candidate state: node="Node at 898d05af-9a20-5558-0bc1-724b0c15e5a9 [Candidate]" term=3
2025-07-14T05:38:26.655Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: election won: term=3 tally=2
2025-07-14T05:38:26.655Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: entering leader state: leader="Node at 898d05af-9a20-5558-0bc1-724b0c15e5a9 [Leader]"
2025-07-14T05:38:26.655Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: added peer, starting replication: peer=server-e22c746c-13aa-5696-6e78-b4dced84091e
2025-07-14T05:38:26.655Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: added peer, starting replication: peer=server-cbec16bf-b81d-e19b-4549-7d03d1508249
2025-07-14T05:38:26.655Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: pipelining replication: peer="{Voter server-cbec16bf-b81d-e19b-4549-7d03d1508249 cbec16bf-b81d-e19b-4549-7d03d1508249}"
2025-07-14T05:38:26.655Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: pipelining replication: peer="{Voter server-e22c746c-13aa-5696-6e78-b4dced84091e e22c746c-13aa-5696-6e78-b4dced84091e}"
2025-07-14T05:38:26.904Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:27.009Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:26.904315125 +0000 UTC m=+6.318775523 (1 nodes), 25.638µs from start of poll, 495.790516ms from cluster start. Timeout at 2025-07-14 05:38:27.009790368 +0000 UTC m=+6.424250746, 105.475223ms after stability
2025-07-14T05:38:27.009Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: aborting pipeline replication: peer="{Voter server-cbec16bf-b81d-e19b-4549-7d03d1508249 cbec16bf-b81d-e19b-4549-7d03d1508249}"
2025-07-14T05:38:27.009Z [INFO]  server-898d05af-9a20-5558-0bc1-724b0c15e5a9: aborting pipeline replication: peer="{Voter server-e22c746c-13aa-5696-6e78-b4dced84091e e22c746c-13aa-5696-6e78-b4dced84091e}"
--- PASS: TestRaft_RecoverCluster (1.79s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.59s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.60s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-07-14T05:38:27.011Z [INFO]  server-34e87e70-4227-174c-8523-3be0681ad4ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34e87e70-4227-174c-8523-3be0681ad4ac Address:34e87e70-4227-174c-8523-3be0681ad4ac}]"
2025-07-14T05:38:27.011Z [INFO]  server-34e87e70-4227-174c-8523-3be0681ad4ac: entering follower state: follower="Node at 34e87e70-4227-174c-8523-3be0681ad4ac [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.082Z [WARN]  server-34e87e70-4227-174c-8523-3be0681ad4ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:27.082Z [INFO]  server-34e87e70-4227-174c-8523-3be0681ad4ac: entering candidate state: node="Node at 34e87e70-4227-174c-8523-3be0681ad4ac [Candidate]" term=2
2025-07-14T05:38:27.082Z [INFO]  server-34e87e70-4227-174c-8523-3be0681ad4ac: election won: term=2 tally=1
2025-07-14T05:38:27.082Z [INFO]  server-34e87e70-4227-174c-8523-3be0681ad4ac: entering leader state: leader="Node at 34e87e70-4227-174c-8523-3be0681ad4ac [Leader]"
--- PASS: TestRaft_SingleNode (0.07s)
=== RUN   TestRaft_TripleNode
2025-07-14T05:38:27.083Z [INFO]  server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14 Address:443f433b-e4d3-ddc6-ebd3-de6dfd10ab14} {Suffrage:Voter ID:server-b62c4f43-19b2-fc01-826b-d5cde89044e2 Address:b62c4f43-19b2-fc01-826b-d5cde89044e2} {Suffrage:Voter ID:server-a5967ecd-de60-1516-728b-2419bbbe1781 Address:a5967ecd-de60-1516-728b-2419bbbe1781}]"
2025-07-14T05:38:27.083Z [INFO]  server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14: entering follower state: follower="Node at 443f433b-e4d3-ddc6-ebd3-de6dfd10ab14 [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.083Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14 Address:443f433b-e4d3-ddc6-ebd3-de6dfd10ab14} {Suffrage:Voter ID:server-b62c4f43-19b2-fc01-826b-d5cde89044e2 Address:b62c4f43-19b2-fc01-826b-d5cde89044e2} {Suffrage:Voter ID:server-a5967ecd-de60-1516-728b-2419bbbe1781 Address:a5967ecd-de60-1516-728b-2419bbbe1781}]"
2025-07-14T05:38:27.084Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: entering follower state: follower="Node at b62c4f43-19b2-fc01-826b-d5cde89044e2 [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.084Z [INFO]  server-a5967ecd-de60-1516-728b-2419bbbe1781: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14 Address:443f433b-e4d3-ddc6-ebd3-de6dfd10ab14} {Suffrage:Voter ID:server-b62c4f43-19b2-fc01-826b-d5cde89044e2 Address:b62c4f43-19b2-fc01-826b-d5cde89044e2} {Suffrage:Voter ID:server-a5967ecd-de60-1516-728b-2419bbbe1781 Address:a5967ecd-de60-1516-728b-2419bbbe1781}]"
2025-07-14T05:38:27.084Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:27.084Z [INFO]  server-a5967ecd-de60-1516-728b-2419bbbe1781: entering follower state: follower="Node at a5967ecd-de60-1516-728b-2419bbbe1781 [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.139Z [WARN]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:27.139Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: entering candidate state: node="Node at b62c4f43-19b2-fc01-826b-d5cde89044e2 [Candidate]" term=2
2025-07-14T05:38:27.139Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: election won: term=2 tally=2
2025-07-14T05:38:27.139Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: entering leader state: leader="Node at b62c4f43-19b2-fc01-826b-d5cde89044e2 [Leader]"
2025-07-14T05:38:27.139Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: added peer, starting replication: peer=server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14
2025-07-14T05:38:27.139Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: added peer, starting replication: peer=server-a5967ecd-de60-1516-728b-2419bbbe1781
2025-07-14T05:38:27.139Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: pipelining replication: peer="{Voter server-a5967ecd-de60-1516-728b-2419bbbe1781 a5967ecd-de60-1516-728b-2419bbbe1781}"
2025-07-14T05:38:27.139Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: pipelining replication: peer="{Voter server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14 443f433b-e4d3-ddc6-ebd3-de6dfd10ab14}"
2025-07-14T05:38:27.245Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:27.139643317 +0000 UTC m=+6.554103715 (2 nodes), 55.50674ms from start of poll, 55.858607ms from cluster start. Timeout at 2025-07-14 05:38:27.245394855 +0000 UTC m=+6.659855233, 105.751518ms after stability
2025-07-14T05:38:27.245Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:27.351Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:27.245494815 +0000 UTC m=+6.659955193 (1 nodes), 1.272µs from start of poll, 161.710085ms from cluster start. Timeout at 2025-07-14 05:38:27.351522897 +0000 UTC m=+6.765983285, 106.028092ms after stability
2025-07-14T05:38:27.362Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: aborting pipeline replication: peer="{Voter server-a5967ecd-de60-1516-728b-2419bbbe1781 a5967ecd-de60-1516-728b-2419bbbe1781}"
2025-07-14T05:38:27.362Z [INFO]  server-b62c4f43-19b2-fc01-826b-d5cde89044e2: aborting pipeline replication: peer="{Voter server-443f433b-e4d3-ddc6-ebd3-de6dfd10ab14 443f433b-e4d3-ddc6-ebd3-de6dfd10ab14}"
--- PASS: TestRaft_TripleNode (0.28s)
=== RUN   TestRaft_LeaderFail
2025-07-14T05:38:27.363Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25cab9e6-b178-6ad3-55b4-f113d00de13e Address:25cab9e6-b178-6ad3-55b4-f113d00de13e} {Suffrage:Voter ID:server-2a903903-a8c7-46e6-1ad7-9f62a1869983 Address:2a903903-a8c7-46e6-1ad7-9f62a1869983} {Suffrage:Voter ID:server-72df8f8b-2fb0-a150-b591-655a851dc0cf Address:72df8f8b-2fb0-a150-b591-655a851dc0cf}]"
2025-07-14T05:38:27.364Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: entering follower state: follower="Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.364Z [INFO]  server-2a903903-a8c7-46e6-1ad7-9f62a1869983: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25cab9e6-b178-6ad3-55b4-f113d00de13e Address:25cab9e6-b178-6ad3-55b4-f113d00de13e} {Suffrage:Voter ID:server-2a903903-a8c7-46e6-1ad7-9f62a1869983 Address:2a903903-a8c7-46e6-1ad7-9f62a1869983} {Suffrage:Voter ID:server-72df8f8b-2fb0-a150-b591-655a851dc0cf Address:72df8f8b-2fb0-a150-b591-655a851dc0cf}]"
2025-07-14T05:38:27.364Z [INFO]  server-2a903903-a8c7-46e6-1ad7-9f62a1869983: entering follower state: follower="Node at 2a903903-a8c7-46e6-1ad7-9f62a1869983 [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.364Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25cab9e6-b178-6ad3-55b4-f113d00de13e Address:25cab9e6-b178-6ad3-55b4-f113d00de13e} {Suffrage:Voter ID:server-2a903903-a8c7-46e6-1ad7-9f62a1869983 Address:2a903903-a8c7-46e6-1ad7-9f62a1869983} {Suffrage:Voter ID:server-72df8f8b-2fb0-a150-b591-655a851dc0cf Address:72df8f8b-2fb0-a150-b591-655a851dc0cf}]"
2025-07-14T05:38:27.364Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:27.364Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: entering follower state: follower="Node at 72df8f8b-2fb0-a150-b591-655a851dc0cf [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.422Z [WARN]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:27.422Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: entering candidate state: node="Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Candidate]" term=2
2025-07-14T05:38:27.422Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: election won: term=2 tally=2
2025-07-14T05:38:27.422Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: entering leader state: leader="Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Leader]"
2025-07-14T05:38:27.422Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: added peer, starting replication: peer=server-2a903903-a8c7-46e6-1ad7-9f62a1869983
2025-07-14T05:38:27.422Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: added peer, starting replication: peer=server-72df8f8b-2fb0-a150-b591-655a851dc0cf
2025-07-14T05:38:27.422Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: pipelining replication: peer="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}"
2025-07-14T05:38:27.423Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: pipelining replication: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}"
2025-07-14T05:38:27.528Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:27.422913703 +0000 UTC m=+6.837374091 (2 nodes), 58.11315ms from start of poll, 59.330946ms from cluster start. Timeout at 2025-07-14 05:38:27.528116357 +0000 UTC m=+6.942576775, 105.202684ms after stability
2025-07-14T05:38:27.528Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:27.633Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:27.528196988 +0000 UTC m=+6.942657366 (1 nodes), 1.272µs from start of poll, 164.614221ms from cluster start. Timeout at 2025-07-14 05:38:27.633921114 +0000 UTC m=+7.048381482, 105.724116ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Leader]
2025-07-14T05:38:27.648Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=2a903903-a8c7-46e6-1ad7-9f62a1869983 backoff time=10ms error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983"
2025-07-14T05:38:27.648Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:27.650Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=72df8f8b-2fb0-a150-b591-655a851dc0cf backoff time=10ms error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.651Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to pipeline appendEntries: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}" error="append pipeline closed"
2025-07-14T05:38:27.651Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to pipeline appendEntries: peer="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}" error="append pipeline closed"
2025-07-14T05:38:27.651Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: aborting pipeline replication: peer="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}"
2025-07-14T05:38:27.651Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: aborting pipeline replication: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}"
2025-07-14T05:38:27.656Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to appendEntries to: peer="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}" error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.659Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to appendEntries to: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}" error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983"
2025-07-14T05:38:27.665Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=2a903903-a8c7-46e6-1ad7-9f62a1869983 backoff time=10ms error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983"
2025-07-14T05:38:27.669Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=72df8f8b-2fb0-a150-b591-655a851dc0cf backoff time=10ms error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.674Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to appendEntries to: peer="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}" error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.680Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to appendEntries to: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}" error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983"
2025-07-14T05:38:27.683Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=2a903903-a8c7-46e6-1ad7-9f62a1869983 backoff time=10ms error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983"
2025-07-14T05:38:27.688Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=72df8f8b-2fb0-a150-b591-655a851dc0cf backoff time=10ms error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.691Z [WARN]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to contact: server-id=server-72df8f8b-2fb0-a150-b591-655a851dc0cf time=50.371497ms
2025-07-14T05:38:27.691Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to appendEntries to: peer="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}" error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.698Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to appendEntries to: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}" error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983"
2025-07-14T05:38:27.699Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=2a903903-a8c7-46e6-1ad7-9f62a1869983 backoff time=20ms error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983"
2025-07-14T05:38:27.702Z [WARN]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to contact: server-id=server-2a903903-a8c7-46e6-1ad7-9f62a1869983 time=58.95331ms
2025-07-14T05:38:27.702Z [WARN]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to contact: server-id=server-72df8f8b-2fb0-a150-b591-655a851dc0cf time=61.117356ms
2025-07-14T05:38:27.702Z [WARN]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to contact quorum of nodes, stepping down
2025-07-14T05:38:27.702Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: entering follower state: follower="Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.707Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to heartbeat to: peer=72df8f8b-2fb0-a150-b591-655a851dc0cf backoff time=20ms error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.719Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to appendEntries to: peer="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}" error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf"
2025-07-14T05:38:27.733Z [WARN]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: heartbeat timeout reached, starting election: last-leader-addr=25cab9e6-b178-6ad3-55b4-f113d00de13e last-leader-id=server-25cab9e6-b178-6ad3-55b4-f113d00de13e
2025-07-14T05:38:27.733Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: entering candidate state: node="Node at 72df8f8b-2fb0-a150-b591-655a851dc0cf [Candidate]" term=3
2025-07-14T05:38:27.733Z [WARN]  server-2a903903-a8c7-46e6-1ad7-9f62a1869983: rejecting vote request since we have a leader: from=72df8f8b-2fb0-a150-b591-655a851dc0cf leader=25cab9e6-b178-6ad3-55b4-f113d00de13e leader-id=server-25cab9e6-b178-6ad3-55b4-f113d00de13e
2025-07-14T05:38:27.733Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to make requestVote RPC: target="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}" error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e" term=3
2025-07-14T05:38:27.754Z [WARN]  server-2a903903-a8c7-46e6-1ad7-9f62a1869983: heartbeat timeout reached, starting election: last-leader-addr=25cab9e6-b178-6ad3-55b4-f113d00de13e last-leader-id=server-25cab9e6-b178-6ad3-55b4-f113d00de13e
2025-07-14T05:38:27.754Z [INFO]  server-2a903903-a8c7-46e6-1ad7-9f62a1869983: entering candidate state: node="Node at 2a903903-a8c7-46e6-1ad7-9f62a1869983 [Candidate]" term=3
2025-07-14T05:38:27.754Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: duplicate requestVote for same term: term=3
2025-07-14T05:38:27.754Z [ERROR] server-2a903903-a8c7-46e6-1ad7-9f62a1869983: failed to make requestVote RPC: target="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}" error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e" term=3
2025-07-14T05:38:27.773Z [WARN]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:27.773Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: entering candidate state: node="Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Candidate]" term=3
2025-07-14T05:38:27.773Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to make requestVote RPC: target="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}" error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf" term=3
2025-07-14T05:38:27.773Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to make requestVote RPC: target="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}" error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983" term=3
2025-07-14T05:38:27.803Z [WARN]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: Election timeout reached, restarting election
2025-07-14T05:38:27.803Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: entering candidate state: node="Node at 72df8f8b-2fb0-a150-b591-655a851dc0cf [Candidate]" term=4
2025-07-14T05:38:27.803Z [INFO]  server-2a903903-a8c7-46e6-1ad7-9f62a1869983: entering follower state: follower="Node at 2a903903-a8c7-46e6-1ad7-9f62a1869983 [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.803Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to make requestVote RPC: target="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}" error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e" term=4
2025-07-14T05:38:27.803Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: election won: term=4 tally=2
2025-07-14T05:38:27.803Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: entering leader state: leader="Node at 72df8f8b-2fb0-a150-b591-655a851dc0cf [Leader]"
2025-07-14T05:38:27.803Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: added peer, starting replication: peer=server-25cab9e6-b178-6ad3-55b4-f113d00de13e
2025-07-14T05:38:27.803Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: added peer, starting replication: peer=server-2a903903-a8c7-46e6-1ad7-9f62a1869983
2025-07-14T05:38:27.803Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: pipelining replication: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}"
2025-07-14T05:38:27.803Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to appendEntries to: peer="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}" error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.809Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to heartbeat to: peer=25cab9e6-b178-6ad3-55b4-f113d00de13e backoff time=10ms error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.823Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to appendEntries to: peer="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}" error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.827Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to heartbeat to: peer=25cab9e6-b178-6ad3-55b4-f113d00de13e backoff time=10ms error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.836Z [WARN]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: Election timeout reached, restarting election
2025-07-14T05:38:27.836Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: entering candidate state: node="Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Candidate]" term=4
2025-07-14T05:38:27.837Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to make requestVote RPC: target="{Voter server-72df8f8b-2fb0-a150-b591-655a851dc0cf 72df8f8b-2fb0-a150-b591-655a851dc0cf}" error="failed to connect to peer: 72df8f8b-2fb0-a150-b591-655a851dc0cf" term=4
2025-07-14T05:38:27.837Z [ERROR] server-25cab9e6-b178-6ad3-55b4-f113d00de13e: failed to make requestVote RPC: target="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}" error="failed to connect to peer: 2a903903-a8c7-46e6-1ad7-9f62a1869983" term=4
2025-07-14T05:38:27.840Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to appendEntries to: peer="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}" error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.847Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to heartbeat to: peer=25cab9e6-b178-6ad3-55b4-f113d00de13e backoff time=10ms error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.854Z [WARN]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to contact: server-id=server-25cab9e6-b178-6ad3-55b4-f113d00de13e time=50.562714ms
2025-07-14T05:38:27.867Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to heartbeat to: peer=25cab9e6-b178-6ad3-55b4-f113d00de13e backoff time=20ms error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.869Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to appendEntries to: peer="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}" error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.895Z [ERROR] server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to heartbeat to: peer=25cab9e6-b178-6ad3-55b4-f113d00de13e backoff time=25ms error="failed to connect to peer: 25cab9e6-b178-6ad3-55b4-f113d00de13e"
2025-07-14T05:38:27.904Z [WARN]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: failed to contact: server-id=server-25cab9e6-b178-6ad3-55b4-f113d00de13e time=100.507902ms
2025-07-14T05:38:27.909Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:27.803828966 +0000 UTC m=+7.218289334 (1 nodes), 155.67962ms from start of poll, 440.246189ms from cluster start. Timeout at 2025-07-14 05:38:27.909736347 +0000 UTC m=+7.324196745, 105.907411ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Candidate]
2025-07-14T05:38:27.920Z [INFO]  server-25cab9e6-b178-6ad3-55b4-f113d00de13e: entering follower state: follower="Node at 25cab9e6-b178-6ad3-55b4-f113d00de13e [Follower]" leader-address=72df8f8b-2fb0-a150-b591-655a851dc0cf leader-id=server-72df8f8b-2fb0-a150-b591-655a851dc0cf
2025-07-14T05:38:27.920Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: pipelining replication: peer="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}"
2025-07-14T05:38:27.925Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: aborting pipeline replication: peer="{Voter server-2a903903-a8c7-46e6-1ad7-9f62a1869983 2a903903-a8c7-46e6-1ad7-9f62a1869983}"
2025-07-14T05:38:27.925Z [INFO]  server-72df8f8b-2fb0-a150-b591-655a851dc0cf: aborting pipeline replication: peer="{Voter server-25cab9e6-b178-6ad3-55b4-f113d00de13e 25cab9e6-b178-6ad3-55b4-f113d00de13e}"
--- PASS: TestRaft_LeaderFail (0.56s)
=== RUN   TestRaft_BehindFollower
2025-07-14T05:38:27.926Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9 Address:2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9} {Suffrage:Voter ID:server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 Address:fa2dbaae-e1ed-dcbc-5482-61812dc94ce8} {Suffrage:Voter ID:server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55 Address:ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55}]"
2025-07-14T05:38:27.926Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: entering follower state: follower="Node at 2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.926Z [INFO]  server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9 Address:2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9} {Suffrage:Voter ID:server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 Address:fa2dbaae-e1ed-dcbc-5482-61812dc94ce8} {Suffrage:Voter ID:server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55 Address:ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55}]"
2025-07-14T05:38:27.926Z [INFO]  server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8: entering follower state: follower="Node at fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 [Follower]" leader-address= leader-id=
2025-07-14T05:38:27.926Z [INFO]  server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9 Address:2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9} {Suffrage:Voter ID:server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 Address:fa2dbaae-e1ed-dcbc-5482-61812dc94ce8} {Suffrage:Voter ID:server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55 Address:ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55}]"
2025-07-14T05:38:27.926Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:27.927Z [INFO]  server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55: entering follower state: follower="Node at ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55 [Follower]" leader-address= leader-id=
2025-07-14T05:38:28.001Z [WARN]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:28.001Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: entering candidate state: node="Node at 2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9 [Candidate]" term=2
2025-07-14T05:38:28.001Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: election won: term=2 tally=2
2025-07-14T05:38:28.001Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: entering leader state: leader="Node at 2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9 [Leader]"
2025-07-14T05:38:28.001Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: added peer, starting replication: peer=server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8
2025-07-14T05:38:28.001Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: added peer, starting replication: peer=server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55
2025-07-14T05:38:28.001Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: pipelining replication: peer="{Voter server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55 ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55}"
2025-07-14T05:38:28.001Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: pipelining replication: peer="{Voter server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 fa2dbaae-e1ed-dcbc-5482-61812dc94ce8}"
2025-07-14T05:38:28.107Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:28.001569937 +0000 UTC m=+7.416030315 (1 nodes), 74.569644ms from start of poll, 74.83328ms from cluster start. Timeout at 2025-07-14 05:38:28.107486542 +0000 UTC m=+7.521946920, 105.916605ms after stability
2025-07-14T05:38:28.107Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:28.212Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:28.107555485 +0000 UTC m=+7.522015854 (2 nodes), 1.272µs from start of poll, 180.818819ms from cluster start. Timeout at 2025-07-14 05:38:28.21291885 +0000 UTC m=+7.627379239, 105.363385ms after stability
2025-07-14T05:38:28.213Z [ERROR] server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: failed to pipeline appendEntries: peer="{Voter server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 fa2dbaae-e1ed-dcbc-5482-61812dc94ce8}" error="append pipeline closed"
2025-07-14T05:38:28.213Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: aborting pipeline replication: peer="{Voter server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 fa2dbaae-e1ed-dcbc-5482-61812dc94ce8}"
2025-07-14T05:38:28.213Z [ERROR] server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: failed to appendEntries to: peer="{Voter server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 fa2dbaae-e1ed-dcbc-5482-61812dc94ce8}" error="failed to connect to peer: fa2dbaae-e1ed-dcbc-5482-61812dc94ce8"
2025-07-14T05:38:28.215Z [ERROR] server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: failed to heartbeat to: peer=fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 backoff time=10ms error="failed to connect to peer: fa2dbaae-e1ed-dcbc-5482-61812dc94ce8"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-07-14T05:38:28.223Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: pipelining replication: peer="{Voter server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 fa2dbaae-e1ed-dcbc-5482-61812dc94ce8}"
2025-07-14T05:38:28.239Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:28.344Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:28.239622726 +0000 UTC m=+7.654083094 (1 nodes), 941ns from start of poll, 312.886059ms from cluster start. Timeout at 2025-07-14 05:38:28.344629206 +0000 UTC m=+7.759089604, 105.00651ms after stability
2025-07-14T05:38:28.344Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: aborting pipeline replication: peer="{Voter server-ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55 ff2c1bfc-e17a-8a40-f4d7-fdd65fa40d55}"
2025-07-14T05:38:28.344Z [INFO]  server-2ed6b0cf-009c-2bd1-8c50-ae6a2fa681a9: aborting pipeline replication: peer="{Voter server-fa2dbaae-e1ed-dcbc-5482-61812dc94ce8 fa2dbaae-e1ed-dcbc-5482-61812dc94ce8}"
--- PASS: TestRaft_BehindFollower (0.42s)
=== RUN   TestRaft_ApplyNonLeader
2025-07-14T05:38:28.345Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a Address:e1e8e511-d0bd-fc8d-d734-8804a9c66c8a} {Suffrage:Voter ID:server-fa517ab9-4f28-a8bb-8212-30f3833ee338 Address:fa517ab9-4f28-a8bb-8212-30f3833ee338} {Suffrage:Voter ID:server-87867ac9-bd84-4d57-9e4e-9ee49568178f Address:87867ac9-bd84-4d57-9e4e-9ee49568178f}]"
2025-07-14T05:38:28.345Z [INFO]  server-fa517ab9-4f28-a8bb-8212-30f3833ee338: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a Address:e1e8e511-d0bd-fc8d-d734-8804a9c66c8a} {Suffrage:Voter ID:server-fa517ab9-4f28-a8bb-8212-30f3833ee338 Address:fa517ab9-4f28-a8bb-8212-30f3833ee338} {Suffrage:Voter ID:server-87867ac9-bd84-4d57-9e4e-9ee49568178f Address:87867ac9-bd84-4d57-9e4e-9ee49568178f}]"
2025-07-14T05:38:28.345Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: entering follower state: follower="Node at e1e8e511-d0bd-fc8d-d734-8804a9c66c8a [Follower]" leader-address= leader-id=
2025-07-14T05:38:28.345Z [INFO]  server-fa517ab9-4f28-a8bb-8212-30f3833ee338: entering follower state: follower="Node at fa517ab9-4f28-a8bb-8212-30f3833ee338 [Follower]" leader-address= leader-id=
2025-07-14T05:38:28.346Z [INFO]  server-87867ac9-bd84-4d57-9e4e-9ee49568178f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a Address:e1e8e511-d0bd-fc8d-d734-8804a9c66c8a} {Suffrage:Voter ID:server-fa517ab9-4f28-a8bb-8212-30f3833ee338 Address:fa517ab9-4f28-a8bb-8212-30f3833ee338} {Suffrage:Voter ID:server-87867ac9-bd84-4d57-9e4e-9ee49568178f Address:87867ac9-bd84-4d57-9e4e-9ee49568178f}]"
2025-07-14T05:38:28.346Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:28.346Z [INFO]  server-87867ac9-bd84-4d57-9e4e-9ee49568178f: entering follower state: follower="Node at 87867ac9-bd84-4d57-9e4e-9ee49568178f [Follower]" leader-address= leader-id=
2025-07-14T05:38:28.411Z [WARN]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:28.411Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: entering candidate state: node="Node at e1e8e511-d0bd-fc8d-d734-8804a9c66c8a [Candidate]" term=2
2025-07-14T05:38:28.411Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: election won: term=2 tally=2
2025-07-14T05:38:28.411Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: entering leader state: leader="Node at e1e8e511-d0bd-fc8d-d734-8804a9c66c8a [Leader]"
2025-07-14T05:38:28.411Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: added peer, starting replication: peer=server-fa517ab9-4f28-a8bb-8212-30f3833ee338
2025-07-14T05:38:28.411Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: added peer, starting replication: peer=server-87867ac9-bd84-4d57-9e4e-9ee49568178f
2025-07-14T05:38:28.411Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: pipelining replication: peer="{Voter server-87867ac9-bd84-4d57-9e4e-9ee49568178f 87867ac9-bd84-4d57-9e4e-9ee49568178f}"
2025-07-14T05:38:28.411Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: pipelining replication: peer="{Voter server-fa517ab9-4f28-a8bb-8212-30f3833ee338 fa517ab9-4f28-a8bb-8212-30f3833ee338}"
2025-07-14T05:38:28.516Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:28.41137843 +0000 UTC m=+7.825838808 (1 nodes), 65.275363ms from start of poll, 65.625108ms from cluster start. Timeout at 2025-07-14 05:38:28.516909316 +0000 UTC m=+7.931369714, 105.530906ms after stability
2025-07-14T05:38:28.516Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:28.622Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:28.516971079 +0000 UTC m=+7.931431447 (2 nodes), 861ns from start of poll, 171.217747ms from cluster start. Timeout at 2025-07-14 05:38:28.622467954 +0000 UTC m=+8.036928352, 105.496905ms after stability
2025-07-14T05:38:28.622Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: aborting pipeline replication: peer="{Voter server-87867ac9-bd84-4d57-9e4e-9ee49568178f 87867ac9-bd84-4d57-9e4e-9ee49568178f}"
2025-07-14T05:38:28.622Z [INFO]  server-e1e8e511-d0bd-fc8d-d734-8804a9c66c8a: aborting pipeline replication: peer="{Voter server-fa517ab9-4f28-a8bb-8212-30f3833ee338 fa517ab9-4f28-a8bb-8212-30f3833ee338}"
--- PASS: TestRaft_ApplyNonLeader (0.28s)
=== RUN   TestRaft_ApplyConcurrent
2025-07-14T05:38:28.623Z [INFO]  server-a3005f80-3dad-b64d-2a28-6d40c999df62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3005f80-3dad-b64d-2a28-6d40c999df62 Address:a3005f80-3dad-b64d-2a28-6d40c999df62} {Suffrage:Voter ID:server-bd8a7910-c40d-b102-db0b-5909bf16324b Address:bd8a7910-c40d-b102-db0b-5909bf16324b} {Suffrage:Voter ID:server-66cb059f-9093-fbba-2700-ed89408ef628 Address:66cb059f-9093-fbba-2700-ed89408ef628}]"
2025-07-14T05:38:28.623Z [INFO]  server-a3005f80-3dad-b64d-2a28-6d40c999df62: entering follower state: follower="Node at a3005f80-3dad-b64d-2a28-6d40c999df62 [Follower]" leader-address= leader-id=
2025-07-14T05:38:28.623Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3005f80-3dad-b64d-2a28-6d40c999df62 Address:a3005f80-3dad-b64d-2a28-6d40c999df62} {Suffrage:Voter ID:server-bd8a7910-c40d-b102-db0b-5909bf16324b Address:bd8a7910-c40d-b102-db0b-5909bf16324b} {Suffrage:Voter ID:server-66cb059f-9093-fbba-2700-ed89408ef628 Address:66cb059f-9093-fbba-2700-ed89408ef628}]"
2025-07-14T05:38:28.624Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: entering follower state: follower="Node at bd8a7910-c40d-b102-db0b-5909bf16324b [Follower]" leader-address= leader-id=
2025-07-14T05:38:28.624Z [INFO]  server-66cb059f-9093-fbba-2700-ed89408ef628: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3005f80-3dad-b64d-2a28-6d40c999df62 Address:a3005f80-3dad-b64d-2a28-6d40c999df62} {Suffrage:Voter ID:server-bd8a7910-c40d-b102-db0b-5909bf16324b Address:bd8a7910-c40d-b102-db0b-5909bf16324b} {Suffrage:Voter ID:server-66cb059f-9093-fbba-2700-ed89408ef628 Address:66cb059f-9093-fbba-2700-ed89408ef628}]"
2025-07-14T05:38:28.624Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:28.624Z [INFO]  server-66cb059f-9093-fbba-2700-ed89408ef628: entering follower state: follower="Node at 66cb059f-9093-fbba-2700-ed89408ef628 [Follower]" leader-address= leader-id=
2025-07-14T05:38:28.745Z [WARN]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:28.745Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: entering candidate state: node="Node at bd8a7910-c40d-b102-db0b-5909bf16324b [Candidate]" term=2
2025-07-14T05:38:28.745Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: election won: term=2 tally=2
2025-07-14T05:38:28.745Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: entering leader state: leader="Node at bd8a7910-c40d-b102-db0b-5909bf16324b [Leader]"
2025-07-14T05:38:28.745Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: added peer, starting replication: peer=server-a3005f80-3dad-b64d-2a28-6d40c999df62
2025-07-14T05:38:28.745Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: added peer, starting replication: peer=server-66cb059f-9093-fbba-2700-ed89408ef628
2025-07-14T05:38:28.745Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: pipelining replication: peer="{Voter server-66cb059f-9093-fbba-2700-ed89408ef628 66cb059f-9093-fbba-2700-ed89408ef628}"
2025-07-14T05:38:28.745Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: pipelining replication: peer="{Voter server-a3005f80-3dad-b64d-2a28-6d40c999df62 a3005f80-3dad-b64d-2a28-6d40c999df62}"
2025-07-14T05:38:28.951Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:28.745641975 +0000 UTC m=+8.160102353 (1 nodes), 121.370946ms from start of poll, 121.8469ms from cluster start. Timeout at 2025-07-14 05:38:28.951453052 +0000 UTC m=+8.365913430, 205.811077ms after stability
2025-07-14T05:38:28.964Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: aborting pipeline replication: peer="{Voter server-66cb059f-9093-fbba-2700-ed89408ef628 66cb059f-9093-fbba-2700-ed89408ef628}"
2025-07-14T05:38:28.964Z [INFO]  server-bd8a7910-c40d-b102-db0b-5909bf16324b: aborting pipeline replication: peer="{Voter server-a3005f80-3dad-b64d-2a28-6d40c999df62 a3005f80-3dad-b64d-2a28-6d40c999df62}"
--- PASS: TestRaft_ApplyConcurrent (0.34s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-07-14T05:38:28.965Z [INFO]  server-74f70ea6-9f15-2db0-b068-259cf50da1be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74f70ea6-9f15-2db0-b068-259cf50da1be Address:74f70ea6-9f15-2db0-b068-259cf50da1be}]"
2025-07-14T05:38:28.965Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:28.965Z [INFO]  server-74f70ea6-9f15-2db0-b068-259cf50da1be: entering follower state: follower="Node at 74f70ea6-9f15-2db0-b068-259cf50da1be [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.067Z [WARN]  server-74f70ea6-9f15-2db0-b068-259cf50da1be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:29.067Z [INFO]  server-74f70ea6-9f15-2db0-b068-259cf50da1be: entering candidate state: node="Node at 74f70ea6-9f15-2db0-b068-259cf50da1be [Candidate]" term=2
2025-07-14T05:38:29.067Z [INFO]  server-74f70ea6-9f15-2db0-b068-259cf50da1be: election won: term=2 tally=1
2025-07-14T05:38:29.067Z [INFO]  server-74f70ea6-9f15-2db0-b068-259cf50da1be: entering leader state: leader="Node at 74f70ea6-9f15-2db0-b068-259cf50da1be [Leader]"
2025-07-14T05:38:29.269Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:29.067527764 +0000 UTC m=+8.481988162 (1 nodes), 102.05996ms from start of poll, 102.195193ms from cluster start. Timeout at 2025-07-14 05:38:29.269003718 +0000 UTC m=+8.683464086, 201.475924ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.30s)
=== RUN   TestRaft_JoinNode
2025-07-14T05:38:29.270Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c48e3543-16c8-c277-7eef-40aeb7b286ed Address:c48e3543-16c8-c277-7eef-40aeb7b286ed} {Suffrage:Voter ID:server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079 Address:cfdda1f1-9fe3-3f0f-17f9-69beeba7b079}]"
2025-07-14T05:38:29.270Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: entering follower state: follower="Node at c48e3543-16c8-c277-7eef-40aeb7b286ed [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.270Z [INFO]  server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c48e3543-16c8-c277-7eef-40aeb7b286ed Address:c48e3543-16c8-c277-7eef-40aeb7b286ed} {Suffrage:Voter ID:server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079 Address:cfdda1f1-9fe3-3f0f-17f9-69beeba7b079}]"
2025-07-14T05:38:29.270Z [INFO]  server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079: entering follower state: follower="Node at cfdda1f1-9fe3-3f0f-17f9-69beeba7b079 [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.270Z [INFO]  server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0: initial configuration: index=0 servers=[]
2025-07-14T05:38:29.270Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:29.270Z [INFO]  server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0: entering follower state: follower="Node at 1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0 [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.341Z [WARN]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:29.341Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: entering candidate state: node="Node at c48e3543-16c8-c277-7eef-40aeb7b286ed [Candidate]" term=2
2025-07-14T05:38:29.341Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: election won: term=2 tally=2
2025-07-14T05:38:29.341Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: entering leader state: leader="Node at c48e3543-16c8-c277-7eef-40aeb7b286ed [Leader]"
2025-07-14T05:38:29.341Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: added peer, starting replication: peer=server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079
2025-07-14T05:38:29.341Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: pipelining replication: peer="{Voter server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079 cfdda1f1-9fe3-3f0f-17f9-69beeba7b079}"
2025-07-14T05:38:29.350Z [WARN]  server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0: no known peers, aborting election
2025-07-14T05:38:29.447Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:29.341279201 +0000 UTC m=+8.755739599 (1 nodes), 70.685128ms from start of poll, 71.323265ms from cluster start. Timeout at 2025-07-14 05:38:29.44699207 +0000 UTC m=+8.861452448, 105.712849ms after stability
2025-07-14T05:38:29.447Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: updating configuration: command=AddVoter server-id=server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0 server-addr=1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0 servers="[{Suffrage:Voter ID:server-c48e3543-16c8-c277-7eef-40aeb7b286ed Address:c48e3543-16c8-c277-7eef-40aeb7b286ed} {Suffrage:Voter ID:server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079 Address:cfdda1f1-9fe3-3f0f-17f9-69beeba7b079} {Suffrage:Voter ID:server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0 Address:1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0}]"
2025-07-14T05:38:29.447Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: added peer, starting replication: peer=server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0
2025-07-14T05:38:29.447Z [WARN]  server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-07-14T05:38:29.447Z [WARN]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: appendEntries rejected, sending older logs: peer="{Voter server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0 1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0}" next=1
2025-07-14T05:38:29.447Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: pipelining replication: peer="{Voter server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0 1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0}"
2025-07-14T05:38:29.447Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:29.553Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:29.447462045 +0000 UTC m=+8.861922423 (1 nodes), 861ns from start of poll, 177.506089ms from cluster start. Timeout at 2025-07-14 05:38:29.553284028 +0000 UTC m=+8.967744406, 105.821983ms after stability
2025-07-14T05:38:29.553Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: aborting pipeline replication: peer="{Voter server-1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0 1ad5d6fb-5107-d760-2b1a-8e7e3d664aa0}"
2025-07-14T05:38:29.553Z [INFO]  server-c48e3543-16c8-c277-7eef-40aeb7b286ed: aborting pipeline replication: peer="{Voter server-cfdda1f1-9fe3-3f0f-17f9-69beeba7b079 cfdda1f1-9fe3-3f0f-17f9-69beeba7b079}"
--- PASS: TestRaft_JoinNode (0.28s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-07-14T05:38:29.554Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9 Address:8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9}]"
2025-07-14T05:38:29.554Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: entering follower state: follower="Node at 8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.554Z [INFO]  server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b: initial configuration: index=0 servers=[]
2025-07-14T05:38:29.554Z [INFO]  server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b: entering follower state: follower="Node at 21b6fc8a-fb6f-9a8d-e004-6e91f573db4b [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.554Z [INFO]  server-27614c84-afeb-ceb5-2a1d-d685957cfab9: initial configuration: index=0 servers=[]
2025-07-14T05:38:29.554Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:29.554Z [INFO]  server-27614c84-afeb-ceb5-2a1d-d685957cfab9: entering follower state: follower="Node at 27614c84-afeb-ceb5-2a1d-d685957cfab9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.635Z [WARN]  server-27614c84-afeb-ceb5-2a1d-d685957cfab9: no known peers, aborting election
2025-07-14T05:38:29.651Z [WARN]  server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b: no known peers, aborting election
2025-07-14T05:38:29.651Z [WARN]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:29.651Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: entering candidate state: node="Node at 8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9 [Candidate]" term=2
2025-07-14T05:38:29.651Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: election won: term=2 tally=1
2025-07-14T05:38:29.651Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: entering leader state: leader="Node at 8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9 [Leader]"
2025-07-14T05:38:29.757Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:29.652007473 +0000 UTC m=+9.066467851 (1 nodes), 97.372228ms from start of poll, 97.856154ms from cluster start. Timeout at 2025-07-14 05:38:29.757519031 +0000 UTC m=+9.171979429, 105.511578ms after stability
2025-07-14T05:38:29.757Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: updating configuration: command=AddVoter server-id=server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b server-addr=21b6fc8a-fb6f-9a8d-e004-6e91f573db4b servers="[{Suffrage:Voter ID:server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9 Address:8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9} {Suffrage:Voter ID:server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b Address:21b6fc8a-fb6f-9a8d-e004-6e91f573db4b}]"
2025-07-14T05:38:29.757Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: added peer, starting replication: peer=server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b
2025-07-14T05:38:29.757Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:29.757Z [WARN]  server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-07-14T05:38:29.757Z [WARN]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: appendEntries rejected, sending older logs: peer="{Voter server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b 21b6fc8a-fb6f-9a8d-e004-6e91f573db4b}" next=1
2025-07-14T05:38:29.758Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: pipelining replication: peer="{Voter server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b 21b6fc8a-fb6f-9a8d-e004-6e91f573db4b}"
2025-07-14T05:38:29.863Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:29.757727894 +0000 UTC m=+9.172188262 (1 nodes), 851ns from start of poll, 203.576565ms from cluster start. Timeout at 2025-07-14 05:38:29.863460483 +0000 UTC m=+9.277920881, 105.732619ms after stability
2025-07-14T05:38:29.863Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: updating configuration: command=AddVoter server-id=server-27614c84-afeb-ceb5-2a1d-d685957cfab9 server-addr=27614c84-afeb-ceb5-2a1d-d685957cfab9 servers="[{Suffrage:Voter ID:server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9 Address:8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9} {Suffrage:Voter ID:server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b Address:21b6fc8a-fb6f-9a8d-e004-6e91f573db4b} {Suffrage:Voter ID:server-27614c84-afeb-ceb5-2a1d-d685957cfab9 Address:27614c84-afeb-ceb5-2a1d-d685957cfab9}]"
2025-07-14T05:38:29.863Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: added peer, starting replication: peer=server-27614c84-afeb-ceb5-2a1d-d685957cfab9
2025-07-14T05:38:29.863Z [WARN]  server-27614c84-afeb-ceb5-2a1d-d685957cfab9: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-07-14T05:38:29.863Z [WARN]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: appendEntries rejected, sending older logs: peer="{Voter server-27614c84-afeb-ceb5-2a1d-d685957cfab9 27614c84-afeb-ceb5-2a1d-d685957cfab9}" next=1
2025-07-14T05:38:29.863Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: pipelining replication: peer="{Voter server-27614c84-afeb-ceb5-2a1d-d685957cfab9 27614c84-afeb-ceb5-2a1d-d685957cfab9}"
2025-07-14T05:38:29.863Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:29.969Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:29.863797749 +0000 UTC m=+9.278258117 (1 nodes), 551ns from start of poll, 309.64642ms from cluster start. Timeout at 2025-07-14 05:38:29.969219812 +0000 UTC m=+9.383680190, 105.422073ms after stability
2025-07-14T05:38:29.969Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: aborting pipeline replication: peer="{Voter server-27614c84-afeb-ceb5-2a1d-d685957cfab9 27614c84-afeb-ceb5-2a1d-d685957cfab9}"
2025-07-14T05:38:29.969Z [INFO]  server-8a8e4f77-e1dc-9f0b-4ce5-64a5e05ca9a9: aborting pipeline replication: peer="{Voter server-21b6fc8a-fb6f-9a8d-e004-6e91f573db4b 21b6fc8a-fb6f-9a8d-e004-6e91f573db4b}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.42s)
=== RUN   TestRaft_RemoveFollower
2025-07-14T05:38:29.970Z [INFO]  server-79003a0f-5195-e6af-9d4a-22f1e1f8756b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79003a0f-5195-e6af-9d4a-22f1e1f8756b Address:79003a0f-5195-e6af-9d4a-22f1e1f8756b} {Suffrage:Voter ID:server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82 Address:55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82} {Suffrage:Voter ID:server-0733abf3-4a56-ea1f-3b29-bbe307a783a1 Address:0733abf3-4a56-ea1f-3b29-bbe307a783a1}]"
2025-07-14T05:38:29.970Z [INFO]  server-79003a0f-5195-e6af-9d4a-22f1e1f8756b: entering follower state: follower="Node at 79003a0f-5195-e6af-9d4a-22f1e1f8756b [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.970Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79003a0f-5195-e6af-9d4a-22f1e1f8756b Address:79003a0f-5195-e6af-9d4a-22f1e1f8756b} {Suffrage:Voter ID:server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82 Address:55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82} {Suffrage:Voter ID:server-0733abf3-4a56-ea1f-3b29-bbe307a783a1 Address:0733abf3-4a56-ea1f-3b29-bbe307a783a1}]"
2025-07-14T05:38:29.970Z [INFO]  server-0733abf3-4a56-ea1f-3b29-bbe307a783a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79003a0f-5195-e6af-9d4a-22f1e1f8756b Address:79003a0f-5195-e6af-9d4a-22f1e1f8756b} {Suffrage:Voter ID:server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82 Address:55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82} {Suffrage:Voter ID:server-0733abf3-4a56-ea1f-3b29-bbe307a783a1 Address:0733abf3-4a56-ea1f-3b29-bbe307a783a1}]"
2025-07-14T05:38:29.970Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:29.970Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: entering follower state: follower="Node at 55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82 [Follower]" leader-address= leader-id=
2025-07-14T05:38:29.970Z [INFO]  server-0733abf3-4a56-ea1f-3b29-bbe307a783a1: entering follower state: follower="Node at 0733abf3-4a56-ea1f-3b29-bbe307a783a1 [Follower]" leader-address= leader-id=
2025-07-14T05:38:30.056Z [WARN]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:30.056Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: entering candidate state: node="Node at 55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82 [Candidate]" term=2
2025-07-14T05:38:30.056Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: election won: term=2 tally=2
2025-07-14T05:38:30.056Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: entering leader state: leader="Node at 55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82 [Leader]"
2025-07-14T05:38:30.056Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: added peer, starting replication: peer=server-79003a0f-5195-e6af-9d4a-22f1e1f8756b
2025-07-14T05:38:30.056Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: added peer, starting replication: peer=server-0733abf3-4a56-ea1f-3b29-bbe307a783a1
2025-07-14T05:38:30.056Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: pipelining replication: peer="{Voter server-79003a0f-5195-e6af-9d4a-22f1e1f8756b 79003a0f-5195-e6af-9d4a-22f1e1f8756b}"
2025-07-14T05:38:30.056Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: pipelining replication: peer="{Voter server-0733abf3-4a56-ea1f-3b29-bbe307a783a1 0733abf3-4a56-ea1f-3b29-bbe307a783a1}"
2025-07-14T05:38:30.163Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:30.056931529 +0000 UTC m=+9.471391897 (1 nodes), 86.279209ms from start of poll, 86.550205ms from cluster start. Timeout at 2025-07-14 05:38:30.162989176 +0000 UTC m=+9.577449574, 106.057677ms after stability
2025-07-14T05:38:30.168Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:30.273Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:30.168353783 +0000 UTC m=+9.582814151 (2 nodes), 2.134µs from start of poll, 197.972459ms from cluster start. Timeout at 2025-07-14 05:38:30.273561084 +0000 UTC m=+9.688021452, 105.207301ms after stability
2025-07-14T05:38:30.273Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: updating configuration: command=RemoveServer server-id=server-79003a0f-5195-e6af-9d4a-22f1e1f8756b server-addr= servers="[{Suffrage:Voter ID:server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82 Address:55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82} {Suffrage:Voter ID:server-0733abf3-4a56-ea1f-3b29-bbe307a783a1 Address:0733abf3-4a56-ea1f-3b29-bbe307a783a1}]"
2025-07-14T05:38:30.273Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: removed peer, stopping replication: peer=server-79003a0f-5195-e6af-9d4a-22f1e1f8756b last-index=3
2025-07-14T05:38:30.273Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: aborting pipeline replication: peer="{Voter server-79003a0f-5195-e6af-9d4a-22f1e1f8756b 79003a0f-5195-e6af-9d4a-22f1e1f8756b}"
2025-07-14T05:38:30.375Z [WARN]  server-79003a0f-5195-e6af-9d4a-22f1e1f8756b: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-07-14T05:38:30.378Z [INFO]  server-55b7f3b2-2c60-ce1c-f1f4-2e7e2df56f82: aborting pipeline replication: peer="{Voter server-0733abf3-4a56-ea1f-3b29-bbe307a783a1 0733abf3-4a56-ea1f-3b29-bbe307a783a1}"
--- PASS: TestRaft_RemoveFollower (0.41s)
=== RUN   TestRaft_RemoveLeader
2025-07-14T05:38:30.380Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5251ca93-1732-eb94-7f73-d15f05683584 Address:5251ca93-1732-eb94-7f73-d15f05683584} {Suffrage:Voter ID:server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 Address:d4eb91d6-68dc-cb08-fae1-7526f750fee9} {Suffrage:Voter ID:server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c Address:7ed89ee8-b60d-5b41-ad06-ff48c4223c3c}]"
2025-07-14T05:38:30.380Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: entering follower state: follower="Node at 5251ca93-1732-eb94-7f73-d15f05683584 [Follower]" leader-address= leader-id=
2025-07-14T05:38:30.380Z [INFO]  server-d4eb91d6-68dc-cb08-fae1-7526f750fee9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5251ca93-1732-eb94-7f73-d15f05683584 Address:5251ca93-1732-eb94-7f73-d15f05683584} {Suffrage:Voter ID:server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 Address:d4eb91d6-68dc-cb08-fae1-7526f750fee9} {Suffrage:Voter ID:server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c Address:7ed89ee8-b60d-5b41-ad06-ff48c4223c3c}]"
2025-07-14T05:38:30.380Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5251ca93-1732-eb94-7f73-d15f05683584 Address:5251ca93-1732-eb94-7f73-d15f05683584} {Suffrage:Voter ID:server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 Address:d4eb91d6-68dc-cb08-fae1-7526f750fee9} {Suffrage:Voter ID:server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c Address:7ed89ee8-b60d-5b41-ad06-ff48c4223c3c}]"
2025-07-14T05:38:30.380Z [INFO]  server-d4eb91d6-68dc-cb08-fae1-7526f750fee9: entering follower state: follower="Node at d4eb91d6-68dc-cb08-fae1-7526f750fee9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:30.380Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:30.380Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: entering follower state: follower="Node at 7ed89ee8-b60d-5b41-ad06-ff48c4223c3c [Follower]" leader-address= leader-id=
2025-07-14T05:38:30.451Z [WARN]  server-5251ca93-1732-eb94-7f73-d15f05683584: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:30.451Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: entering candidate state: node="Node at 5251ca93-1732-eb94-7f73-d15f05683584 [Candidate]" term=2
2025-07-14T05:38:30.451Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: election won: term=2 tally=2
2025-07-14T05:38:30.451Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: entering leader state: leader="Node at 5251ca93-1732-eb94-7f73-d15f05683584 [Leader]"
2025-07-14T05:38:30.451Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: added peer, starting replication: peer=server-d4eb91d6-68dc-cb08-fae1-7526f750fee9
2025-07-14T05:38:30.451Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: added peer, starting replication: peer=server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c
2025-07-14T05:38:30.451Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: pipelining replication: peer="{Voter server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c 7ed89ee8-b60d-5b41-ad06-ff48c4223c3c}"
2025-07-14T05:38:30.451Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: pipelining replication: peer="{Voter server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 d4eb91d6-68dc-cb08-fae1-7526f750fee9}"
2025-07-14T05:38:30.556Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:30.451590938 +0000 UTC m=+9.866051336 (1 nodes), 71.244236ms from start of poll, 71.562293ms from cluster start. Timeout at 2025-07-14 05:38:30.55685864 +0000 UTC m=+9.971319028, 105.267692ms after stability
2025-07-14T05:38:30.562Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:30.668Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:30.562299371 +0000 UTC m=+9.976759749 (2 nodes), 811ns from start of poll, 182.270706ms from cluster start. Timeout at 2025-07-14 05:38:30.668330528 +0000 UTC m=+10.082790926, 106.031177ms after stability
2025-07-14T05:38:30.668Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: updating configuration: command=RemoveServer server-id=server-5251ca93-1732-eb94-7f73-d15f05683584 server-addr= servers="[{Suffrage:Voter ID:server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 Address:d4eb91d6-68dc-cb08-fae1-7526f750fee9} {Suffrage:Voter ID:server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c Address:7ed89ee8-b60d-5b41-ad06-ff48c4223c3c}]"
2025-07-14T05:38:30.668Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: removed ourself, shutting down
2025-07-14T05:38:30.668Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: aborting pipeline replication: peer="{Voter server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c 7ed89ee8-b60d-5b41-ad06-ff48c4223c3c}"
2025-07-14T05:38:30.668Z [INFO]  server-5251ca93-1732-eb94-7f73-d15f05683584: aborting pipeline replication: peer="{Voter server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 d4eb91d6-68dc-cb08-fae1-7526f750fee9}"
2025-07-14T05:38:30.721Z [WARN]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: heartbeat timeout reached, starting election: last-leader-addr=5251ca93-1732-eb94-7f73-d15f05683584 last-leader-id=server-5251ca93-1732-eb94-7f73-d15f05683584
2025-07-14T05:38:30.721Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: entering candidate state: node="Node at 7ed89ee8-b60d-5b41-ad06-ff48c4223c3c [Candidate]" term=3
2025-07-14T05:38:30.721Z [WARN]  server-d4eb91d6-68dc-cb08-fae1-7526f750fee9: rejecting vote request since we have a leader: from=7ed89ee8-b60d-5b41-ad06-ff48c4223c3c leader=5251ca93-1732-eb94-7f73-d15f05683584 leader-id=server-5251ca93-1732-eb94-7f73-d15f05683584
2025-07-14T05:38:30.793Z [WARN]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: Election timeout reached, restarting election
2025-07-14T05:38:30.793Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: entering candidate state: node="Node at 7ed89ee8-b60d-5b41-ad06-ff48c4223c3c [Candidate]" term=4
2025-07-14T05:38:30.793Z [WARN]  server-d4eb91d6-68dc-cb08-fae1-7526f750fee9: rejecting vote request since we have a leader: from=7ed89ee8-b60d-5b41-ad06-ff48c4223c3c leader=5251ca93-1732-eb94-7f73-d15f05683584 leader-id=server-5251ca93-1732-eb94-7f73-d15f05683584
2025-07-14T05:38:30.800Z [WARN]  server-d4eb91d6-68dc-cb08-fae1-7526f750fee9: heartbeat timeout reached, starting election: last-leader-addr=5251ca93-1732-eb94-7f73-d15f05683584 last-leader-id=server-5251ca93-1732-eb94-7f73-d15f05683584
2025-07-14T05:38:30.800Z [INFO]  server-d4eb91d6-68dc-cb08-fae1-7526f750fee9: entering candidate state: node="Node at d4eb91d6-68dc-cb08-fae1-7526f750fee9 [Candidate]" term=3
2025-07-14T05:38:30.800Z [INFO]  server-d4eb91d6-68dc-cb08-fae1-7526f750fee9: entering follower state: follower="Node at d4eb91d6-68dc-cb08-fae1-7526f750fee9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:30.874Z [WARN]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: Election timeout reached, restarting election
2025-07-14T05:38:30.874Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: entering candidate state: node="Node at 7ed89ee8-b60d-5b41-ad06-ff48c4223c3c [Candidate]" term=5
2025-07-14T05:38:30.874Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: election won: term=5 tally=2
2025-07-14T05:38:30.874Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: entering leader state: leader="Node at 7ed89ee8-b60d-5b41-ad06-ff48c4223c3c [Leader]"
2025-07-14T05:38:30.874Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: added peer, starting replication: peer=server-d4eb91d6-68dc-cb08-fae1-7526f750fee9
2025-07-14T05:38:30.874Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: pipelining replication: peer="{Voter server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 d4eb91d6-68dc-cb08-fae1-7526f750fee9}"
2025-07-14T05:38:30.879Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:30.985Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:30.879679872 +0000 UTC m=+10.294140270 (1 nodes), 32.449µs from start of poll, 499.651227ms from cluster start. Timeout at 2025-07-14 05:38:30.985219261 +0000 UTC m=+10.399679629, 105.539359ms after stability
2025-07-14T05:38:30.985Z [INFO]  server-7ed89ee8-b60d-5b41-ad06-ff48c4223c3c: aborting pipeline replication: peer="{Voter server-d4eb91d6-68dc-cb08-fae1-7526f750fee9 d4eb91d6-68dc-cb08-fae1-7526f750fee9}"
--- PASS: TestRaft_RemoveLeader (0.67s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-07-14T05:38:31.054Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf678ab3-16a6-509a-a101-e882aedfccec Address:bf678ab3-16a6-509a-a101-e882aedfccec} {Suffrage:Voter ID:server-0af64df1-65d7-ecc9-546b-45686ececb95 Address:0af64df1-65d7-ecc9-546b-45686ececb95} {Suffrage:Voter ID:server-ef382926-1405-6436-a664-69c670da539f Address:ef382926-1405-6436-a664-69c670da539f}]"
2025-07-14T05:38:31.054Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: entering follower state: follower="Node at bf678ab3-16a6-509a-a101-e882aedfccec [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.054Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf678ab3-16a6-509a-a101-e882aedfccec Address:bf678ab3-16a6-509a-a101-e882aedfccec} {Suffrage:Voter ID:server-0af64df1-65d7-ecc9-546b-45686ececb95 Address:0af64df1-65d7-ecc9-546b-45686ececb95} {Suffrage:Voter ID:server-ef382926-1405-6436-a664-69c670da539f Address:ef382926-1405-6436-a664-69c670da539f}]"
2025-07-14T05:38:31.054Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: entering follower state: follower="Node at 0af64df1-65d7-ecc9-546b-45686ececb95 [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.054Z [INFO]  server-ef382926-1405-6436-a664-69c670da539f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf678ab3-16a6-509a-a101-e882aedfccec Address:bf678ab3-16a6-509a-a101-e882aedfccec} {Suffrage:Voter ID:server-0af64df1-65d7-ecc9-546b-45686ececb95 Address:0af64df1-65d7-ecc9-546b-45686ececb95} {Suffrage:Voter ID:server-ef382926-1405-6436-a664-69c670da539f Address:ef382926-1405-6436-a664-69c670da539f}]"
2025-07-14T05:38:31.054Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:31.054Z [INFO]  server-ef382926-1405-6436-a664-69c670da539f: entering follower state: follower="Node at ef382926-1405-6436-a664-69c670da539f [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.119Z [WARN]  server-0af64df1-65d7-ecc9-546b-45686ececb95: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:31.119Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: entering candidate state: node="Node at 0af64df1-65d7-ecc9-546b-45686ececb95 [Candidate]" term=2
2025-07-14T05:38:31.119Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: election won: term=2 tally=2
2025-07-14T05:38:31.119Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: entering leader state: leader="Node at 0af64df1-65d7-ecc9-546b-45686ececb95 [Leader]"
2025-07-14T05:38:31.119Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: added peer, starting replication: peer=server-bf678ab3-16a6-509a-a101-e882aedfccec
2025-07-14T05:38:31.119Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: added peer, starting replication: peer=server-ef382926-1405-6436-a664-69c670da539f
2025-07-14T05:38:31.119Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: pipelining replication: peer="{Voter server-ef382926-1405-6436-a664-69c670da539f ef382926-1405-6436-a664-69c670da539f}"
2025-07-14T05:38:31.119Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: pipelining replication: peer="{Voter server-bf678ab3-16a6-509a-a101-e882aedfccec bf678ab3-16a6-509a-a101-e882aedfccec}"
2025-07-14T05:38:31.225Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:31.11986526 +0000 UTC m=+10.534325658 (2 nodes), 65.365308ms from start of poll, 65.825709ms from cluster start. Timeout at 2025-07-14 05:38:31.225640733 +0000 UTC m=+10.640101121, 105.775463ms after stability
2025-07-14T05:38:31.225Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:31.330Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:31.225699251 +0000 UTC m=+10.640159629 (1 nodes), 832ns from start of poll, 171.65968ms from cluster start. Timeout at 2025-07-14 05:38:31.330710538 +0000 UTC m=+10.745170906, 105.011277ms after stability
2025-07-14T05:38:31.333Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: updating configuration: command=RemoveServer server-id=server-0af64df1-65d7-ecc9-546b-45686ececb95 server-addr= servers="[{Suffrage:Voter ID:server-bf678ab3-16a6-509a-a101-e882aedfccec Address:bf678ab3-16a6-509a-a101-e882aedfccec} {Suffrage:Voter ID:server-ef382926-1405-6436-a664-69c670da539f Address:ef382926-1405-6436-a664-69c670da539f}]"
2025-07-14T05:38:31.334Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: removed ourself, transitioning to follower
2025-07-14T05:38:31.334Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: entering follower state: follower="Node at 0af64df1-65d7-ecc9-546b-45686ececb95 [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.334Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: aborting pipeline replication: peer="{Voter server-ef382926-1405-6436-a664-69c670da539f ef382926-1405-6436-a664-69c670da539f}"
2025-07-14T05:38:31.334Z [INFO]  server-0af64df1-65d7-ecc9-546b-45686ececb95: aborting pipeline replication: peer="{Voter server-bf678ab3-16a6-509a-a101-e882aedfccec bf678ab3-16a6-509a-a101-e882aedfccec}"
2025-07-14T05:38:31.388Z [WARN]  server-bf678ab3-16a6-509a-a101-e882aedfccec: heartbeat timeout reached, starting election: last-leader-addr=0af64df1-65d7-ecc9-546b-45686ececb95 last-leader-id=server-0af64df1-65d7-ecc9-546b-45686ececb95
2025-07-14T05:38:31.388Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: entering candidate state: node="Node at bf678ab3-16a6-509a-a101-e882aedfccec [Candidate]" term=3
2025-07-14T05:38:31.388Z [WARN]  server-ef382926-1405-6436-a664-69c670da539f: rejecting vote request since we have a leader: from=bf678ab3-16a6-509a-a101-e882aedfccec leader=0af64df1-65d7-ecc9-546b-45686ececb95 leader-id=server-0af64df1-65d7-ecc9-546b-45686ececb95
2025-07-14T05:38:31.424Z [WARN]  server-ef382926-1405-6436-a664-69c670da539f: heartbeat timeout reached, starting election: last-leader-addr=0af64df1-65d7-ecc9-546b-45686ececb95 last-leader-id=server-0af64df1-65d7-ecc9-546b-45686ececb95
2025-07-14T05:38:31.424Z [INFO]  server-ef382926-1405-6436-a664-69c670da539f: entering candidate state: node="Node at ef382926-1405-6436-a664-69c670da539f [Candidate]" term=3
2025-07-14T05:38:31.424Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: duplicate requestVote for same term: term=3
2025-07-14T05:38:31.430Z [WARN]  server-0af64df1-65d7-ecc9-546b-45686ececb95: not part of stable configuration, aborting election
2025-07-14T05:38:31.439Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:31.458Z [WARN]  server-bf678ab3-16a6-509a-a101-e882aedfccec: Election timeout reached, restarting election
2025-07-14T05:38:31.458Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: entering candidate state: node="Node at bf678ab3-16a6-509a-a101-e882aedfccec [Candidate]" term=4
2025-07-14T05:38:31.458Z [INFO]  server-ef382926-1405-6436-a664-69c670da539f: entering follower state: follower="Node at ef382926-1405-6436-a664-69c670da539f [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.458Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: election won: term=4 tally=2
2025-07-14T05:38:31.458Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: entering leader state: leader="Node at bf678ab3-16a6-509a-a101-e882aedfccec [Leader]"
2025-07-14T05:38:31.458Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: added peer, starting replication: peer=server-ef382926-1405-6436-a664-69c670da539f
2025-07-14T05:38:31.458Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: pipelining replication: peer="{Voter server-ef382926-1405-6436-a664-69c670da539f ef382926-1405-6436-a664-69c670da539f}"
2025-07-14T05:38:31.564Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:31.458509596 +0000 UTC m=+10.872969964 (1 nodes), 18.729444ms from start of poll, 404.470015ms from cluster start. Timeout at 2025-07-14 05:38:31.564144989 +0000 UTC m=+10.978605367, 105.635403ms after stability
2025-07-14T05:38:31.669Z [INFO]  server-bf678ab3-16a6-509a-a101-e882aedfccec: aborting pipeline replication: peer="{Voter server-ef382926-1405-6436-a664-69c670da539f ef382926-1405-6436-a664-69c670da539f}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-07-14T05:38:31.671Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e10d5ca-ec80-9910-803a-42c583d40e64 Address:8e10d5ca-ec80-9910-803a-42c583d40e64} {Suffrage:Voter ID:server-fbe7299e-6013-e005-eed8-a584714724b1 Address:fbe7299e-6013-e005-eed8-a584714724b1} {Suffrage:Voter ID:server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 Address:4a03c6e6-c95f-4ff0-e568-ce21d70492e4} {Suffrage:Voter ID:server-94770c92-0b7f-678e-85c6-b3fbbea56831 Address:94770c92-0b7f-678e-85c6-b3fbbea56831}]"
2025-07-14T05:38:31.671Z [INFO]  server-fbe7299e-6013-e005-eed8-a584714724b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e10d5ca-ec80-9910-803a-42c583d40e64 Address:8e10d5ca-ec80-9910-803a-42c583d40e64} {Suffrage:Voter ID:server-fbe7299e-6013-e005-eed8-a584714724b1 Address:fbe7299e-6013-e005-eed8-a584714724b1} {Suffrage:Voter ID:server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 Address:4a03c6e6-c95f-4ff0-e568-ce21d70492e4} {Suffrage:Voter ID:server-94770c92-0b7f-678e-85c6-b3fbbea56831 Address:94770c92-0b7f-678e-85c6-b3fbbea56831}]"
2025-07-14T05:38:31.671Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: entering follower state: follower="Node at 8e10d5ca-ec80-9910-803a-42c583d40e64 [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.671Z [INFO]  server-fbe7299e-6013-e005-eed8-a584714724b1: entering follower state: follower="Node at fbe7299e-6013-e005-eed8-a584714724b1 [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.671Z [INFO]  server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e10d5ca-ec80-9910-803a-42c583d40e64 Address:8e10d5ca-ec80-9910-803a-42c583d40e64} {Suffrage:Voter ID:server-fbe7299e-6013-e005-eed8-a584714724b1 Address:fbe7299e-6013-e005-eed8-a584714724b1} {Suffrage:Voter ID:server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 Address:4a03c6e6-c95f-4ff0-e568-ce21d70492e4} {Suffrage:Voter ID:server-94770c92-0b7f-678e-85c6-b3fbbea56831 Address:94770c92-0b7f-678e-85c6-b3fbbea56831}]"
2025-07-14T05:38:31.671Z [INFO]  server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4: entering follower state: follower="Node at 4a03c6e6-c95f-4ff0-e568-ce21d70492e4 [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.671Z [INFO]  server-94770c92-0b7f-678e-85c6-b3fbbea56831: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e10d5ca-ec80-9910-803a-42c583d40e64 Address:8e10d5ca-ec80-9910-803a-42c583d40e64} {Suffrage:Voter ID:server-fbe7299e-6013-e005-eed8-a584714724b1 Address:fbe7299e-6013-e005-eed8-a584714724b1} {Suffrage:Voter ID:server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 Address:4a03c6e6-c95f-4ff0-e568-ce21d70492e4} {Suffrage:Voter ID:server-94770c92-0b7f-678e-85c6-b3fbbea56831 Address:94770c92-0b7f-678e-85c6-b3fbbea56831}]"
2025-07-14T05:38:31.671Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:31.671Z [INFO]  server-94770c92-0b7f-678e-85c6-b3fbbea56831: entering follower state: follower="Node at 94770c92-0b7f-678e-85c6-b3fbbea56831 [Follower]" leader-address= leader-id=
2025-07-14T05:38:31.726Z [WARN]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: entering candidate state: node="Node at 8e10d5ca-ec80-9910-803a-42c583d40e64 [Candidate]" term=2
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: election won: term=2 tally=3
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: entering leader state: leader="Node at 8e10d5ca-ec80-9910-803a-42c583d40e64 [Leader]"
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: added peer, starting replication: peer=server-fbe7299e-6013-e005-eed8-a584714724b1
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: added peer, starting replication: peer=server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: added peer, starting replication: peer=server-94770c92-0b7f-678e-85c6-b3fbbea56831
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: pipelining replication: peer="{Voter server-94770c92-0b7f-678e-85c6-b3fbbea56831 94770c92-0b7f-678e-85c6-b3fbbea56831}"
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: pipelining replication: peer="{Voter server-fbe7299e-6013-e005-eed8-a584714724b1 fbe7299e-6013-e005-eed8-a584714724b1}"
2025-07-14T05:38:31.726Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: pipelining replication: peer="{Voter server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 4a03c6e6-c95f-4ff0-e568-ce21d70492e4}"
2025-07-14T05:38:31.832Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:31.726720352 +0000 UTC m=+11.141180740 (1 nodes), 55.227441ms from start of poll, 55.655974ms from cluster start. Timeout at 2025-07-14 05:38:31.832369246 +0000 UTC m=+11.246829614, 105.648874ms after stability
2025-07-14T05:38:31.938Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:32.043Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:31.938118379 +0000 UTC m=+11.352578747 (3 nodes), 821ns from start of poll, 267.053981ms from cluster start. Timeout at 2025-07-14 05:38:32.043174123 +0000 UTC m=+11.457634521, 105.055774ms after stability
2025-07-14T05:38:32.043Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: updating configuration: command=RemoveServer server-id=server-94770c92-0b7f-678e-85c6-b3fbbea56831 server-addr= servers="[{Suffrage:Voter ID:server-8e10d5ca-ec80-9910-803a-42c583d40e64 Address:8e10d5ca-ec80-9910-803a-42c583d40e64} {Suffrage:Voter ID:server-fbe7299e-6013-e005-eed8-a584714724b1 Address:fbe7299e-6013-e005-eed8-a584714724b1} {Suffrage:Voter ID:server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 Address:4a03c6e6-c95f-4ff0-e568-ce21d70492e4}]"
2025-07-14T05:38:32.043Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: removed peer, stopping replication: peer=server-94770c92-0b7f-678e-85c6-b3fbbea56831 last-index=3
2025-07-14T05:38:32.043Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to pipeline appendEntries: peer="{Voter server-fbe7299e-6013-e005-eed8-a584714724b1 fbe7299e-6013-e005-eed8-a584714724b1}" error="append pipeline closed"
2025-07-14T05:38:32.043Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to pipeline appendEntries: peer="{Voter server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 4a03c6e6-c95f-4ff0-e568-ce21d70492e4}" error="append pipeline closed"
2025-07-14T05:38:32.043Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: aborting pipeline replication: peer="{Voter server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 4a03c6e6-c95f-4ff0-e568-ce21d70492e4}"
2025-07-14T05:38:32.043Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: aborting pipeline replication: peer="{Voter server-94770c92-0b7f-678e-85c6-b3fbbea56831 94770c92-0b7f-678e-85c6-b3fbbea56831}"
2025-07-14T05:38:32.043Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: aborting pipeline replication: peer="{Voter server-fbe7299e-6013-e005-eed8-a584714724b1 fbe7299e-6013-e005-eed8-a584714724b1}"
2025-07-14T05:38:32.044Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to heartbeat to: peer=4a03c6e6-c95f-4ff0-e568-ce21d70492e4 backoff time=10ms error="failed to connect to peer: 4a03c6e6-c95f-4ff0-e568-ce21d70492e4"
2025-07-14T05:38:32.049Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to heartbeat to: peer=fbe7299e-6013-e005-eed8-a584714724b1 backoff time=10ms error="failed to connect to peer: fbe7299e-6013-e005-eed8-a584714724b1"
2025-07-14T05:38:32.051Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-fbe7299e-6013-e005-eed8-a584714724b1 fbe7299e-6013-e005-eed8-a584714724b1}" error="failed to connect to peer: fbe7299e-6013-e005-eed8-a584714724b1"
2025-07-14T05:38:32.052Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 4a03c6e6-c95f-4ff0-e568-ce21d70492e4}" error="failed to connect to peer: 4a03c6e6-c95f-4ff0-e568-ce21d70492e4"
2025-07-14T05:38:32.063Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to heartbeat to: peer=4a03c6e6-c95f-4ff0-e568-ce21d70492e4 backoff time=10ms error="failed to connect to peer: 4a03c6e6-c95f-4ff0-e568-ce21d70492e4"
2025-07-14T05:38:32.067Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to heartbeat to: peer=fbe7299e-6013-e005-eed8-a584714724b1 backoff time=10ms error="failed to connect to peer: fbe7299e-6013-e005-eed8-a584714724b1"
2025-07-14T05:38:32.068Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-fbe7299e-6013-e005-eed8-a584714724b1 fbe7299e-6013-e005-eed8-a584714724b1}" error="failed to connect to peer: fbe7299e-6013-e005-eed8-a584714724b1"
2025-07-14T05:38:32.069Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 4a03c6e6-c95f-4ff0-e568-ce21d70492e4}" error="failed to connect to peer: 4a03c6e6-c95f-4ff0-e568-ce21d70492e4"
2025-07-14T05:38:32.082Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to heartbeat to: peer=4a03c6e6-c95f-4ff0-e568-ce21d70492e4 backoff time=10ms error="failed to connect to peer: 4a03c6e6-c95f-4ff0-e568-ce21d70492e4"
2025-07-14T05:38:32.084Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-fbe7299e-6013-e005-eed8-a584714724b1 fbe7299e-6013-e005-eed8-a584714724b1}" error="failed to connect to peer: fbe7299e-6013-e005-eed8-a584714724b1"
2025-07-14T05:38:32.087Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 4a03c6e6-c95f-4ff0-e568-ce21d70492e4}" error="failed to connect to peer: 4a03c6e6-c95f-4ff0-e568-ce21d70492e4"
2025-07-14T05:38:32.087Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to heartbeat to: peer=fbe7299e-6013-e005-eed8-a584714724b1 backoff time=10ms error="failed to connect to peer: fbe7299e-6013-e005-eed8-a584714724b1"
2025-07-14T05:38:32.090Z [WARN]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to contact: server-id=server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 time=50.281683ms
2025-07-14T05:38:32.100Z [WARN]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to contact: server-id=server-fbe7299e-6013-e005-eed8-a584714724b1 time=57.624887ms
2025-07-14T05:38:32.100Z [WARN]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to contact: server-id=server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 time=60.948573ms
2025-07-14T05:38:32.100Z [WARN]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to contact quorum of nodes, stepping down
2025-07-14T05:38:32.100Z [INFO]  server-8e10d5ca-ec80-9910-803a-42c583d40e64: entering follower state: follower="Node at 8e10d5ca-ec80-9910-803a-42c583d40e64 [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.100Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-fbe7299e-6013-e005-eed8-a584714724b1 fbe7299e-6013-e005-eed8-a584714724b1}" error="failed to connect to peer: fbe7299e-6013-e005-eed8-a584714724b1"
2025-07-14T05:38:32.100Z [ERROR] server-8e10d5ca-ec80-9910-803a-42c583d40e64: failed to appendEntries to: peer="{Voter server-4a03c6e6-c95f-4ff0-e568-ce21d70492e4 4a03c6e6-c95f-4ff0-e568-ce21d70492e4}" error="failed to connect to peer: 4a03c6e6-c95f-4ff0-e568-ce21d70492e4"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s)
=== RUN   TestRaft_AddKnownPeer
2025-07-14T05:38:32.102Z [INFO]  server-f408e67e-0197-423b-f377-ce173ed52380: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f408e67e-0197-423b-f377-ce173ed52380 Address:f408e67e-0197-423b-f377-ce173ed52380} {Suffrage:Voter ID:server-3385126c-1b3b-d8cb-c53d-347939396b34 Address:3385126c-1b3b-d8cb-c53d-347939396b34} {Suffrage:Voter ID:server-e6b4e644-3dbd-6147-23bb-49371f8db39f Address:e6b4e644-3dbd-6147-23bb-49371f8db39f}]"
2025-07-14T05:38:32.102Z [INFO]  server-f408e67e-0197-423b-f377-ce173ed52380: entering follower state: follower="Node at f408e67e-0197-423b-f377-ce173ed52380 [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.102Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f408e67e-0197-423b-f377-ce173ed52380 Address:f408e67e-0197-423b-f377-ce173ed52380} {Suffrage:Voter ID:server-3385126c-1b3b-d8cb-c53d-347939396b34 Address:3385126c-1b3b-d8cb-c53d-347939396b34} {Suffrage:Voter ID:server-e6b4e644-3dbd-6147-23bb-49371f8db39f Address:e6b4e644-3dbd-6147-23bb-49371f8db39f}]"
2025-07-14T05:38:32.102Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: entering follower state: follower="Node at 3385126c-1b3b-d8cb-c53d-347939396b34 [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.102Z [INFO]  server-e6b4e644-3dbd-6147-23bb-49371f8db39f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f408e67e-0197-423b-f377-ce173ed52380 Address:f408e67e-0197-423b-f377-ce173ed52380} {Suffrage:Voter ID:server-3385126c-1b3b-d8cb-c53d-347939396b34 Address:3385126c-1b3b-d8cb-c53d-347939396b34} {Suffrage:Voter ID:server-e6b4e644-3dbd-6147-23bb-49371f8db39f Address:e6b4e644-3dbd-6147-23bb-49371f8db39f}]"
2025-07-14T05:38:32.102Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:32.102Z [INFO]  server-e6b4e644-3dbd-6147-23bb-49371f8db39f: entering follower state: follower="Node at e6b4e644-3dbd-6147-23bb-49371f8db39f [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.156Z [WARN]  server-3385126c-1b3b-d8cb-c53d-347939396b34: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:32.156Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: entering candidate state: node="Node at 3385126c-1b3b-d8cb-c53d-347939396b34 [Candidate]" term=2
2025-07-14T05:38:32.157Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: election won: term=2 tally=2
2025-07-14T05:38:32.157Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: entering leader state: leader="Node at 3385126c-1b3b-d8cb-c53d-347939396b34 [Leader]"
2025-07-14T05:38:32.157Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: added peer, starting replication: peer=server-f408e67e-0197-423b-f377-ce173ed52380
2025-07-14T05:38:32.157Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: added peer, starting replication: peer=server-e6b4e644-3dbd-6147-23bb-49371f8db39f
2025-07-14T05:38:32.157Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: pipelining replication: peer="{Voter server-e6b4e644-3dbd-6147-23bb-49371f8db39f e6b4e644-3dbd-6147-23bb-49371f8db39f}"
2025-07-14T05:38:32.157Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: pipelining replication: peer="{Voter server-f408e67e-0197-423b-f377-ce173ed52380 f408e67e-0197-423b-f377-ce173ed52380}"
2025-07-14T05:38:32.262Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:32.157106131 +0000 UTC m=+11.571566499 (1 nodes), 54.70661ms from start of poll, 54.987662ms from cluster start. Timeout at 2025-07-14 05:38:32.262715515 +0000 UTC m=+11.677175913, 105.609414ms after stability
2025-07-14T05:38:32.262Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:32.368Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:32.262780422 +0000 UTC m=+11.677240800 (2 nodes), 1.121µs from start of poll, 160.661963ms from cluster start. Timeout at 2025-07-14 05:38:32.368466712 +0000 UTC m=+11.782927240, 105.68644ms after stability
2025-07-14T05:38:32.368Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: updating configuration: command=AddVoter server-id=server-f408e67e-0197-423b-f377-ce173ed52380 server-addr=f408e67e-0197-423b-f377-ce173ed52380 servers="[{Suffrage:Voter ID:server-f408e67e-0197-423b-f377-ce173ed52380 Address:f408e67e-0197-423b-f377-ce173ed52380} {Suffrage:Voter ID:server-3385126c-1b3b-d8cb-c53d-347939396b34 Address:3385126c-1b3b-d8cb-c53d-347939396b34} {Suffrage:Voter ID:server-e6b4e644-3dbd-6147-23bb-49371f8db39f Address:e6b4e644-3dbd-6147-23bb-49371f8db39f}]"
2025-07-14T05:38:32.368Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: aborting pipeline replication: peer="{Voter server-e6b4e644-3dbd-6147-23bb-49371f8db39f e6b4e644-3dbd-6147-23bb-49371f8db39f}"
2025-07-14T05:38:32.368Z [INFO]  server-3385126c-1b3b-d8cb-c53d-347939396b34: aborting pipeline replication: peer="{Voter server-f408e67e-0197-423b-f377-ce173ed52380 f408e67e-0197-423b-f377-ce173ed52380}"
--- PASS: TestRaft_AddKnownPeer (0.27s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-07-14T05:38:32.369Z [INFO]  server-d0df9775-44da-acde-23bb-792d39fa3fb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0df9775-44da-acde-23bb-792d39fa3fb0 Address:d0df9775-44da-acde-23bb-792d39fa3fb0} {Suffrage:Voter ID:server-88655591-f392-8ab9-ad18-bf2c056e1800 Address:88655591-f392-8ab9-ad18-bf2c056e1800} {Suffrage:Voter ID:server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad Address:d10fc00a-239a-969d-23fa-d8ba5abeb1ad}]"
2025-07-14T05:38:32.369Z [INFO]  server-d0df9775-44da-acde-23bb-792d39fa3fb0: entering follower state: follower="Node at d0df9775-44da-acde-23bb-792d39fa3fb0 [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.369Z [INFO]  server-88655591-f392-8ab9-ad18-bf2c056e1800: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0df9775-44da-acde-23bb-792d39fa3fb0 Address:d0df9775-44da-acde-23bb-792d39fa3fb0} {Suffrage:Voter ID:server-88655591-f392-8ab9-ad18-bf2c056e1800 Address:88655591-f392-8ab9-ad18-bf2c056e1800} {Suffrage:Voter ID:server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad Address:d10fc00a-239a-969d-23fa-d8ba5abeb1ad}]"
2025-07-14T05:38:32.369Z [INFO]  server-88655591-f392-8ab9-ad18-bf2c056e1800: entering follower state: follower="Node at 88655591-f392-8ab9-ad18-bf2c056e1800 [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.369Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0df9775-44da-acde-23bb-792d39fa3fb0 Address:d0df9775-44da-acde-23bb-792d39fa3fb0} {Suffrage:Voter ID:server-88655591-f392-8ab9-ad18-bf2c056e1800 Address:88655591-f392-8ab9-ad18-bf2c056e1800} {Suffrage:Voter ID:server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad Address:d10fc00a-239a-969d-23fa-d8ba5abeb1ad}]"
2025-07-14T05:38:32.370Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:32.370Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: entering follower state: follower="Node at d10fc00a-239a-969d-23fa-d8ba5abeb1ad [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.430Z [WARN]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:32.430Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: entering candidate state: node="Node at d10fc00a-239a-969d-23fa-d8ba5abeb1ad [Candidate]" term=2
2025-07-14T05:38:32.430Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: election won: term=2 tally=2
2025-07-14T05:38:32.430Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: entering leader state: leader="Node at d10fc00a-239a-969d-23fa-d8ba5abeb1ad [Leader]"
2025-07-14T05:38:32.430Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: added peer, starting replication: peer=server-d0df9775-44da-acde-23bb-792d39fa3fb0
2025-07-14T05:38:32.430Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: added peer, starting replication: peer=server-88655591-f392-8ab9-ad18-bf2c056e1800
2025-07-14T05:38:32.430Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: pipelining replication: peer="{Voter server-88655591-f392-8ab9-ad18-bf2c056e1800 88655591-f392-8ab9-ad18-bf2c056e1800}"
2025-07-14T05:38:32.430Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: pipelining replication: peer="{Voter server-d0df9775-44da-acde-23bb-792d39fa3fb0 d0df9775-44da-acde-23bb-792d39fa3fb0}"
2025-07-14T05:38:32.535Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:32.430261404 +0000 UTC m=+11.844721782 (1 nodes), 60.221553ms from start of poll, 60.520972ms from cluster start. Timeout at 2025-07-14 05:38:32.535500903 +0000 UTC m=+11.949961291, 105.239509ms after stability
2025-07-14T05:38:32.535Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: updating configuration: command=RemoveServer server-id=e426c79d-5ad9-7cc0-22fd-f8a91eb5f038 server-addr= servers="[{Suffrage:Voter ID:server-d0df9775-44da-acde-23bb-792d39fa3fb0 Address:d0df9775-44da-acde-23bb-792d39fa3fb0} {Suffrage:Voter ID:server-88655591-f392-8ab9-ad18-bf2c056e1800 Address:88655591-f392-8ab9-ad18-bf2c056e1800} {Suffrage:Voter ID:server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad Address:d10fc00a-239a-969d-23fa-d8ba5abeb1ad}]"
2025-07-14T05:38:32.535Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: aborting pipeline replication: peer="{Voter server-88655591-f392-8ab9-ad18-bf2c056e1800 88655591-f392-8ab9-ad18-bf2c056e1800}"
2025-07-14T05:38:32.535Z [INFO]  server-d10fc00a-239a-969d-23fa-d8ba5abeb1ad: aborting pipeline replication: peer="{Voter server-d0df9775-44da-acde-23bb-792d39fa3fb0 d0df9775-44da-acde-23bb-792d39fa3fb0}"
--- PASS: TestRaft_RemoveUnknownPeer (0.17s)
=== RUN   TestRaft_SnapshotRestore
2025-07-14T05:38:32.536Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d1db090-414b-8e8a-65f3-c10a3333f9db Address:1d1db090-414b-8e8a-65f3-c10a3333f9db}]"
2025-07-14T05:38:32.536Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:32.536Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: entering follower state: follower="Node at 1d1db090-414b-8e8a-65f3-c10a3333f9db [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.612Z [WARN]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:32.612Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: entering candidate state: node="Node at 1d1db090-414b-8e8a-65f3-c10a3333f9db [Candidate]" term=2
2025-07-14T05:38:32.612Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: election won: term=2 tally=1
2025-07-14T05:38:32.612Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: entering leader state: leader="Node at 1d1db090-414b-8e8a-65f3-c10a3333f9db [Leader]"
2025-07-14T05:38:32.718Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:32.613000112 +0000 UTC m=+12.027460510 (1 nodes), 76.260361ms from start of poll, 76.366089ms from cluster start. Timeout at 2025-07-14 05:38:32.71804564 +0000 UTC m=+12.132506038, 105.045528ms after stability
2025-07-14T05:38:32.719Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: starting snapshot up to: index=102
2025-07-14T05:38:32.719Z [INFO]  creating new snapshot: path=/tmp/raft3236566235/snapshots/2-102-1752471512719.tmp
2025-07-14T05:38:32.720Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: compacting logs: from=1 to=92
2025-07-14T05:38:32.720Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: snapshot complete up to: index=102
2025-07-14T05:38:32.720Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: starting restore from snapshot: id=2-102-1752471512719 last-index=102 last-term=2 size-in-bytes=693
2025-07-14T05:38:32.720Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: snapshot restore progress: id=2-102-1752471512719 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-14T05:38:32.720Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: restored from snapshot: id=2-102-1752471512719 last-index=102 last-term=2 size-in-bytes=693
2025-07-14T05:38:32.720Z [INFO]  server-1d1db090-414b-8e8a-65f3-c10a3333f9db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d1db090-414b-8e8a-65f3-c10a3333f9db Address:1d1db090-414b-8e8a-65f3-c10a3333f9db}]"
--- PASS: TestRaft_SnapshotRestore (0.25s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-07-14T05:38:32.790Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75425897-65d3-e660-d1f1-c28aeabf721e Address:75425897-65d3-e660-d1f1-c28aeabf721e}]"
2025-07-14T05:38:32.790Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:32.790Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: entering follower state: follower="Node at 75425897-65d3-e660-d1f1-c28aeabf721e [Follower]" leader-address= leader-id=
2025-07-14T05:38:32.885Z [WARN]  server-75425897-65d3-e660-d1f1-c28aeabf721e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:32.885Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: entering candidate state: node="Node at 75425897-65d3-e660-d1f1-c28aeabf721e [Candidate]" term=2
2025-07-14T05:38:32.885Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: election won: term=2 tally=1
2025-07-14T05:38:32.885Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: entering leader state: leader="Node at 75425897-65d3-e660-d1f1-c28aeabf721e [Leader]"
2025-07-14T05:38:32.991Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:32.885560642 +0000 UTC m=+12.300021011 (1 nodes), 95.217206ms from start of poll, 95.358278ms from cluster start. Timeout at 2025-07-14 05:38:32.991313061 +0000 UTC m=+12.405773429, 105.752418ms after stability
2025-07-14T05:38:32.992Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: starting snapshot up to: index=102
2025-07-14T05:38:32.993Z [INFO]  creating new snapshot: path=/tmp/raft3109092701/snapshots/2-102-1752471512993.tmp
2025-07-14T05:38:32.993Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: compacting logs: from=1 to=92
2025-07-14T05:38:32.993Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: snapshot complete up to: index=102
2025-07-14T05:38:32.993Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: starting restore from snapshot: id=2-102-1752471512993 last-index=102 last-term=2 size-in-bytes=693
2025-07-14T05:38:32.993Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: snapshot restore progress: id=2-102-1752471512993 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-14T05:38:32.994Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: restored from snapshot: id=2-102-1752471512993 last-index=102 last-term=2 size-in-bytes=693
2025-07-14T05:38:32.994Z [INFO]  server-75425897-65d3-e660-d1f1-c28aeabf721e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75425897-65d3-e660-d1f1-c28aeabf721e Address:75425897-65d3-e660-d1f1-c28aeabf721e}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.21s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-07-14T05:38:32.995Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a746ec89-f483-0979-a6c1-03c806b44a7e Address:a746ec89-f483-0979-a6c1-03c806b44a7e}]"
2025-07-14T05:38:32.995Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:32.995Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: entering follower state: follower="Node at a746ec89-f483-0979-a6c1-03c806b44a7e [Follower]" leader-address= leader-id=
2025-07-14T05:38:33.084Z [WARN]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:33.084Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: entering candidate state: node="Node at a746ec89-f483-0979-a6c1-03c806b44a7e [Candidate]" term=2
2025-07-14T05:38:33.084Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: election won: term=2 tally=1
2025-07-14T05:38:33.084Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: entering leader state: leader="Node at a746ec89-f483-0979-a6c1-03c806b44a7e [Leader]"
2025-07-14T05:38:33.190Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:33.084886001 +0000 UTC m=+12.499346379 (1 nodes), 89.292269ms from start of poll, 89.4063ms from cluster start. Timeout at 2025-07-14 05:38:33.190030998 +0000 UTC m=+12.604491386, 105.145007ms after stability
2025-07-14T05:38:33.191Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: starting snapshot up to: index=102
2025-07-14T05:38:33.191Z [INFO]  creating new snapshot: path=/tmp/raft420859638/snapshots/2-102-1752471513191.tmp
2025-07-14T05:38:33.192Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: compacting logs: from=1 to=92
2025-07-14T05:38:33.192Z [INFO]  server-a746ec89-f483-0979-a6c1-03c806b44a7e: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.20s)
=== RUN   TestRaft_NoRestoreOnStart
2025-07-14T05:38:33.194Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca Address:4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca}]"
2025-07-14T05:38:33.194Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:33.194Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: entering follower state: follower="Node at 4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca [Follower]" leader-address= leader-id=
2025-07-14T05:38:33.252Z [WARN]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:33.252Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: entering candidate state: node="Node at 4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca [Candidate]" term=2
2025-07-14T05:38:33.252Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: election won: term=2 tally=1
2025-07-14T05:38:33.252Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: entering leader state: leader="Node at 4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca [Leader]"
2025-07-14T05:38:33.358Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:33.252819581 +0000 UTC m=+12.667279979 (1 nodes), 58.647371ms from start of poll, 58.748873ms from cluster start. Timeout at 2025-07-14 05:38:33.357984208 +0000 UTC m=+12.772444586, 105.164607ms after stability
2025-07-14T05:38:33.360Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: starting snapshot up to: index=102
2025-07-14T05:38:33.360Z [INFO]  creating new snapshot: path=/tmp/raft4106131857/snapshots/2-102-1752471513360.tmp
2025-07-14T05:38:33.361Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: compacting logs: from=1 to=92
2025-07-14T05:38:33.361Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: snapshot complete up to: index=102
2025-07-14T05:38:33.361Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca Address:4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca}]"
--- PASS: TestRaft_NoRestoreOnStart (0.17s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
2025-07-14T05:38:33.361Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: entering follower state: follower="Node at 4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca [Follower]" leader-address= leader-id=
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-07-14T05:38:33.361Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4998327-ceb6-d10a-2443-1524a8b5dcfc Address:f4998327-ceb6-d10a-2443-1524a8b5dcfc}]"
2025-07-14T05:38:33.361Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:33.361Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: entering follower state: follower="Node at f4998327-ceb6-d10a-2443-1524a8b5dcfc [Follower]" leader-address= leader-id=
2025-07-14T05:38:33.424Z [WARN]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:33.424Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: entering candidate state: node="Node at f4998327-ceb6-d10a-2443-1524a8b5dcfc [Candidate]" term=2
2025-07-14T05:38:33.424Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: election won: term=2 tally=1
2025-07-14T05:38:33.424Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: entering leader state: leader="Node at f4998327-ceb6-d10a-2443-1524a8b5dcfc [Leader]"
2025-07-14T05:38:33.438Z [WARN]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:33.439Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: entering candidate state: node="Node at 4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca [Candidate]" term=3
2025-07-14T05:38:33.439Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: election won: term=3 tally=1
2025-07-14T05:38:33.439Z [INFO]  server-4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca: entering leader state: leader="Node at 4d635d62-4dd3-0e0f-0c34-ccd577e8c1ca [Leader]"
2025-07-14T05:38:33.530Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:33.424751769 +0000 UTC m=+12.839212147 (1 nodes), 63.067671ms from start of poll, 63.167892ms from cluster start. Timeout at 2025-07-14 05:38:33.530221113 +0000 UTC m=+12.944681501, 105.469354ms after stability
2025-07-14T05:38:33.538Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: starting snapshot up to: index=102
2025-07-14T05:38:33.538Z [INFO]  creating new snapshot: path=/tmp/raft2814955886/snapshots/2-102-1752471513538.tmp
2025-07-14T05:38:33.538Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: compacting logs: from=1 to=92
2025-07-14T05:38:33.539Z [INFO]  server-f4998327-ceb6-d10a-2443-1524a8b5dcfc: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.28s)
=== RUN   TestRaft_UserSnapshot
2025-07-14T05:38:33.637Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d Address:f4d6beed-1aad-e30c-c3c8-e136ae767e8d}]"
2025-07-14T05:38:33.637Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:33.637Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: entering follower state: follower="Node at f4d6beed-1aad-e30c-c3c8-e136ae767e8d [Follower]" leader-address= leader-id=
2025-07-14T05:38:33.693Z [WARN]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:33.693Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: entering candidate state: node="Node at f4d6beed-1aad-e30c-c3c8-e136ae767e8d [Candidate]" term=2
2025-07-14T05:38:33.694Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: election won: term=2 tally=1
2025-07-14T05:38:33.694Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: entering leader state: leader="Node at f4d6beed-1aad-e30c-c3c8-e136ae767e8d [Leader]"
2025-07-14T05:38:33.799Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:33.694056666 +0000 UTC m=+13.108517044 (1 nodes), 56.06672ms from start of poll, 56.192418ms from cluster start. Timeout at 2025-07-14 05:38:33.799276596 +0000 UTC m=+13.213736984, 105.21994ms after stability
2025-07-14T05:38:33.799Z [ERROR] server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: failed to take snapshot: error="nothing new to snapshot"
2025-07-14T05:38:33.799Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: starting snapshot up to: index=12
2025-07-14T05:38:33.799Z [INFO]  creating new snapshot: path=/tmp/raft3125444736/snapshots/2-12-1752471513799.tmp
2025-07-14T05:38:33.800Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: compacting logs: from=1 to=2
2025-07-14T05:38:33.800Z [INFO]  server-f4d6beed-1aad-e30c-c3c8-e136ae767e8d: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.16s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-07-14T05:38:33.802Z [INFO]  server-798124a2-db0d-4841-3a30-5b8db8316dac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-798124a2-db0d-4841-3a30-5b8db8316dac Address:798124a2-db0d-4841-3a30-5b8db8316dac} {Suffrage:Voter ID:server-e83627b3-d479-0f02-e70b-4a52087f29ba Address:e83627b3-d479-0f02-e70b-4a52087f29ba} {Suffrage:Voter ID:server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7 Address:cc3d9832-7052-3c0f-a7c6-b57492d5a5b7}]"
2025-07-14T05:38:33.802Z [INFO]  server-798124a2-db0d-4841-3a30-5b8db8316dac: entering follower state: follower="Node at 798124a2-db0d-4841-3a30-5b8db8316dac [Follower]" leader-address= leader-id=
2025-07-14T05:38:33.802Z [INFO]  server-e83627b3-d479-0f02-e70b-4a52087f29ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-798124a2-db0d-4841-3a30-5b8db8316dac Address:798124a2-db0d-4841-3a30-5b8db8316dac} {Suffrage:Voter ID:server-e83627b3-d479-0f02-e70b-4a52087f29ba Address:e83627b3-d479-0f02-e70b-4a52087f29ba} {Suffrage:Voter ID:server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7 Address:cc3d9832-7052-3c0f-a7c6-b57492d5a5b7}]"
2025-07-14T05:38:33.802Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-798124a2-db0d-4841-3a30-5b8db8316dac Address:798124a2-db0d-4841-3a30-5b8db8316dac} {Suffrage:Voter ID:server-e83627b3-d479-0f02-e70b-4a52087f29ba Address:e83627b3-d479-0f02-e70b-4a52087f29ba} {Suffrage:Voter ID:server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7 Address:cc3d9832-7052-3c0f-a7c6-b57492d5a5b7}]"
2025-07-14T05:38:33.802Z [INFO]  server-e83627b3-d479-0f02-e70b-4a52087f29ba: entering follower state: follower="Node at e83627b3-d479-0f02-e70b-4a52087f29ba [Follower]" leader-address= leader-id=
2025-07-14T05:38:33.802Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: entering follower state: follower="Node at cc3d9832-7052-3c0f-a7c6-b57492d5a5b7 [Follower]" leader-address= leader-id=
2025-07-14T05:38:33.802Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:34.337Z [WARN]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:34.337Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: entering candidate state: node="Node at cc3d9832-7052-3c0f-a7c6-b57492d5a5b7 [Candidate]" term=2
2025-07-14T05:38:34.337Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: election won: term=2 tally=2
2025-07-14T05:38:34.337Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: entering leader state: leader="Node at cc3d9832-7052-3c0f-a7c6-b57492d5a5b7 [Leader]"
2025-07-14T05:38:34.337Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: added peer, starting replication: peer=server-798124a2-db0d-4841-3a30-5b8db8316dac
2025-07-14T05:38:34.337Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: added peer, starting replication: peer=server-e83627b3-d479-0f02-e70b-4a52087f29ba
2025-07-14T05:38:34.337Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: pipelining replication: peer="{Voter server-e83627b3-d479-0f02-e70b-4a52087f29ba e83627b3-d479-0f02-e70b-4a52087f29ba}"
2025-07-14T05:38:34.337Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: pipelining replication: peer="{Voter server-798124a2-db0d-4841-3a30-5b8db8316dac 798124a2-db0d-4841-3a30-5b8db8316dac}"
2025-07-14T05:38:35.343Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:34.337175214 +0000 UTC m=+13.751635592 (1 nodes), 534.322974ms from start of poll, 534.643946ms from cluster start. Timeout at 2025-07-14 05:38:35.343056686 +0000 UTC m=+14.757517054, 1.005881462s after stability
2025-07-14T05:38:35.343Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: starting snapshot up to: index=12
2025-07-14T05:38:35.343Z [INFO]  creating new snapshot: path=/tmp/raft4031291060/snapshots/2-12-1752471515343.tmp
2025-07-14T05:38:35.344Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: snapshot complete up to: index=12
2025-07-14T05:38:35.345Z [INFO]  creating new snapshot: path=/tmp/raft4031291060/snapshots/2-23-1752471515345.tmp
2025-07-14T05:38:35.345Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: copied to local snapshot: bytes=71
2025-07-14T05:38:35.345Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: snapshot restore progress: id=2-23-1752471515345 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:35.345Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: restored user snapshot: index=23
2025-07-14T05:38:35.345Z [ERROR] server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: failed to get log: index=23 error="log not found"
2025-07-14T05:38:35.345Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: aborting pipeline replication: peer="{Voter server-e83627b3-d479-0f02-e70b-4a52087f29ba e83627b3-d479-0f02-e70b-4a52087f29ba}"
2025-07-14T05:38:35.345Z [ERROR] server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: failed to get log: index=23 error="log not found"
2025-07-14T05:38:35.345Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: aborting pipeline replication: peer="{Voter server-798124a2-db0d-4841-3a30-5b8db8316dac 798124a2-db0d-4841-3a30-5b8db8316dac}"
2025-07-14T05:38:35.353Z [ERROR] server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: failed to get log: index=23 error="log not found"
2025-07-14T05:38:35.354Z [INFO]  creating new snapshot: path=/tmp/raft3886932697/snapshots/2-23-1752471515354.tmp
2025-07-14T05:38:35.354Z [INFO]  server-e83627b3-d479-0f02-e70b-4a52087f29ba: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:35.354Z [ERROR] server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: failed to get log: index=23 error="log not found"
2025-07-14T05:38:35.354Z [INFO]  creating new snapshot: path=/tmp/raft3890575787/snapshots/2-23-1752471515354.tmp
2025-07-14T05:38:35.354Z [INFO]  server-e83627b3-d479-0f02-e70b-4a52087f29ba: copied to local snapshot: bytes=71
2025-07-14T05:38:35.354Z [INFO]  server-e83627b3-d479-0f02-e70b-4a52087f29ba: snapshot restore progress: id=2-23-1752471515354 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:35.354Z [INFO]  server-e83627b3-d479-0f02-e70b-4a52087f29ba: Installed remote snapshot
2025-07-14T05:38:35.354Z [INFO]  server-798124a2-db0d-4841-3a30-5b8db8316dac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:35.354Z [WARN]  server-e83627b3-d479-0f02-e70b-4a52087f29ba: skipping application of old log: index=22
2025-07-14T05:38:35.354Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: pipelining replication: peer="{Voter server-e83627b3-d479-0f02-e70b-4a52087f29ba e83627b3-d479-0f02-e70b-4a52087f29ba}"
2025-07-14T05:38:35.354Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:35.354Z [INFO]  server-798124a2-db0d-4841-3a30-5b8db8316dac: copied to local snapshot: bytes=71
2025-07-14T05:38:35.355Z [INFO]  server-798124a2-db0d-4841-3a30-5b8db8316dac: snapshot restore progress: id=2-23-1752471515354 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:35.355Z [INFO]  server-798124a2-db0d-4841-3a30-5b8db8316dac: Installed remote snapshot
2025-07-14T05:38:35.355Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: pipelining replication: peer="{Voter server-798124a2-db0d-4841-3a30-5b8db8316dac 798124a2-db0d-4841-3a30-5b8db8316dac}"
2025-07-14T05:38:36.360Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:35.354713271 +0000 UTC m=+14.769173649 (2 nodes), 771ns from start of poll, 1.552182003s from cluster start. Timeout at 2025-07-14 05:38:36.360169305 +0000 UTC m=+15.774629683, 1.005456034s after stability
2025-07-14T05:38:36.371Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: aborting pipeline replication: peer="{Voter server-e83627b3-d479-0f02-e70b-4a52087f29ba e83627b3-d479-0f02-e70b-4a52087f29ba}"
2025-07-14T05:38:36.371Z [INFO]  server-cc3d9832-7052-3c0f-a7c6-b57492d5a5b7: aborting pipeline replication: peer="{Voter server-798124a2-db0d-4841-3a30-5b8db8316dac 798124a2-db0d-4841-3a30-5b8db8316dac}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-07-14T05:38:36.392Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9612cfa3-46cb-4394-5cb1-1443477fca5a Address:9612cfa3-46cb-4394-5cb1-1443477fca5a} {Suffrage:Voter ID:server-c5f1d07c-89d1-0f25-e857-f57069292244 Address:c5f1d07c-89d1-0f25-e857-f57069292244} {Suffrage:Voter ID:server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 Address:d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57}]"
2025-07-14T05:38:36.392Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9612cfa3-46cb-4394-5cb1-1443477fca5a Address:9612cfa3-46cb-4394-5cb1-1443477fca5a} {Suffrage:Voter ID:server-c5f1d07c-89d1-0f25-e857-f57069292244 Address:c5f1d07c-89d1-0f25-e857-f57069292244} {Suffrage:Voter ID:server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 Address:d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57}]"
2025-07-14T05:38:36.392Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: entering follower state: follower="Node at 9612cfa3-46cb-4394-5cb1-1443477fca5a [Follower]" leader-address= leader-id=
2025-07-14T05:38:36.392Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9612cfa3-46cb-4394-5cb1-1443477fca5a Address:9612cfa3-46cb-4394-5cb1-1443477fca5a} {Suffrage:Voter ID:server-c5f1d07c-89d1-0f25-e857-f57069292244 Address:c5f1d07c-89d1-0f25-e857-f57069292244} {Suffrage:Voter ID:server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 Address:d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57}]"
2025-07-14T05:38:36.392Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: entering follower state: follower="Node at c5f1d07c-89d1-0f25-e857-f57069292244 [Follower]" leader-address= leader-id=
2025-07-14T05:38:36.392Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: entering follower state: follower="Node at d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 [Follower]" leader-address= leader-id=
2025-07-14T05:38:36.392Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:36.960Z [WARN]  server-c5f1d07c-89d1-0f25-e857-f57069292244: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:36.960Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: entering candidate state: node="Node at c5f1d07c-89d1-0f25-e857-f57069292244 [Candidate]" term=2
2025-07-14T05:38:36.960Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: election won: term=2 tally=2
2025-07-14T05:38:36.960Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: entering leader state: leader="Node at c5f1d07c-89d1-0f25-e857-f57069292244 [Leader]"
2025-07-14T05:38:36.960Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: added peer, starting replication: peer=server-9612cfa3-46cb-4394-5cb1-1443477fca5a
2025-07-14T05:38:36.960Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: added peer, starting replication: peer=server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57
2025-07-14T05:38:36.960Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: pipelining replication: peer="{Voter server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57}"
2025-07-14T05:38:36.960Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: pipelining replication: peer="{Voter server-9612cfa3-46cb-4394-5cb1-1443477fca5a 9612cfa3-46cb-4394-5cb1-1443477fca5a}"
2025-07-14T05:38:37.966Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:36.960633171 +0000 UTC m=+16.375093569 (1 nodes), 568.236054ms from start of poll, 568.635584ms from cluster start. Timeout at 2025-07-14 05:38:37.966415164 +0000 UTC m=+17.380875552, 1.005781983s after stability
2025-07-14T05:38:37.966Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: starting snapshot up to: index=12
2025-07-14T05:38:37.966Z [INFO]  creating new snapshot: path=/tmp/raft2072728492/snapshots/2-12-1752471517966.tmp
2025-07-14T05:38:37.967Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: snapshot complete up to: index=12
2025-07-14T05:38:37.967Z [INFO]  creating new snapshot: path=/tmp/raft2072728492/snapshots/2-23-1752471517967.tmp
2025-07-14T05:38:37.968Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: copied to local snapshot: bytes=71
2025-07-14T05:38:37.968Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: snapshot restore progress: id=2-23-1752471517967 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:37.968Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: removing all old logs from log store
2025-07-14T05:38:37.968Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: compacting logs: from=1 to=22
2025-07-14T05:38:37.968Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: restored user snapshot: index=23
2025-07-14T05:38:37.968Z [ERROR] server-c5f1d07c-89d1-0f25-e857-f57069292244: failed to get log: index=22 error="log not found"
2025-07-14T05:38:37.968Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: aborting pipeline replication: peer="{Voter server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57}"
2025-07-14T05:38:37.968Z [ERROR] server-c5f1d07c-89d1-0f25-e857-f57069292244: failed to get log: index=22 error="log not found"
2025-07-14T05:38:37.968Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: aborting pipeline replication: peer="{Voter server-9612cfa3-46cb-4394-5cb1-1443477fca5a 9612cfa3-46cb-4394-5cb1-1443477fca5a}"
2025-07-14T05:38:37.976Z [ERROR] server-c5f1d07c-89d1-0f25-e857-f57069292244: failed to get log: index=22 error="log not found"
2025-07-14T05:38:37.976Z [INFO]  creating new snapshot: path=/tmp/raft853042590/snapshots/2-23-1752471517976.tmp
2025-07-14T05:38:37.977Z [ERROR] server-c5f1d07c-89d1-0f25-e857-f57069292244: failed to get log: index=22 error="log not found"
2025-07-14T05:38:37.977Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:37.977Z [INFO]  creating new snapshot: path=/tmp/raft1017594596/snapshots/2-23-1752471517977.tmp
2025-07-14T05:38:37.977Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:37.984Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: copied to local snapshot: bytes=71
2025-07-14T05:38:37.984Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: copied to local snapshot: bytes=71
2025-07-14T05:38:37.984Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: snapshot restore progress: id=2-23-1752471517977 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:37.984Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: removing all old logs from log store
2025-07-14T05:38:37.984Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: compacting logs: from=1 to=22
2025-07-14T05:38:37.984Z [INFO]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: Installed remote snapshot
2025-07-14T05:38:37.984Z [WARN]  server-9612cfa3-46cb-4394-5cb1-1443477fca5a: skipping application of old log: index=22
2025-07-14T05:38:37.984Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: pipelining replication: peer="{Voter server-9612cfa3-46cb-4394-5cb1-1443477fca5a 9612cfa3-46cb-4394-5cb1-1443477fca5a}"
2025-07-14T05:38:37.984Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: snapshot restore progress: id=2-23-1752471517976 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:37.984Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: removing all old logs from log store
2025-07-14T05:38:37.984Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: compacting logs: from=1 to=22
2025-07-14T05:38:37.984Z [INFO]  server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57: Installed remote snapshot
2025-07-14T05:38:37.984Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:37.984Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: pipelining replication: peer="{Voter server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57}"
2025-07-14T05:38:38.990Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:37.984753832 +0000 UTC m=+17.399214220 (2 nodes), 551ns from start of poll, 1.592756235s from cluster start. Timeout at 2025-07-14 05:38:38.990433431 +0000 UTC m=+18.404893829, 1.005679609s after stability
2025-07-14T05:38:39.002Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: aborting pipeline replication: peer="{Voter server-d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57 d83c5e5d-b8b3-0b8a-7f8b-69ebe4c0ff57}"
2025-07-14T05:38:39.002Z [INFO]  server-c5f1d07c-89d1-0f25-e857-f57069292244: aborting pipeline replication: peer="{Voter server-9612cfa3-46cb-4394-5cb1-1443477fca5a 9612cfa3-46cb-4394-5cb1-1443477fca5a}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-07-14T05:38:39.003Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33afde97-8cdb-d633-d9c2-292ebfa97134 Address:33afde97-8cdb-d633-d9c2-292ebfa97134} {Suffrage:Voter ID:server-74e0741b-07f1-adda-4835-187499b44f52 Address:74e0741b-07f1-adda-4835-187499b44f52} {Suffrage:Voter ID:server-30bf91f5-394f-eacf-225e-08687771b6c2 Address:30bf91f5-394f-eacf-225e-08687771b6c2}]"
2025-07-14T05:38:39.003Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: entering follower state: follower="Node at 33afde97-8cdb-d633-d9c2-292ebfa97134 [Follower]" leader-address= leader-id=
2025-07-14T05:38:39.003Z [INFO]  server-74e0741b-07f1-adda-4835-187499b44f52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33afde97-8cdb-d633-d9c2-292ebfa97134 Address:33afde97-8cdb-d633-d9c2-292ebfa97134} {Suffrage:Voter ID:server-74e0741b-07f1-adda-4835-187499b44f52 Address:74e0741b-07f1-adda-4835-187499b44f52} {Suffrage:Voter ID:server-30bf91f5-394f-eacf-225e-08687771b6c2 Address:30bf91f5-394f-eacf-225e-08687771b6c2}]"
2025-07-14T05:38:39.003Z [INFO]  server-74e0741b-07f1-adda-4835-187499b44f52: entering follower state: follower="Node at 74e0741b-07f1-adda-4835-187499b44f52 [Follower]" leader-address= leader-id=
2025-07-14T05:38:39.003Z [INFO]  server-30bf91f5-394f-eacf-225e-08687771b6c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33afde97-8cdb-d633-d9c2-292ebfa97134 Address:33afde97-8cdb-d633-d9c2-292ebfa97134} {Suffrage:Voter ID:server-74e0741b-07f1-adda-4835-187499b44f52 Address:74e0741b-07f1-adda-4835-187499b44f52} {Suffrage:Voter ID:server-30bf91f5-394f-eacf-225e-08687771b6c2 Address:30bf91f5-394f-eacf-225e-08687771b6c2}]"
2025-07-14T05:38:39.003Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:39.003Z [INFO]  server-30bf91f5-394f-eacf-225e-08687771b6c2: entering follower state: follower="Node at 30bf91f5-394f-eacf-225e-08687771b6c2 [Follower]" leader-address= leader-id=
2025-07-14T05:38:39.623Z [WARN]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:39.624Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: entering candidate state: node="Node at 33afde97-8cdb-d633-d9c2-292ebfa97134 [Candidate]" term=2
2025-07-14T05:38:39.624Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: election won: term=2 tally=2
2025-07-14T05:38:39.624Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: entering leader state: leader="Node at 33afde97-8cdb-d633-d9c2-292ebfa97134 [Leader]"
2025-07-14T05:38:39.624Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: added peer, starting replication: peer=server-74e0741b-07f1-adda-4835-187499b44f52
2025-07-14T05:38:39.624Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: added peer, starting replication: peer=server-30bf91f5-394f-eacf-225e-08687771b6c2
2025-07-14T05:38:39.624Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: pipelining replication: peer="{Voter server-30bf91f5-394f-eacf-225e-08687771b6c2 30bf91f5-394f-eacf-225e-08687771b6c2}"
2025-07-14T05:38:39.624Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: pipelining replication: peer="{Voter server-74e0741b-07f1-adda-4835-187499b44f52 74e0741b-07f1-adda-4835-187499b44f52}"
2025-07-14T05:38:40.630Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:39.624185549 +0000 UTC m=+19.038645927 (1 nodes), 620.25991ms from start of poll, 620.541753ms from cluster start. Timeout at 2025-07-14 05:38:40.630002414 +0000 UTC m=+20.044462812, 1.005816885s after stability
2025-07-14T05:38:40.630Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: starting snapshot up to: index=12
2025-07-14T05:38:40.630Z [INFO]  creating new snapshot: path=/tmp/raft3093653423/snapshots/2-12-1752471520630.tmp
2025-07-14T05:38:40.631Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: snapshot complete up to: index=12
2025-07-14T05:38:40.632Z [INFO]  server-77fd14c6-01bb-136c-4c43-4f481046b98c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77fd14c6-01bb-136c-4c43-4f481046b98c Address:77fd14c6-01bb-136c-4c43-4f481046b98c} {Suffrage:Voter ID:server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf Address:79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf} {Suffrage:Voter ID:server-b63289b0-34e5-edbf-5d28-a3f15396035d Address:b63289b0-34e5-edbf-5d28-a3f15396035d}]"
2025-07-14T05:38:40.632Z [INFO]  server-77fd14c6-01bb-136c-4c43-4f481046b98c: entering follower state: follower="Node at 77fd14c6-01bb-136c-4c43-4f481046b98c [Follower]" leader-address= leader-id=
2025-07-14T05:38:40.632Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77fd14c6-01bb-136c-4c43-4f481046b98c Address:77fd14c6-01bb-136c-4c43-4f481046b98c} {Suffrage:Voter ID:server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf Address:79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf} {Suffrage:Voter ID:server-b63289b0-34e5-edbf-5d28-a3f15396035d Address:b63289b0-34e5-edbf-5d28-a3f15396035d}]"
2025-07-14T05:38:40.632Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: entering follower state: follower="Node at 79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf [Follower]" leader-address= leader-id=
2025-07-14T05:38:40.632Z [INFO]  server-b63289b0-34e5-edbf-5d28-a3f15396035d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77fd14c6-01bb-136c-4c43-4f481046b98c Address:77fd14c6-01bb-136c-4c43-4f481046b98c} {Suffrage:Voter ID:server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf Address:79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf} {Suffrage:Voter ID:server-b63289b0-34e5-edbf-5d28-a3f15396035d Address:b63289b0-34e5-edbf-5d28-a3f15396035d}]"
2025-07-14T05:38:40.632Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:40.632Z [INFO]  server-b63289b0-34e5-edbf-5d28-a3f15396035d: entering follower state: follower="Node at b63289b0-34e5-edbf-5d28-a3f15396035d [Follower]" leader-address= leader-id=
2025-07-14T05:38:41.148Z [WARN]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:41.148Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: entering candidate state: node="Node at 79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf [Candidate]" term=2
2025-07-14T05:38:41.148Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: election won: term=2 tally=2
2025-07-14T05:38:41.148Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: entering leader state: leader="Node at 79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf [Leader]"
2025-07-14T05:38:41.148Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: added peer, starting replication: peer=server-77fd14c6-01bb-136c-4c43-4f481046b98c
2025-07-14T05:38:41.148Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: added peer, starting replication: peer=server-b63289b0-34e5-edbf-5d28-a3f15396035d
2025-07-14T05:38:41.148Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: pipelining replication: peer="{Voter server-b63289b0-34e5-edbf-5d28-a3f15396035d b63289b0-34e5-edbf-5d28-a3f15396035d}"
2025-07-14T05:38:41.148Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: pipelining replication: peer="{Voter server-77fd14c6-01bb-136c-4c43-4f481046b98c 77fd14c6-01bb-136c-4c43-4f481046b98c}"
2025-07-14T05:38:42.153Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:41.148360551 +0000 UTC m=+20.562820939 (1 nodes), 515.811096ms from start of poll, 516.183695ms from cluster start. Timeout at 2025-07-14 05:38:42.153662193 +0000 UTC m=+21.568122572, 1.005301633s after stability
2025-07-14T05:38:42.153Z [INFO]  creating new snapshot: path=/tmp/raft4054101669/snapshots/2-13-1752471522153.tmp
2025-07-14T05:38:42.154Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: copied to local snapshot: bytes=71
2025-07-14T05:38:42.154Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: snapshot restore progress: id=2-13-1752471522153 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:42.154Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: restored user snapshot: index=13
2025-07-14T05:38:42.154Z [ERROR] server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: failed to get log: index=3 error="log not found"
2025-07-14T05:38:42.154Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: aborting pipeline replication: peer="{Voter server-b63289b0-34e5-edbf-5d28-a3f15396035d b63289b0-34e5-edbf-5d28-a3f15396035d}"
2025-07-14T05:38:42.154Z [ERROR] server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: failed to get log: index=3 error="log not found"
2025-07-14T05:38:42.154Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: aborting pipeline replication: peer="{Voter server-77fd14c6-01bb-136c-4c43-4f481046b98c 77fd14c6-01bb-136c-4c43-4f481046b98c}"
2025-07-14T05:38:42.163Z [ERROR] server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: failed to get log: index=3 error="log not found"
2025-07-14T05:38:42.163Z [ERROR] server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: failed to get log: index=3 error="log not found"
2025-07-14T05:38:42.163Z [INFO]  creating new snapshot: path=/tmp/raft2757725921/snapshots/2-13-1752471522163.tmp
2025-07-14T05:38:42.163Z [INFO]  creating new snapshot: path=/tmp/raft201885017/snapshots/2-13-1752471522163.tmp
2025-07-14T05:38:42.163Z [INFO]  server-b63289b0-34e5-edbf-5d28-a3f15396035d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:42.164Z [INFO]  server-b63289b0-34e5-edbf-5d28-a3f15396035d: copied to local snapshot: bytes=71
2025-07-14T05:38:42.164Z [INFO]  server-77fd14c6-01bb-136c-4c43-4f481046b98c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:42.164Z [INFO]  server-b63289b0-34e5-edbf-5d28-a3f15396035d: snapshot restore progress: id=2-13-1752471522163 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:42.164Z [INFO]  server-b63289b0-34e5-edbf-5d28-a3f15396035d: Installed remote snapshot
2025-07-14T05:38:42.164Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: pipelining replication: peer="{Voter server-b63289b0-34e5-edbf-5d28-a3f15396035d b63289b0-34e5-edbf-5d28-a3f15396035d}"
2025-07-14T05:38:42.165Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:42.165Z [INFO]  server-77fd14c6-01bb-136c-4c43-4f481046b98c: copied to local snapshot: bytes=71
2025-07-14T05:38:42.165Z [INFO]  server-77fd14c6-01bb-136c-4c43-4f481046b98c: snapshot restore progress: id=2-13-1752471522163 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:42.165Z [INFO]  server-77fd14c6-01bb-136c-4c43-4f481046b98c: Installed remote snapshot
2025-07-14T05:38:42.165Z [INFO]  server-79c9c3e2-221a-c60a-dc5a-7b240a7ac8bf: pipelining replication: peer="{Voter server-77fd14c6-01bb-136c-4c43-4f481046b98c 77fd14c6-01bb-136c-4c43-4f481046b98c}"
2025-07-14T05:38:43.170Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:42.165035023 +0000 UTC m=+21.579495411 (2 nodes), 1.372µs from start of poll, 1.532858167s from cluster start. Timeout at 2025-07-14 05:38:43.170164156 +0000 UTC m=+22.584624534, 1.005129123s after stability
2025-07-14T05:38:43.182Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: aborting pipeline replication: peer="{Voter server-30bf91f5-394f-eacf-225e-08687771b6c2 30bf91f5-394f-eacf-225e-08687771b6c2}"
2025-07-14T05:38:43.182Z [INFO]  server-33afde97-8cdb-d633-d9c2-292ebfa97134: aborting pipeline replication: peer="{Voter server-74e0741b-07f1-adda-4835-187499b44f52 74e0741b-07f1-adda-4835-187499b44f52}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-07-14T05:38:43.183Z [INFO]  server-ecd780fc-ad67-1e29-9b6a-98ca814983cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd780fc-ad67-1e29-9b6a-98ca814983cd Address:ecd780fc-ad67-1e29-9b6a-98ca814983cd} {Suffrage:Voter ID:server-00d335fc-0932-dc54-0cb2-ac56d54eaef8 Address:00d335fc-0932-dc54-0cb2-ac56d54eaef8} {Suffrage:Voter ID:server-814741b9-ece1-e22b-2dec-b580d16c0090 Address:814741b9-ece1-e22b-2dec-b580d16c0090}]"
2025-07-14T05:38:43.183Z [INFO]  server-ecd780fc-ad67-1e29-9b6a-98ca814983cd: entering follower state: follower="Node at ecd780fc-ad67-1e29-9b6a-98ca814983cd [Follower]" leader-address= leader-id=
2025-07-14T05:38:43.183Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd780fc-ad67-1e29-9b6a-98ca814983cd Address:ecd780fc-ad67-1e29-9b6a-98ca814983cd} {Suffrage:Voter ID:server-00d335fc-0932-dc54-0cb2-ac56d54eaef8 Address:00d335fc-0932-dc54-0cb2-ac56d54eaef8} {Suffrage:Voter ID:server-814741b9-ece1-e22b-2dec-b580d16c0090 Address:814741b9-ece1-e22b-2dec-b580d16c0090}]"
2025-07-14T05:38:43.184Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: entering follower state: follower="Node at 00d335fc-0932-dc54-0cb2-ac56d54eaef8 [Follower]" leader-address= leader-id=
2025-07-14T05:38:43.184Z [INFO]  server-814741b9-ece1-e22b-2dec-b580d16c0090: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd780fc-ad67-1e29-9b6a-98ca814983cd Address:ecd780fc-ad67-1e29-9b6a-98ca814983cd} {Suffrage:Voter ID:server-00d335fc-0932-dc54-0cb2-ac56d54eaef8 Address:00d335fc-0932-dc54-0cb2-ac56d54eaef8} {Suffrage:Voter ID:server-814741b9-ece1-e22b-2dec-b580d16c0090 Address:814741b9-ece1-e22b-2dec-b580d16c0090}]"
2025-07-14T05:38:43.184Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:43.184Z [INFO]  server-814741b9-ece1-e22b-2dec-b580d16c0090: entering follower state: follower="Node at 814741b9-ece1-e22b-2dec-b580d16c0090 [Follower]" leader-address= leader-id=
2025-07-14T05:38:43.726Z [WARN]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:43.726Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: entering candidate state: node="Node at 00d335fc-0932-dc54-0cb2-ac56d54eaef8 [Candidate]" term=2
2025-07-14T05:38:43.726Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: election won: term=2 tally=2
2025-07-14T05:38:43.726Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: entering leader state: leader="Node at 00d335fc-0932-dc54-0cb2-ac56d54eaef8 [Leader]"
2025-07-14T05:38:43.726Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: added peer, starting replication: peer=server-ecd780fc-ad67-1e29-9b6a-98ca814983cd
2025-07-14T05:38:43.726Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: added peer, starting replication: peer=server-814741b9-ece1-e22b-2dec-b580d16c0090
2025-07-14T05:38:43.726Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: pipelining replication: peer="{Voter server-ecd780fc-ad67-1e29-9b6a-98ca814983cd ecd780fc-ad67-1e29-9b6a-98ca814983cd}"
2025-07-14T05:38:43.726Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: pipelining replication: peer="{Voter server-814741b9-ece1-e22b-2dec-b580d16c0090 814741b9-ece1-e22b-2dec-b580d16c0090}"
2025-07-14T05:38:44.731Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:43.726608443 +0000 UTC m=+23.141068831 (1 nodes), 542.467471ms from start of poll, 542.867662ms from cluster start. Timeout at 2025-07-14 05:38:44.731616064 +0000 UTC m=+24.146076462, 1.005007631s after stability
2025-07-14T05:38:44.732Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: starting snapshot up to: index=12
2025-07-14T05:38:44.732Z [INFO]  creating new snapshot: path=/tmp/raft736293471/snapshots/2-12-1752471524732.tmp
2025-07-14T05:38:44.732Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: snapshot complete up to: index=12
2025-07-14T05:38:44.733Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-242dc216-5599-8249-9d52-e94f139317dd Address:242dc216-5599-8249-9d52-e94f139317dd} {Suffrage:Voter ID:server-71d68104-a7da-5a4a-0361-9ee50c00bfd1 Address:71d68104-a7da-5a4a-0361-9ee50c00bfd1} {Suffrage:Voter ID:server-7c80eeff-9cbd-ba3c-ade7-215118333653 Address:7c80eeff-9cbd-ba3c-ade7-215118333653}]"
2025-07-14T05:38:44.733Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-242dc216-5599-8249-9d52-e94f139317dd Address:242dc216-5599-8249-9d52-e94f139317dd} {Suffrage:Voter ID:server-71d68104-a7da-5a4a-0361-9ee50c00bfd1 Address:71d68104-a7da-5a4a-0361-9ee50c00bfd1} {Suffrage:Voter ID:server-7c80eeff-9cbd-ba3c-ade7-215118333653 Address:7c80eeff-9cbd-ba3c-ade7-215118333653}]"
2025-07-14T05:38:44.733Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-242dc216-5599-8249-9d52-e94f139317dd Address:242dc216-5599-8249-9d52-e94f139317dd} {Suffrage:Voter ID:server-71d68104-a7da-5a4a-0361-9ee50c00bfd1 Address:71d68104-a7da-5a4a-0361-9ee50c00bfd1} {Suffrage:Voter ID:server-7c80eeff-9cbd-ba3c-ade7-215118333653 Address:7c80eeff-9cbd-ba3c-ade7-215118333653}]"
2025-07-14T05:38:44.733Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:44.733Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: entering follower state: follower="Node at 242dc216-5599-8249-9d52-e94f139317dd [Follower]" leader-address= leader-id=
2025-07-14T05:38:44.733Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: entering follower state: follower="Node at 71d68104-a7da-5a4a-0361-9ee50c00bfd1 [Follower]" leader-address= leader-id=
2025-07-14T05:38:44.733Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: entering follower state: follower="Node at 7c80eeff-9cbd-ba3c-ade7-215118333653 [Follower]" leader-address= leader-id=
2025-07-14T05:38:45.544Z [WARN]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:45.544Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: entering candidate state: node="Node at 71d68104-a7da-5a4a-0361-9ee50c00bfd1 [Candidate]" term=2
2025-07-14T05:38:45.545Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: election won: term=2 tally=2
2025-07-14T05:38:45.545Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: entering leader state: leader="Node at 71d68104-a7da-5a4a-0361-9ee50c00bfd1 [Leader]"
2025-07-14T05:38:45.545Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: added peer, starting replication: peer=server-242dc216-5599-8249-9d52-e94f139317dd
2025-07-14T05:38:45.545Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: added peer, starting replication: peer=server-7c80eeff-9cbd-ba3c-ade7-215118333653
2025-07-14T05:38:45.545Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: pipelining replication: peer="{Voter server-7c80eeff-9cbd-ba3c-ade7-215118333653 7c80eeff-9cbd-ba3c-ade7-215118333653}"
2025-07-14T05:38:45.545Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: pipelining replication: peer="{Voter server-242dc216-5599-8249-9d52-e94f139317dd 242dc216-5599-8249-9d52-e94f139317dd}"
2025-07-14T05:38:46.550Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:45.545105758 +0000 UTC m=+24.959566136 (1 nodes), 811.298197ms from start of poll, 811.574461ms from cluster start. Timeout at 2025-07-14 05:38:46.550252108 +0000 UTC m=+25.964712476, 1.00514634s after stability
2025-07-14T05:38:46.550Z [INFO]  creating new snapshot: path=/tmp/raft1403035133/snapshots/2-13-1752471526550.tmp
2025-07-14T05:38:46.550Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: copied to local snapshot: bytes=71
2025-07-14T05:38:46.551Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: snapshot restore progress: id=2-13-1752471526550 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:46.551Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: removing all old logs from log store
2025-07-14T05:38:46.551Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: compacting logs: from=1 to=2
2025-07-14T05:38:46.551Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: restored user snapshot: index=13
2025-07-14T05:38:46.551Z [ERROR] server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: failed to get log: index=2 error="log not found"
2025-07-14T05:38:46.551Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: aborting pipeline replication: peer="{Voter server-7c80eeff-9cbd-ba3c-ade7-215118333653 7c80eeff-9cbd-ba3c-ade7-215118333653}"
2025-07-14T05:38:46.551Z [ERROR] server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: failed to get log: index=2 error="log not found"
2025-07-14T05:38:46.551Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: aborting pipeline replication: peer="{Voter server-242dc216-5599-8249-9d52-e94f139317dd 242dc216-5599-8249-9d52-e94f139317dd}"
2025-07-14T05:38:46.558Z [ERROR] server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: failed to get log: index=2 error="log not found"
2025-07-14T05:38:46.559Z [INFO]  creating new snapshot: path=/tmp/raft789728190/snapshots/2-13-1752471526559.tmp
2025-07-14T05:38:46.559Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:46.559Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: copied to local snapshot: bytes=71
2025-07-14T05:38:46.559Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: snapshot restore progress: id=2-13-1752471526559 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:46.559Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: removing all old logs from log store
2025-07-14T05:38:46.559Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: compacting logs: from=1 to=2
2025-07-14T05:38:46.559Z [INFO]  server-7c80eeff-9cbd-ba3c-ade7-215118333653: Installed remote snapshot
2025-07-14T05:38:46.559Z [ERROR] server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: failed to get log: index=2 error="log not found"
2025-07-14T05:38:46.559Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: pipelining replication: peer="{Voter server-7c80eeff-9cbd-ba3c-ade7-215118333653 7c80eeff-9cbd-ba3c-ade7-215118333653}"
2025-07-14T05:38:46.559Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:46.560Z [INFO]  creating new snapshot: path=/tmp/raft1538195102/snapshots/2-13-1752471526560.tmp
2025-07-14T05:38:46.560Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:46.560Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: copied to local snapshot: bytes=71
2025-07-14T05:38:46.560Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: snapshot restore progress: id=2-13-1752471526560 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:46.560Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: removing all old logs from log store
2025-07-14T05:38:46.560Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: compacting logs: from=1 to=2
2025-07-14T05:38:46.560Z [INFO]  server-242dc216-5599-8249-9d52-e94f139317dd: Installed remote snapshot
2025-07-14T05:38:46.560Z [INFO]  server-71d68104-a7da-5a4a-0361-9ee50c00bfd1: pipelining replication: peer="{Voter server-242dc216-5599-8249-9d52-e94f139317dd 242dc216-5599-8249-9d52-e94f139317dd}"
2025-07-14T05:38:47.565Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:46.559927601 +0000 UTC m=+25.974387979 (2 nodes), 921ns from start of poll, 1.826396304s from cluster start. Timeout at 2025-07-14 05:38:47.565108752 +0000 UTC m=+26.979569121, 1.005181142s after stability
2025-07-14T05:38:47.576Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: aborting pipeline replication: peer="{Voter server-814741b9-ece1-e22b-2dec-b580d16c0090 814741b9-ece1-e22b-2dec-b580d16c0090}"
2025-07-14T05:38:47.576Z [INFO]  server-00d335fc-0932-dc54-0cb2-ac56d54eaef8: aborting pipeline replication: peer="{Voter server-ecd780fc-ad67-1e29-9b6a-98ca814983cd ecd780fc-ad67-1e29-9b6a-98ca814983cd}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-07-14T05:38:47.577Z [INFO]  server-3c753a94-1bd7-0b14-ee75-01bd190864d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c753a94-1bd7-0b14-ee75-01bd190864d0 Address:3c753a94-1bd7-0b14-ee75-01bd190864d0} {Suffrage:Voter ID:server-01716046-7d52-62c8-1445-b4b6ed92d3dc Address:01716046-7d52-62c8-1445-b4b6ed92d3dc} {Suffrage:Voter ID:server-dad66121-dabb-9572-1ca2-58cc62613ac7 Address:dad66121-dabb-9572-1ca2-58cc62613ac7}]"
2025-07-14T05:38:47.577Z [INFO]  server-3c753a94-1bd7-0b14-ee75-01bd190864d0: entering follower state: follower="Node at 3c753a94-1bd7-0b14-ee75-01bd190864d0 [Follower]" leader-address= leader-id=
2025-07-14T05:38:47.577Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c753a94-1bd7-0b14-ee75-01bd190864d0 Address:3c753a94-1bd7-0b14-ee75-01bd190864d0} {Suffrage:Voter ID:server-01716046-7d52-62c8-1445-b4b6ed92d3dc Address:01716046-7d52-62c8-1445-b4b6ed92d3dc} {Suffrage:Voter ID:server-dad66121-dabb-9572-1ca2-58cc62613ac7 Address:dad66121-dabb-9572-1ca2-58cc62613ac7}]"
2025-07-14T05:38:47.577Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: entering follower state: follower="Node at 01716046-7d52-62c8-1445-b4b6ed92d3dc [Follower]" leader-address= leader-id=
2025-07-14T05:38:47.577Z [INFO]  server-dad66121-dabb-9572-1ca2-58cc62613ac7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c753a94-1bd7-0b14-ee75-01bd190864d0 Address:3c753a94-1bd7-0b14-ee75-01bd190864d0} {Suffrage:Voter ID:server-01716046-7d52-62c8-1445-b4b6ed92d3dc Address:01716046-7d52-62c8-1445-b4b6ed92d3dc} {Suffrage:Voter ID:server-dad66121-dabb-9572-1ca2-58cc62613ac7 Address:dad66121-dabb-9572-1ca2-58cc62613ac7}]"
2025-07-14T05:38:47.577Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:47.577Z [INFO]  server-dad66121-dabb-9572-1ca2-58cc62613ac7: entering follower state: follower="Node at dad66121-dabb-9572-1ca2-58cc62613ac7 [Follower]" leader-address= leader-id=
2025-07-14T05:38:48.218Z [WARN]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:48.218Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: entering candidate state: node="Node at 01716046-7d52-62c8-1445-b4b6ed92d3dc [Candidate]" term=2
2025-07-14T05:38:48.218Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: election won: term=2 tally=2
2025-07-14T05:38:48.218Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: entering leader state: leader="Node at 01716046-7d52-62c8-1445-b4b6ed92d3dc [Leader]"
2025-07-14T05:38:48.218Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: added peer, starting replication: peer=server-3c753a94-1bd7-0b14-ee75-01bd190864d0
2025-07-14T05:38:48.218Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: added peer, starting replication: peer=server-dad66121-dabb-9572-1ca2-58cc62613ac7
2025-07-14T05:38:48.218Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: pipelining replication: peer="{Voter server-dad66121-dabb-9572-1ca2-58cc62613ac7 dad66121-dabb-9572-1ca2-58cc62613ac7}"
2025-07-14T05:38:48.218Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: pipelining replication: peer="{Voter server-3c753a94-1bd7-0b14-ee75-01bd190864d0 3c753a94-1bd7-0b14-ee75-01bd190864d0}"
2025-07-14T05:38:49.224Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:48.218383254 +0000 UTC m=+27.632843632 (1 nodes), 640.874111ms from start of poll, 641.132518ms from cluster start. Timeout at 2025-07-14 05:38:49.224321121 +0000 UTC m=+28.638781509, 1.005937877s after stability
2025-07-14T05:38:49.224Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: starting snapshot up to: index=12
2025-07-14T05:38:49.224Z [INFO]  creating new snapshot: path=/tmp/raft3682010802/snapshots/2-12-1752471529224.tmp
2025-07-14T05:38:49.243Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: snapshot complete up to: index=12
2025-07-14T05:38:49.243Z [INFO]  creating new snapshot: path=/tmp/raft3682010802/snapshots/2-23-1752471529243.tmp
2025-07-14T05:38:49.406Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: copied to local snapshot: bytes=71
2025-07-14T05:38:49.406Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: snapshot restore progress: id=2-23-1752471529243 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:49.406Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: restored user snapshot: index=23
2025-07-14T05:38:49.406Z [ERROR] server-01716046-7d52-62c8-1445-b4b6ed92d3dc: failed to get log: index=23 error="log not found"
2025-07-14T05:38:49.406Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: aborting pipeline replication: peer="{Voter server-dad66121-dabb-9572-1ca2-58cc62613ac7 dad66121-dabb-9572-1ca2-58cc62613ac7}"
2025-07-14T05:38:49.406Z [ERROR] server-01716046-7d52-62c8-1445-b4b6ed92d3dc: failed to get log: index=23 error="log not found"
2025-07-14T05:38:49.406Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: aborting pipeline replication: peer="{Voter server-3c753a94-1bd7-0b14-ee75-01bd190864d0 3c753a94-1bd7-0b14-ee75-01bd190864d0}"
2025-07-14T05:38:49.412Z [ERROR] server-01716046-7d52-62c8-1445-b4b6ed92d3dc: failed to get log: index=23 error="log not found"
2025-07-14T05:38:49.413Z [INFO]  creating new snapshot: path=/tmp/raft2391842783/snapshots/2-23-1752471529413.tmp
2025-07-14T05:38:49.413Z [INFO]  server-dad66121-dabb-9572-1ca2-58cc62613ac7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:49.415Z [ERROR] server-01716046-7d52-62c8-1445-b4b6ed92d3dc: failed to get log: index=23 error="log not found"
2025-07-14T05:38:49.415Z [INFO]  creating new snapshot: path=/tmp/raft425618078/snapshots/2-23-1752471529415.tmp
2025-07-14T05:38:49.415Z [INFO]  server-3c753a94-1bd7-0b14-ee75-01bd190864d0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:49.599Z [INFO]  server-3c753a94-1bd7-0b14-ee75-01bd190864d0: copied to local snapshot: bytes=71
2025-07-14T05:38:49.599Z [INFO]  server-dad66121-dabb-9572-1ca2-58cc62613ac7: copied to local snapshot: bytes=71
2025-07-14T05:38:49.599Z [INFO]  server-dad66121-dabb-9572-1ca2-58cc62613ac7: snapshot restore progress: id=2-23-1752471529413 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:49.599Z [INFO]  server-dad66121-dabb-9572-1ca2-58cc62613ac7: Installed remote snapshot
2025-07-14T05:38:49.599Z [INFO]  server-3c753a94-1bd7-0b14-ee75-01bd190864d0: snapshot restore progress: id=2-23-1752471529415 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:49.599Z [INFO]  server-3c753a94-1bd7-0b14-ee75-01bd190864d0: Installed remote snapshot
2025-07-14T05:38:49.599Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: pipelining replication: peer="{Voter server-dad66121-dabb-9572-1ca2-58cc62613ac7 dad66121-dabb-9572-1ca2-58cc62613ac7}"
2025-07-14T05:38:49.599Z [WARN]  server-3c753a94-1bd7-0b14-ee75-01bd190864d0: skipping application of old log: index=23
2025-07-14T05:38:49.599Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:49.599Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: pipelining replication: peer="{Voter server-3c753a94-1bd7-0b14-ee75-01bd190864d0 3c753a94-1bd7-0b14-ee75-01bd190864d0}"
2025-07-14T05:38:50.605Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:49.599758276 +0000 UTC m=+29.014218655 (2 nodes), 1.032µs from start of poll, 2.022507541s from cluster start. Timeout at 2025-07-14 05:38:50.60563385 +0000 UTC m=+30.020094238, 1.005875583s after stability
2025-07-14T05:38:50.617Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: aborting pipeline replication: peer="{Voter server-dad66121-dabb-9572-1ca2-58cc62613ac7 dad66121-dabb-9572-1ca2-58cc62613ac7}"
2025-07-14T05:38:50.617Z [INFO]  server-01716046-7d52-62c8-1445-b4b6ed92d3dc: aborting pipeline replication: peer="{Voter server-3c753a94-1bd7-0b14-ee75-01bd190864d0 3c753a94-1bd7-0b14-ee75-01bd190864d0}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-07-14T05:38:50.618Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-328dfba3-3047-af31-cca3-d84d2b41e3d9 Address:328dfba3-3047-af31-cca3-d84d2b41e3d9} {Suffrage:Voter ID:server-b7c64f3a-8551-1345-1e93-ce0643c161d3 Address:b7c64f3a-8551-1345-1e93-ce0643c161d3} {Suffrage:Voter ID:server-f7199126-5697-9d8f-35e9-4b850415f63b Address:f7199126-5697-9d8f-35e9-4b850415f63b}]"
2025-07-14T05:38:50.618Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: entering follower state: follower="Node at 328dfba3-3047-af31-cca3-d84d2b41e3d9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:50.618Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-328dfba3-3047-af31-cca3-d84d2b41e3d9 Address:328dfba3-3047-af31-cca3-d84d2b41e3d9} {Suffrage:Voter ID:server-b7c64f3a-8551-1345-1e93-ce0643c161d3 Address:b7c64f3a-8551-1345-1e93-ce0643c161d3} {Suffrage:Voter ID:server-f7199126-5697-9d8f-35e9-4b850415f63b Address:f7199126-5697-9d8f-35e9-4b850415f63b}]"
2025-07-14T05:38:50.618Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: entering follower state: follower="Node at b7c64f3a-8551-1345-1e93-ce0643c161d3 [Follower]" leader-address= leader-id=
2025-07-14T05:38:50.618Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-328dfba3-3047-af31-cca3-d84d2b41e3d9 Address:328dfba3-3047-af31-cca3-d84d2b41e3d9} {Suffrage:Voter ID:server-b7c64f3a-8551-1345-1e93-ce0643c161d3 Address:b7c64f3a-8551-1345-1e93-ce0643c161d3} {Suffrage:Voter ID:server-f7199126-5697-9d8f-35e9-4b850415f63b Address:f7199126-5697-9d8f-35e9-4b850415f63b}]"
2025-07-14T05:38:50.618Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:50.618Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: entering follower state: follower="Node at f7199126-5697-9d8f-35e9-4b850415f63b [Follower]" leader-address= leader-id=
2025-07-14T05:38:51.447Z [WARN]  server-f7199126-5697-9d8f-35e9-4b850415f63b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:51.447Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: entering candidate state: node="Node at f7199126-5697-9d8f-35e9-4b850415f63b [Candidate]" term=2
2025-07-14T05:38:51.447Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: election won: term=2 tally=2
2025-07-14T05:38:51.447Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: entering leader state: leader="Node at f7199126-5697-9d8f-35e9-4b850415f63b [Leader]"
2025-07-14T05:38:51.447Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: added peer, starting replication: peer=server-328dfba3-3047-af31-cca3-d84d2b41e3d9
2025-07-14T05:38:51.447Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: added peer, starting replication: peer=server-b7c64f3a-8551-1345-1e93-ce0643c161d3
2025-07-14T05:38:51.447Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: pipelining replication: peer="{Voter server-b7c64f3a-8551-1345-1e93-ce0643c161d3 b7c64f3a-8551-1345-1e93-ce0643c161d3}"
2025-07-14T05:38:51.447Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: pipelining replication: peer="{Voter server-328dfba3-3047-af31-cca3-d84d2b41e3d9 328dfba3-3047-af31-cca3-d84d2b41e3d9}"
2025-07-14T05:38:52.452Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:51.447301161 +0000 UTC m=+30.861761539 (1 nodes), 828.538909ms from start of poll, 828.808523ms from cluster start. Timeout at 2025-07-14 05:38:52.452429633 +0000 UTC m=+31.866890031, 1.005128492s after stability
2025-07-14T05:38:52.452Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: starting snapshot up to: index=12
2025-07-14T05:38:52.452Z [INFO]  creating new snapshot: path=/tmp/raft609381254/snapshots/2-12-1752471532452.tmp
2025-07-14T05:38:52.453Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: snapshot complete up to: index=12
2025-07-14T05:38:52.454Z [INFO]  creating new snapshot: path=/tmp/raft609381254/snapshots/2-23-1752471532454.tmp
2025-07-14T05:38:52.545Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: copied to local snapshot: bytes=71
2025-07-14T05:38:52.545Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: snapshot restore progress: id=2-23-1752471532454 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:52.545Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: removing all old logs from log store
2025-07-14T05:38:52.545Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: compacting logs: from=1 to=22
2025-07-14T05:38:52.545Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: restored user snapshot: index=23
2025-07-14T05:38:52.545Z [ERROR] server-f7199126-5697-9d8f-35e9-4b850415f63b: failed to get log: index=22 error="log not found"
2025-07-14T05:38:52.545Z [ERROR] server-f7199126-5697-9d8f-35e9-4b850415f63b: failed to get log: index=22 error="log not found"
2025-07-14T05:38:52.545Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: aborting pipeline replication: peer="{Voter server-328dfba3-3047-af31-cca3-d84d2b41e3d9 328dfba3-3047-af31-cca3-d84d2b41e3d9}"
2025-07-14T05:38:52.545Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: aborting pipeline replication: peer="{Voter server-b7c64f3a-8551-1345-1e93-ce0643c161d3 b7c64f3a-8551-1345-1e93-ce0643c161d3}"
2025-07-14T05:38:52.553Z [ERROR] server-f7199126-5697-9d8f-35e9-4b850415f63b: failed to get log: index=22 error="log not found"
2025-07-14T05:38:52.554Z [INFO]  creating new snapshot: path=/tmp/raft3666286112/snapshots/2-23-1752471532554.tmp
2025-07-14T05:38:52.554Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:52.554Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: copied to local snapshot: bytes=71
2025-07-14T05:38:52.554Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: snapshot restore progress: id=2-23-1752471532554 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:52.554Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: removing all old logs from log store
2025-07-14T05:38:52.554Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: compacting logs: from=1 to=22
2025-07-14T05:38:52.554Z [INFO]  server-328dfba3-3047-af31-cca3-d84d2b41e3d9: Installed remote snapshot
2025-07-14T05:38:52.555Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: pipelining replication: peer="{Voter server-328dfba3-3047-af31-cca3-d84d2b41e3d9 328dfba3-3047-af31-cca3-d84d2b41e3d9}"
2025-07-14T05:38:52.555Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:52.556Z [ERROR] server-f7199126-5697-9d8f-35e9-4b850415f63b: failed to get log: index=22 error="log not found"
2025-07-14T05:38:52.556Z [INFO]  creating new snapshot: path=/tmp/raft1466369256/snapshots/2-23-1752471532556.tmp
2025-07-14T05:38:52.556Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:52.620Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: copied to local snapshot: bytes=71
2025-07-14T05:38:52.620Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: snapshot restore progress: id=2-23-1752471532556 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:52.620Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: removing all old logs from log store
2025-07-14T05:38:52.620Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: compacting logs: from=1 to=22
2025-07-14T05:38:52.620Z [INFO]  server-b7c64f3a-8551-1345-1e93-ce0643c161d3: Installed remote snapshot
2025-07-14T05:38:52.620Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: pipelining replication: peer="{Voter server-b7c64f3a-8551-1345-1e93-ce0643c161d3 b7c64f3a-8551-1345-1e93-ce0643c161d3}"
2025-07-14T05:38:53.560Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:52.555064525 +0000 UTC m=+31.969524893 (2 nodes), 801ns from start of poll, 1.936571877s from cluster start. Timeout at 2025-07-14 05:38:53.560639708 +0000 UTC m=+32.975100076, 1.005575183s after stability
2025-07-14T05:38:53.572Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: aborting pipeline replication: peer="{Voter server-b7c64f3a-8551-1345-1e93-ce0643c161d3 b7c64f3a-8551-1345-1e93-ce0643c161d3}"
2025-07-14T05:38:53.572Z [INFO]  server-f7199126-5697-9d8f-35e9-4b850415f63b: aborting pipeline replication: peer="{Voter server-328dfba3-3047-af31-cca3-d84d2b41e3d9 328dfba3-3047-af31-cca3-d84d2b41e3d9}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-07-14T05:38:53.573Z [INFO]  server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c Address:8de7e6ae-fa6f-db32-54ac-3ccab1c4238c} {Suffrage:Voter ID:server-bcf97d72-3747-1f0c-c01b-524b2611f2d4 Address:bcf97d72-3747-1f0c-c01b-524b2611f2d4} {Suffrage:Voter ID:server-0a6c6be7-294e-b565-6b22-c954d0041e8e Address:0a6c6be7-294e-b565-6b22-c954d0041e8e}]"
2025-07-14T05:38:53.573Z [INFO]  server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c: entering follower state: follower="Node at 8de7e6ae-fa6f-db32-54ac-3ccab1c4238c [Follower]" leader-address= leader-id=
2025-07-14T05:38:53.573Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c Address:8de7e6ae-fa6f-db32-54ac-3ccab1c4238c} {Suffrage:Voter ID:server-bcf97d72-3747-1f0c-c01b-524b2611f2d4 Address:bcf97d72-3747-1f0c-c01b-524b2611f2d4} {Suffrage:Voter ID:server-0a6c6be7-294e-b565-6b22-c954d0041e8e Address:0a6c6be7-294e-b565-6b22-c954d0041e8e}]"
2025-07-14T05:38:53.573Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: entering follower state: follower="Node at bcf97d72-3747-1f0c-c01b-524b2611f2d4 [Follower]" leader-address= leader-id=
2025-07-14T05:38:53.573Z [INFO]  server-0a6c6be7-294e-b565-6b22-c954d0041e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c Address:8de7e6ae-fa6f-db32-54ac-3ccab1c4238c} {Suffrage:Voter ID:server-bcf97d72-3747-1f0c-c01b-524b2611f2d4 Address:bcf97d72-3747-1f0c-c01b-524b2611f2d4} {Suffrage:Voter ID:server-0a6c6be7-294e-b565-6b22-c954d0041e8e Address:0a6c6be7-294e-b565-6b22-c954d0041e8e}]"
2025-07-14T05:38:53.573Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:53.573Z [INFO]  server-0a6c6be7-294e-b565-6b22-c954d0041e8e: entering follower state: follower="Node at 0a6c6be7-294e-b565-6b22-c954d0041e8e [Follower]" leader-address= leader-id=
2025-07-14T05:38:54.095Z [WARN]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:54.095Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: entering candidate state: node="Node at bcf97d72-3747-1f0c-c01b-524b2611f2d4 [Candidate]" term=2
2025-07-14T05:38:54.095Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: election won: term=2 tally=2
2025-07-14T05:38:54.095Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: entering leader state: leader="Node at bcf97d72-3747-1f0c-c01b-524b2611f2d4 [Leader]"
2025-07-14T05:38:54.095Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: added peer, starting replication: peer=server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c
2025-07-14T05:38:54.095Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: added peer, starting replication: peer=server-0a6c6be7-294e-b565-6b22-c954d0041e8e
2025-07-14T05:38:54.095Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: pipelining replication: peer="{Voter server-0a6c6be7-294e-b565-6b22-c954d0041e8e 0a6c6be7-294e-b565-6b22-c954d0041e8e}"
2025-07-14T05:38:54.095Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: pipelining replication: peer="{Voter server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c 8de7e6ae-fa6f-db32-54ac-3ccab1c4238c}"
2025-07-14T05:38:55.100Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:54.095163654 +0000 UTC m=+33.509624052 (1 nodes), 521.163665ms from start of poll, 521.503926ms from cluster start. Timeout at 2025-07-14 05:38:55.100427059 +0000 UTC m=+34.514887437, 1.005263385s after stability
2025-07-14T05:38:55.100Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: starting snapshot up to: index=12
2025-07-14T05:38:55.100Z [INFO]  creating new snapshot: path=/tmp/raft745507446/snapshots/2-12-1752471535100.tmp
2025-07-14T05:38:55.201Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: snapshot complete up to: index=12
2025-07-14T05:38:55.203Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cef33621-d649-1cd1-546f-f9a97b1cff4d Address:cef33621-d649-1cd1-546f-f9a97b1cff4d} {Suffrage:Voter ID:server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9 Address:3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9} {Suffrage:Voter ID:server-e165e8af-a2a0-57c1-3a13-8ecf451652eb Address:e165e8af-a2a0-57c1-3a13-8ecf451652eb}]"
2025-07-14T05:38:55.203Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: entering follower state: follower="Node at cef33621-d649-1cd1-546f-f9a97b1cff4d [Follower]" leader-address= leader-id=
2025-07-14T05:38:55.203Z [INFO]  server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cef33621-d649-1cd1-546f-f9a97b1cff4d Address:cef33621-d649-1cd1-546f-f9a97b1cff4d} {Suffrage:Voter ID:server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9 Address:3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9} {Suffrage:Voter ID:server-e165e8af-a2a0-57c1-3a13-8ecf451652eb Address:e165e8af-a2a0-57c1-3a13-8ecf451652eb}]"
2025-07-14T05:38:55.203Z [INFO]  server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9: entering follower state: follower="Node at 3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9 [Follower]" leader-address= leader-id=
2025-07-14T05:38:55.203Z [INFO]  server-e165e8af-a2a0-57c1-3a13-8ecf451652eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cef33621-d649-1cd1-546f-f9a97b1cff4d Address:cef33621-d649-1cd1-546f-f9a97b1cff4d} {Suffrage:Voter ID:server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9 Address:3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9} {Suffrage:Voter ID:server-e165e8af-a2a0-57c1-3a13-8ecf451652eb Address:e165e8af-a2a0-57c1-3a13-8ecf451652eb}]"
2025-07-14T05:38:55.203Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:55.203Z [INFO]  server-e165e8af-a2a0-57c1-3a13-8ecf451652eb: entering follower state: follower="Node at e165e8af-a2a0-57c1-3a13-8ecf451652eb [Follower]" leader-address= leader-id=
2025-07-14T05:38:55.854Z [WARN]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:55.854Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: entering candidate state: node="Node at cef33621-d649-1cd1-546f-f9a97b1cff4d [Candidate]" term=2
2025-07-14T05:38:55.854Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: election won: term=2 tally=2
2025-07-14T05:38:55.854Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: entering leader state: leader="Node at cef33621-d649-1cd1-546f-f9a97b1cff4d [Leader]"
2025-07-14T05:38:55.854Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: added peer, starting replication: peer=server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9
2025-07-14T05:38:55.854Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: added peer, starting replication: peer=server-e165e8af-a2a0-57c1-3a13-8ecf451652eb
2025-07-14T05:38:55.854Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: pipelining replication: peer="{Voter server-e165e8af-a2a0-57c1-3a13-8ecf451652eb e165e8af-a2a0-57c1-3a13-8ecf451652eb}"
2025-07-14T05:38:55.855Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: pipelining replication: peer="{Voter server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9 3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9}"
2025-07-14T05:38:56.860Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:55.85496126 +0000 UTC m=+35.269421648 (1 nodes), 651.298197ms from start of poll, 651.660891ms from cluster start. Timeout at 2025-07-14 05:38:56.860109883 +0000 UTC m=+36.274570261, 1.005148613s after stability
2025-07-14T05:38:56.860Z [INFO]  creating new snapshot: path=/tmp/raft28167544/snapshots/2-14-1752471536860.tmp
2025-07-14T05:38:56.860Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: copied to local snapshot: bytes=71
2025-07-14T05:38:56.860Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: snapshot restore progress: id=2-14-1752471536860 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:56.860Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: restored user snapshot: index=14
2025-07-14T05:38:56.860Z [ERROR] server-cef33621-d649-1cd1-546f-f9a97b1cff4d: failed to get log: index=3 error="log not found"
2025-07-14T05:38:56.860Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: aborting pipeline replication: peer="{Voter server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9 3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9}"
2025-07-14T05:38:56.860Z [ERROR] server-cef33621-d649-1cd1-546f-f9a97b1cff4d: failed to get log: index=3 error="log not found"
2025-07-14T05:38:56.860Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: aborting pipeline replication: peer="{Voter server-e165e8af-a2a0-57c1-3a13-8ecf451652eb e165e8af-a2a0-57c1-3a13-8ecf451652eb}"
2025-07-14T05:38:56.868Z [ERROR] server-cef33621-d649-1cd1-546f-f9a97b1cff4d: failed to get log: index=3 error="log not found"
2025-07-14T05:38:56.869Z [INFO]  creating new snapshot: path=/tmp/raft2659091796/snapshots/2-14-1752471536869.tmp
2025-07-14T05:38:56.869Z [INFO]  server-e165e8af-a2a0-57c1-3a13-8ecf451652eb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:56.869Z [INFO]  server-e165e8af-a2a0-57c1-3a13-8ecf451652eb: copied to local snapshot: bytes=71
2025-07-14T05:38:56.869Z [INFO]  server-e165e8af-a2a0-57c1-3a13-8ecf451652eb: snapshot restore progress: id=2-14-1752471536869 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:56.869Z [INFO]  server-e165e8af-a2a0-57c1-3a13-8ecf451652eb: Installed remote snapshot
2025-07-14T05:38:56.869Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: pipelining replication: peer="{Voter server-e165e8af-a2a0-57c1-3a13-8ecf451652eb e165e8af-a2a0-57c1-3a13-8ecf451652eb}"
2025-07-14T05:38:56.869Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:38:56.870Z [ERROR] server-cef33621-d649-1cd1-546f-f9a97b1cff4d: failed to get log: index=3 error="log not found"
2025-07-14T05:38:56.871Z [INFO]  creating new snapshot: path=/tmp/raft372034310/snapshots/2-14-1752471536871.tmp
2025-07-14T05:38:56.871Z [INFO]  server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:56.871Z [INFO]  server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9: copied to local snapshot: bytes=71
2025-07-14T05:38:56.871Z [INFO]  server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9: snapshot restore progress: id=2-14-1752471536871 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:38:56.871Z [INFO]  server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9: Installed remote snapshot
2025-07-14T05:38:56.871Z [INFO]  server-cef33621-d649-1cd1-546f-f9a97b1cff4d: pipelining replication: peer="{Voter server-3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9 3ea3cdcc-7b26-b81e-c0f0-197c7e5cd0d9}"
2025-07-14T05:38:57.875Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:38:56.869717745 +0000 UTC m=+36.284178113 (2 nodes), 681ns from start of poll, 1.666417356s from cluster start. Timeout at 2025-07-14 05:38:57.875791435 +0000 UTC m=+37.290251823, 1.00607371s after stability
2025-07-14T05:38:57.887Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: aborting pipeline replication: peer="{Voter server-0a6c6be7-294e-b565-6b22-c954d0041e8e 0a6c6be7-294e-b565-6b22-c954d0041e8e}"
2025-07-14T05:38:57.887Z [INFO]  server-bcf97d72-3747-1f0c-c01b-524b2611f2d4: aborting pipeline replication: peer="{Voter server-8de7e6ae-fa6f-db32-54ac-3ccab1c4238c 8de7e6ae-fa6f-db32-54ac-3ccab1c4238c}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-07-14T05:38:57.888Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb44b423-cc51-91f4-7302-23d39446e2c7 Address:cb44b423-cc51-91f4-7302-23d39446e2c7} {Suffrage:Voter ID:server-c3089c9d-167c-268b-8f99-92b602545e27 Address:c3089c9d-167c-268b-8f99-92b602545e27} {Suffrage:Voter ID:server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2 Address:4eb6890c-bff5-9bfd-4a6d-e782a4d820a2}]"
2025-07-14T05:38:57.888Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: entering follower state: follower="Node at cb44b423-cc51-91f4-7302-23d39446e2c7 [Follower]" leader-address= leader-id=
2025-07-14T05:38:57.888Z [INFO]  server-c3089c9d-167c-268b-8f99-92b602545e27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb44b423-cc51-91f4-7302-23d39446e2c7 Address:cb44b423-cc51-91f4-7302-23d39446e2c7} {Suffrage:Voter ID:server-c3089c9d-167c-268b-8f99-92b602545e27 Address:c3089c9d-167c-268b-8f99-92b602545e27} {Suffrage:Voter ID:server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2 Address:4eb6890c-bff5-9bfd-4a6d-e782a4d820a2}]"
2025-07-14T05:38:57.888Z [INFO]  server-c3089c9d-167c-268b-8f99-92b602545e27: entering follower state: follower="Node at c3089c9d-167c-268b-8f99-92b602545e27 [Follower]" leader-address= leader-id=
2025-07-14T05:38:57.888Z [INFO]  server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb44b423-cc51-91f4-7302-23d39446e2c7 Address:cb44b423-cc51-91f4-7302-23d39446e2c7} {Suffrage:Voter ID:server-c3089c9d-167c-268b-8f99-92b602545e27 Address:c3089c9d-167c-268b-8f99-92b602545e27} {Suffrage:Voter ID:server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2 Address:4eb6890c-bff5-9bfd-4a6d-e782a4d820a2}]"
2025-07-14T05:38:57.888Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:57.888Z [INFO]  server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2: entering follower state: follower="Node at 4eb6890c-bff5-9bfd-4a6d-e782a4d820a2 [Follower]" leader-address= leader-id=
2025-07-14T05:38:58.799Z [WARN]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:38:58.799Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: entering candidate state: node="Node at cb44b423-cc51-91f4-7302-23d39446e2c7 [Candidate]" term=2
2025-07-14T05:38:58.799Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: election won: term=2 tally=2
2025-07-14T05:38:58.799Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: entering leader state: leader="Node at cb44b423-cc51-91f4-7302-23d39446e2c7 [Leader]"
2025-07-14T05:38:58.799Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: added peer, starting replication: peer=server-c3089c9d-167c-268b-8f99-92b602545e27
2025-07-14T05:38:58.799Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: added peer, starting replication: peer=server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2
2025-07-14T05:38:58.799Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: pipelining replication: peer="{Voter server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2 4eb6890c-bff5-9bfd-4a6d-e782a4d820a2}"
2025-07-14T05:38:58.799Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: pipelining replication: peer="{Voter server-c3089c9d-167c-268b-8f99-92b602545e27 c3089c9d-167c-268b-8f99-92b602545e27}"
2025-07-14T05:38:59.804Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:38:58.799495591 +0000 UTC m=+38.213955959 (1 nodes), 910.909374ms from start of poll, 911.182774ms from cluster start. Timeout at 2025-07-14 05:38:59.804533598 +0000 UTC m=+39.218993976, 1.005038017s after stability
2025-07-14T05:38:59.804Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: starting snapshot up to: index=12
2025-07-14T05:38:59.804Z [INFO]  creating new snapshot: path=/tmp/raft2549516977/snapshots/2-12-1752471539804.tmp
2025-07-14T05:38:59.805Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: snapshot complete up to: index=12
2025-07-14T05:38:59.806Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24ff7584-b3bb-364d-7642-cdd175630025 Address:24ff7584-b3bb-364d-7642-cdd175630025} {Suffrage:Voter ID:server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4 Address:6b7aeb86-3b57-2102-5d88-17145ce3b8a4} {Suffrage:Voter ID:server-d412439b-09fb-8a03-f2fd-816404f2b7e3 Address:d412439b-09fb-8a03-f2fd-816404f2b7e3}]"
2025-07-14T05:38:59.806Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: entering follower state: follower="Node at 24ff7584-b3bb-364d-7642-cdd175630025 [Follower]" leader-address= leader-id=
2025-07-14T05:38:59.806Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24ff7584-b3bb-364d-7642-cdd175630025 Address:24ff7584-b3bb-364d-7642-cdd175630025} {Suffrage:Voter ID:server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4 Address:6b7aeb86-3b57-2102-5d88-17145ce3b8a4} {Suffrage:Voter ID:server-d412439b-09fb-8a03-f2fd-816404f2b7e3 Address:d412439b-09fb-8a03-f2fd-816404f2b7e3}]"
2025-07-14T05:38:59.806Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: entering follower state: follower="Node at 6b7aeb86-3b57-2102-5d88-17145ce3b8a4 [Follower]" leader-address= leader-id=
2025-07-14T05:38:59.806Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24ff7584-b3bb-364d-7642-cdd175630025 Address:24ff7584-b3bb-364d-7642-cdd175630025} {Suffrage:Voter ID:server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4 Address:6b7aeb86-3b57-2102-5d88-17145ce3b8a4} {Suffrage:Voter ID:server-d412439b-09fb-8a03-f2fd-816404f2b7e3 Address:d412439b-09fb-8a03-f2fd-816404f2b7e3}]"
2025-07-14T05:38:59.806Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:38:59.806Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: entering follower state: follower="Node at d412439b-09fb-8a03-f2fd-816404f2b7e3 [Follower]" leader-address= leader-id=
2025-07-14T05:39:00.357Z [WARN]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:00.357Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: entering candidate state: node="Node at d412439b-09fb-8a03-f2fd-816404f2b7e3 [Candidate]" term=2
2025-07-14T05:39:00.357Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: election won: term=2 tally=2
2025-07-14T05:39:00.357Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: entering leader state: leader="Node at d412439b-09fb-8a03-f2fd-816404f2b7e3 [Leader]"
2025-07-14T05:39:00.357Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: added peer, starting replication: peer=server-24ff7584-b3bb-364d-7642-cdd175630025
2025-07-14T05:39:00.357Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: added peer, starting replication: peer=server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4
2025-07-14T05:39:00.357Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: pipelining replication: peer="{Voter server-24ff7584-b3bb-364d-7642-cdd175630025 24ff7584-b3bb-364d-7642-cdd175630025}"
2025-07-14T05:39:00.357Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: pipelining replication: peer="{Voter server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4 6b7aeb86-3b57-2102-5d88-17145ce3b8a4}"
2025-07-14T05:39:01.363Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:00.357707809 +0000 UTC m=+39.772168187 (1 nodes), 551.194722ms from start of poll, 551.462413ms from cluster start. Timeout at 2025-07-14 05:39:01.363505826 +0000 UTC m=+40.777966224, 1.005798037s after stability
2025-07-14T05:39:01.363Z [INFO]  creating new snapshot: path=/tmp/raft4028495488/snapshots/2-14-1752471541363.tmp
2025-07-14T05:39:01.364Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: copied to local snapshot: bytes=71
2025-07-14T05:39:01.365Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: snapshot restore progress: id=2-14-1752471541363 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:01.365Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: removing all old logs from log store
2025-07-14T05:39:01.365Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: compacting logs: from=1 to=2
2025-07-14T05:39:01.365Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: restored user snapshot: index=14
2025-07-14T05:39:01.365Z [ERROR] server-d412439b-09fb-8a03-f2fd-816404f2b7e3: failed to get log: index=2 error="log not found"
2025-07-14T05:39:01.365Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: aborting pipeline replication: peer="{Voter server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4 6b7aeb86-3b57-2102-5d88-17145ce3b8a4}"
2025-07-14T05:39:01.365Z [ERROR] server-d412439b-09fb-8a03-f2fd-816404f2b7e3: failed to get log: index=2 error="log not found"
2025-07-14T05:39:01.365Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: aborting pipeline replication: peer="{Voter server-24ff7584-b3bb-364d-7642-cdd175630025 24ff7584-b3bb-364d-7642-cdd175630025}"
2025-07-14T05:39:01.372Z [ERROR] server-d412439b-09fb-8a03-f2fd-816404f2b7e3: failed to get log: index=2 error="log not found"
2025-07-14T05:39:01.373Z [INFO]  creating new snapshot: path=/tmp/raft366824660/snapshots/2-14-1752471541372.tmp
2025-07-14T05:39:01.373Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:01.373Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: copied to local snapshot: bytes=71
2025-07-14T05:39:01.373Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: snapshot restore progress: id=2-14-1752471541372 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:01.373Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: removing all old logs from log store
2025-07-14T05:39:01.373Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: compacting logs: from=1 to=2
2025-07-14T05:39:01.373Z [INFO]  server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4: Installed remote snapshot
2025-07-14T05:39:01.373Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: pipelining replication: peer="{Voter server-6b7aeb86-3b57-2102-5d88-17145ce3b8a4 6b7aeb86-3b57-2102-5d88-17145ce3b8a4}"
2025-07-14T05:39:01.373Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:01.374Z [ERROR] server-d412439b-09fb-8a03-f2fd-816404f2b7e3: failed to get log: index=2 error="log not found"
2025-07-14T05:39:01.375Z [INFO]  creating new snapshot: path=/tmp/raft4149755564/snapshots/2-14-1752471541375.tmp
2025-07-14T05:39:01.375Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:01.375Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: copied to local snapshot: bytes=71
2025-07-14T05:39:01.375Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: snapshot restore progress: id=2-14-1752471541375 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:01.375Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: removing all old logs from log store
2025-07-14T05:39:01.375Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: compacting logs: from=1 to=2
2025-07-14T05:39:01.375Z [INFO]  server-24ff7584-b3bb-364d-7642-cdd175630025: Installed remote snapshot
2025-07-14T05:39:01.375Z [INFO]  server-d412439b-09fb-8a03-f2fd-816404f2b7e3: pipelining replication: peer="{Voter server-24ff7584-b3bb-364d-7642-cdd175630025 24ff7584-b3bb-364d-7642-cdd175630025}"
2025-07-14T05:39:02.379Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:01.373875741 +0000 UTC m=+40.788336119 (2 nodes), 731ns from start of poll, 1.567630345s from cluster start. Timeout at 2025-07-14 05:39:02.379356572 +0000 UTC m=+41.793816960, 1.005480841s after stability
2025-07-14T05:39:02.390Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: aborting pipeline replication: peer="{Voter server-4eb6890c-bff5-9bfd-4a6d-e782a4d820a2 4eb6890c-bff5-9bfd-4a6d-e782a4d820a2}"
2025-07-14T05:39:02.390Z [INFO]  server-cb44b423-cc51-91f4-7302-23d39446e2c7: aborting pipeline replication: peer="{Voter server-c3089c9d-167c-268b-8f99-92b602545e27 c3089c9d-167c-268b-8f99-92b602545e27}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-07-14T05:39:02.391Z [INFO]  server-cdc0d21a-a28e-f474-ecb3-cb024c51789d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdc0d21a-a28e-f474-ecb3-cb024c51789d Address:cdc0d21a-a28e-f474-ecb3-cb024c51789d} {Suffrage:Voter ID:server-c77352f8-0ca1-dfae-063d-ff888eeb6a73 Address:c77352f8-0ca1-dfae-063d-ff888eeb6a73} {Suffrage:Voter ID:server-80391765-a073-f9de-81d0-5c87378677b7 Address:80391765-a073-f9de-81d0-5c87378677b7}]"
2025-07-14T05:39:02.391Z [INFO]  server-cdc0d21a-a28e-f474-ecb3-cb024c51789d: entering follower state: follower="Node at cdc0d21a-a28e-f474-ecb3-cb024c51789d [Follower]" leader-address= leader-id=
2025-07-14T05:39:02.391Z [INFO]  server-c77352f8-0ca1-dfae-063d-ff888eeb6a73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdc0d21a-a28e-f474-ecb3-cb024c51789d Address:cdc0d21a-a28e-f474-ecb3-cb024c51789d} {Suffrage:Voter ID:server-c77352f8-0ca1-dfae-063d-ff888eeb6a73 Address:c77352f8-0ca1-dfae-063d-ff888eeb6a73} {Suffrage:Voter ID:server-80391765-a073-f9de-81d0-5c87378677b7 Address:80391765-a073-f9de-81d0-5c87378677b7}]"
2025-07-14T05:39:02.392Z [INFO]  server-c77352f8-0ca1-dfae-063d-ff888eeb6a73: entering follower state: follower="Node at c77352f8-0ca1-dfae-063d-ff888eeb6a73 [Follower]" leader-address= leader-id=
2025-07-14T05:39:02.392Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdc0d21a-a28e-f474-ecb3-cb024c51789d Address:cdc0d21a-a28e-f474-ecb3-cb024c51789d} {Suffrage:Voter ID:server-c77352f8-0ca1-dfae-063d-ff888eeb6a73 Address:c77352f8-0ca1-dfae-063d-ff888eeb6a73} {Suffrage:Voter ID:server-80391765-a073-f9de-81d0-5c87378677b7 Address:80391765-a073-f9de-81d0-5c87378677b7}]"
2025-07-14T05:39:02.392Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:02.392Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: entering follower state: follower="Node at 80391765-a073-f9de-81d0-5c87378677b7 [Follower]" leader-address= leader-id=
2025-07-14T05:39:02.914Z [WARN]  server-80391765-a073-f9de-81d0-5c87378677b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:02.914Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: entering candidate state: node="Node at 80391765-a073-f9de-81d0-5c87378677b7 [Candidate]" term=2
2025-07-14T05:39:02.914Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: election won: term=2 tally=2
2025-07-14T05:39:02.914Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: entering leader state: leader="Node at 80391765-a073-f9de-81d0-5c87378677b7 [Leader]"
2025-07-14T05:39:02.914Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: added peer, starting replication: peer=server-cdc0d21a-a28e-f474-ecb3-cb024c51789d
2025-07-14T05:39:02.914Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: added peer, starting replication: peer=server-c77352f8-0ca1-dfae-063d-ff888eeb6a73
2025-07-14T05:39:02.915Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: pipelining replication: peer="{Voter server-c77352f8-0ca1-dfae-063d-ff888eeb6a73 c77352f8-0ca1-dfae-063d-ff888eeb6a73}"
2025-07-14T05:39:02.915Z [WARN]  server-cdc0d21a-a28e-f474-ecb3-cb024c51789d: rejecting vote request since our last term is greater: candidate=80391765-a073-f9de-81d0-5c87378677b7 last-term=2 last-candidate-term=1
2025-07-14T05:39:02.915Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: pipelining replication: peer="{Voter server-cdc0d21a-a28e-f474-ecb3-cb024c51789d cdc0d21a-a28e-f474-ecb3-cb024c51789d}"
2025-07-14T05:39:03.920Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:02.915069732 +0000 UTC m=+42.329530110 (1 nodes), 522.957647ms from start of poll, 523.250866ms from cluster start. Timeout at 2025-07-14 05:39:03.920653167 +0000 UTC m=+43.335113545, 1.005583435s after stability
2025-07-14T05:39:03.921Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: starting snapshot up to: index=12
2025-07-14T05:39:03.921Z [INFO]  creating new snapshot: path=/tmp/raft3383094075/snapshots/2-12-1752471543921.tmp
2025-07-14T05:39:03.921Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: snapshot complete up to: index=12
2025-07-14T05:39:03.922Z [INFO]  creating new snapshot: path=/tmp/raft3383094075/snapshots/2-23-1752471543922.tmp
2025-07-14T05:39:04.013Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: copied to local snapshot: bytes=71
2025-07-14T05:39:04.013Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: snapshot restore progress: id=2-23-1752471543922 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:04.013Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: restored user snapshot: index=23
2025-07-14T05:39:04.013Z [ERROR] server-80391765-a073-f9de-81d0-5c87378677b7: failed to get log: index=23 error="log not found"
2025-07-14T05:39:04.013Z [ERROR] server-80391765-a073-f9de-81d0-5c87378677b7: failed to get log: index=23 error="log not found"
2025-07-14T05:39:04.013Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: aborting pipeline replication: peer="{Voter server-cdc0d21a-a28e-f474-ecb3-cb024c51789d cdc0d21a-a28e-f474-ecb3-cb024c51789d}"
2025-07-14T05:39:04.013Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: aborting pipeline replication: peer="{Voter server-c77352f8-0ca1-dfae-063d-ff888eeb6a73 c77352f8-0ca1-dfae-063d-ff888eeb6a73}"
2025-07-14T05:39:04.020Z [ERROR] server-80391765-a073-f9de-81d0-5c87378677b7: failed to get log: index=23 error="log not found"
2025-07-14T05:39:04.020Z [ERROR] server-80391765-a073-f9de-81d0-5c87378677b7: failed to get log: index=23 error="log not found"
2025-07-14T05:39:04.020Z [INFO]  creating new snapshot: path=/tmp/raft2063624322/snapshots/2-23-1752471544020.tmp
2025-07-14T05:39:04.021Z [INFO]  server-c77352f8-0ca1-dfae-063d-ff888eeb6a73: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:04.021Z [INFO]  server-c77352f8-0ca1-dfae-063d-ff888eeb6a73: copied to local snapshot: bytes=71
2025-07-14T05:39:04.021Z [INFO]  creating new snapshot: path=/tmp/raft2648905072/snapshots/2-23-1752471544021.tmp
2025-07-14T05:39:04.021Z [INFO]  server-cdc0d21a-a28e-f474-ecb3-cb024c51789d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:04.022Z [INFO]  server-cdc0d21a-a28e-f474-ecb3-cb024c51789d: copied to local snapshot: bytes=71
2025-07-14T05:39:04.022Z [INFO]  server-c77352f8-0ca1-dfae-063d-ff888eeb6a73: snapshot restore progress: id=2-23-1752471544020 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:04.022Z [INFO]  server-c77352f8-0ca1-dfae-063d-ff888eeb6a73: Installed remote snapshot
2025-07-14T05:39:04.022Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: pipelining replication: peer="{Voter server-c77352f8-0ca1-dfae-063d-ff888eeb6a73 c77352f8-0ca1-dfae-063d-ff888eeb6a73}"
2025-07-14T05:39:04.022Z [INFO]  server-cdc0d21a-a28e-f474-ecb3-cb024c51789d: snapshot restore progress: id=2-23-1752471544021 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:04.022Z [INFO]  server-cdc0d21a-a28e-f474-ecb3-cb024c51789d: Installed remote snapshot
2025-07-14T05:39:04.022Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:04.022Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: pipelining replication: peer="{Voter server-cdc0d21a-a28e-f474-ecb3-cb024c51789d cdc0d21a-a28e-f474-ecb3-cb024c51789d}"
2025-07-14T05:39:05.028Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:04.022846296 +0000 UTC m=+43.437306674 (2 nodes), 610ns from start of poll, 1.63102743s from cluster start. Timeout at 2025-07-14 05:39:05.028252155 +0000 UTC m=+44.442712553, 1.005405879s after stability
2025-07-14T05:39:05.039Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: aborting pipeline replication: peer="{Voter server-c77352f8-0ca1-dfae-063d-ff888eeb6a73 c77352f8-0ca1-dfae-063d-ff888eeb6a73}"
2025-07-14T05:39:05.039Z [INFO]  server-80391765-a073-f9de-81d0-5c87378677b7: aborting pipeline replication: peer="{Voter server-cdc0d21a-a28e-f474-ecb3-cb024c51789d cdc0d21a-a28e-f474-ecb3-cb024c51789d}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-07-14T05:39:05.137Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a04d5a2f-cdb4-20b9-4f11-924359a14825 Address:a04d5a2f-cdb4-20b9-4f11-924359a14825} {Suffrage:Voter ID:server-d2ee9411-d640-fba7-dceb-53634c8e76ef Address:d2ee9411-d640-fba7-dceb-53634c8e76ef} {Suffrage:Voter ID:server-c6996310-1a5c-10c0-99bf-d2aac4214d89 Address:c6996310-1a5c-10c0-99bf-d2aac4214d89}]"
2025-07-14T05:39:05.137Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: entering follower state: follower="Node at a04d5a2f-cdb4-20b9-4f11-924359a14825 [Follower]" leader-address= leader-id=
2025-07-14T05:39:05.137Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a04d5a2f-cdb4-20b9-4f11-924359a14825 Address:a04d5a2f-cdb4-20b9-4f11-924359a14825} {Suffrage:Voter ID:server-d2ee9411-d640-fba7-dceb-53634c8e76ef Address:d2ee9411-d640-fba7-dceb-53634c8e76ef} {Suffrage:Voter ID:server-c6996310-1a5c-10c0-99bf-d2aac4214d89 Address:c6996310-1a5c-10c0-99bf-d2aac4214d89}]"
2025-07-14T05:39:05.137Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: entering follower state: follower="Node at d2ee9411-d640-fba7-dceb-53634c8e76ef [Follower]" leader-address= leader-id=
2025-07-14T05:39:05.137Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a04d5a2f-cdb4-20b9-4f11-924359a14825 Address:a04d5a2f-cdb4-20b9-4f11-924359a14825} {Suffrage:Voter ID:server-d2ee9411-d640-fba7-dceb-53634c8e76ef Address:d2ee9411-d640-fba7-dceb-53634c8e76ef} {Suffrage:Voter ID:server-c6996310-1a5c-10c0-99bf-d2aac4214d89 Address:c6996310-1a5c-10c0-99bf-d2aac4214d89}]"
2025-07-14T05:39:05.137Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:05.137Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: entering follower state: follower="Node at c6996310-1a5c-10c0-99bf-d2aac4214d89 [Follower]" leader-address= leader-id=
2025-07-14T05:39:05.700Z [WARN]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:05.700Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: entering candidate state: node="Node at c6996310-1a5c-10c0-99bf-d2aac4214d89 [Candidate]" term=2
2025-07-14T05:39:05.700Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: election won: term=2 tally=2
2025-07-14T05:39:05.700Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: entering leader state: leader="Node at c6996310-1a5c-10c0-99bf-d2aac4214d89 [Leader]"
2025-07-14T05:39:05.700Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: added peer, starting replication: peer=server-a04d5a2f-cdb4-20b9-4f11-924359a14825
2025-07-14T05:39:05.700Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: added peer, starting replication: peer=server-d2ee9411-d640-fba7-dceb-53634c8e76ef
2025-07-14T05:39:05.700Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: pipelining replication: peer="{Voter server-d2ee9411-d640-fba7-dceb-53634c8e76ef d2ee9411-d640-fba7-dceb-53634c8e76ef}"
2025-07-14T05:39:05.700Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: pipelining replication: peer="{Voter server-a04d5a2f-cdb4-20b9-4f11-924359a14825 a04d5a2f-cdb4-20b9-4f11-924359a14825}"
2025-07-14T05:39:06.706Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:05.700497623 +0000 UTC m=+45.114958001 (1 nodes), 562.849123ms from start of poll, 563.152438ms from cluster start. Timeout at 2025-07-14 05:39:06.706472635 +0000 UTC m=+46.120933024, 1.005975023s after stability
2025-07-14T05:39:06.708Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: starting snapshot up to: index=12
2025-07-14T05:39:06.708Z [INFO]  creating new snapshot: path=/tmp/raft1919934446/snapshots/2-12-1752471546708.tmp
2025-07-14T05:39:06.712Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: snapshot complete up to: index=12
2025-07-14T05:39:06.715Z [INFO]  creating new snapshot: path=/tmp/raft1919934446/snapshots/2-23-1752471546715.tmp
2025-07-14T05:39:06.715Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: copied to local snapshot: bytes=71
2025-07-14T05:39:06.715Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: snapshot restore progress: id=2-23-1752471546715 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:06.715Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: removing all old logs from log store
2025-07-14T05:39:06.715Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: compacting logs: from=1 to=22
2025-07-14T05:39:06.715Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: restored user snapshot: index=23
2025-07-14T05:39:06.715Z [ERROR] server-c6996310-1a5c-10c0-99bf-d2aac4214d89: failed to get log: index=22 error="log not found"
2025-07-14T05:39:06.715Z [ERROR] server-c6996310-1a5c-10c0-99bf-d2aac4214d89: failed to get log: index=22 error="log not found"
2025-07-14T05:39:06.715Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: aborting pipeline replication: peer="{Voter server-a04d5a2f-cdb4-20b9-4f11-924359a14825 a04d5a2f-cdb4-20b9-4f11-924359a14825}"
2025-07-14T05:39:06.715Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: aborting pipeline replication: peer="{Voter server-d2ee9411-d640-fba7-dceb-53634c8e76ef d2ee9411-d640-fba7-dceb-53634c8e76ef}"
2025-07-14T05:39:06.722Z [ERROR] server-c6996310-1a5c-10c0-99bf-d2aac4214d89: failed to get log: index=22 error="log not found"
2025-07-14T05:39:06.723Z [INFO]  creating new snapshot: path=/tmp/raft738141235/snapshots/2-23-1752471546723.tmp
2025-07-14T05:39:06.723Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:06.724Z [ERROR] server-c6996310-1a5c-10c0-99bf-d2aac4214d89: failed to get log: index=22 error="log not found"
2025-07-14T05:39:06.725Z [INFO]  creating new snapshot: path=/tmp/raft2910258181/snapshots/2-23-1752471546725.tmp
2025-07-14T05:39:06.725Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:06.938Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: copied to local snapshot: bytes=71
2025-07-14T05:39:06.938Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: copied to local snapshot: bytes=71
2025-07-14T05:39:06.938Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: snapshot restore progress: id=2-23-1752471546723 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:06.938Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: removing all old logs from log store
2025-07-14T05:39:06.938Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: compacting logs: from=1 to=22
2025-07-14T05:39:06.938Z [INFO]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: Installed remote snapshot
2025-07-14T05:39:06.938Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: snapshot restore progress: id=2-23-1752471546725 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:06.938Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: removing all old logs from log store
2025-07-14T05:39:06.938Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: compacting logs: from=1 to=22
2025-07-14T05:39:06.938Z [INFO]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: Installed remote snapshot
2025-07-14T05:39:06.938Z [WARN]  server-d2ee9411-d640-fba7-dceb-53634c8e76ef: skipping application of old log: index=22
2025-07-14T05:39:06.938Z [WARN]  server-a04d5a2f-cdb4-20b9-4f11-924359a14825: skipping application of old log: index=22
2025-07-14T05:39:06.938Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: pipelining replication: peer="{Voter server-d2ee9411-d640-fba7-dceb-53634c8e76ef d2ee9411-d640-fba7-dceb-53634c8e76ef}"
2025-07-14T05:39:06.938Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:06.939Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: pipelining replication: peer="{Voter server-a04d5a2f-cdb4-20b9-4f11-924359a14825 a04d5a2f-cdb4-20b9-4f11-924359a14825}"
2025-07-14T05:39:07.945Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:06.939043862 +0000 UTC m=+46.353504240 (2 nodes), 1.392µs from start of poll, 1.801698677s from cluster start. Timeout at 2025-07-14 05:39:07.944784333 +0000 UTC m=+47.359244701, 1.005740461s after stability
2025-07-14T05:39:07.958Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: aborting pipeline replication: peer="{Voter server-d2ee9411-d640-fba7-dceb-53634c8e76ef d2ee9411-d640-fba7-dceb-53634c8e76ef}"
2025-07-14T05:39:07.958Z [INFO]  server-c6996310-1a5c-10c0-99bf-d2aac4214d89: aborting pipeline replication: peer="{Voter server-a04d5a2f-cdb4-20b9-4f11-924359a14825 a04d5a2f-cdb4-20b9-4f11-924359a14825}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-07-14T05:39:07.959Z [INFO]  server-bb81e8ff-1361-c259-4e4a-56b95c694f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb81e8ff-1361-c259-4e4a-56b95c694f61 Address:bb81e8ff-1361-c259-4e4a-56b95c694f61} {Suffrage:Voter ID:server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8 Address:fe51cfd9-67d1-63e9-aa8f-10befcbd31f8} {Suffrage:Voter ID:server-e6961522-27aa-91ca-2eed-290c03482e30 Address:e6961522-27aa-91ca-2eed-290c03482e30}]"
2025-07-14T05:39:07.959Z [INFO]  server-bb81e8ff-1361-c259-4e4a-56b95c694f61: entering follower state: follower="Node at bb81e8ff-1361-c259-4e4a-56b95c694f61 [Follower]" leader-address= leader-id=
2025-07-14T05:39:07.959Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb81e8ff-1361-c259-4e4a-56b95c694f61 Address:bb81e8ff-1361-c259-4e4a-56b95c694f61} {Suffrage:Voter ID:server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8 Address:fe51cfd9-67d1-63e9-aa8f-10befcbd31f8} {Suffrage:Voter ID:server-e6961522-27aa-91ca-2eed-290c03482e30 Address:e6961522-27aa-91ca-2eed-290c03482e30}]"
2025-07-14T05:39:07.959Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: entering follower state: follower="Node at fe51cfd9-67d1-63e9-aa8f-10befcbd31f8 [Follower]" leader-address= leader-id=
2025-07-14T05:39:07.959Z [INFO]  server-e6961522-27aa-91ca-2eed-290c03482e30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb81e8ff-1361-c259-4e4a-56b95c694f61 Address:bb81e8ff-1361-c259-4e4a-56b95c694f61} {Suffrage:Voter ID:server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8 Address:fe51cfd9-67d1-63e9-aa8f-10befcbd31f8} {Suffrage:Voter ID:server-e6961522-27aa-91ca-2eed-290c03482e30 Address:e6961522-27aa-91ca-2eed-290c03482e30}]"
2025-07-14T05:39:07.959Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:07.959Z [INFO]  server-e6961522-27aa-91ca-2eed-290c03482e30: entering follower state: follower="Node at e6961522-27aa-91ca-2eed-290c03482e30 [Follower]" leader-address= leader-id=
2025-07-14T05:39:08.545Z [WARN]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:08.546Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: entering candidate state: node="Node at fe51cfd9-67d1-63e9-aa8f-10befcbd31f8 [Candidate]" term=2
2025-07-14T05:39:08.546Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: election won: term=2 tally=2
2025-07-14T05:39:08.546Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: entering leader state: leader="Node at fe51cfd9-67d1-63e9-aa8f-10befcbd31f8 [Leader]"
2025-07-14T05:39:08.546Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: added peer, starting replication: peer=server-bb81e8ff-1361-c259-4e4a-56b95c694f61
2025-07-14T05:39:08.546Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: added peer, starting replication: peer=server-e6961522-27aa-91ca-2eed-290c03482e30
2025-07-14T05:39:08.546Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: pipelining replication: peer="{Voter server-e6961522-27aa-91ca-2eed-290c03482e30 e6961522-27aa-91ca-2eed-290c03482e30}"
2025-07-14T05:39:08.546Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: pipelining replication: peer="{Voter server-bb81e8ff-1361-c259-4e4a-56b95c694f61 bb81e8ff-1361-c259-4e4a-56b95c694f61}"
2025-07-14T05:39:09.552Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:08.546458494 +0000 UTC m=+47.960918872 (1 nodes), 586.739609ms from start of poll, 587.014552ms from cluster start. Timeout at 2025-07-14 05:39:09.552539025 +0000 UTC m=+48.966999403, 1.006080531s after stability
2025-07-14T05:39:09.553Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: starting snapshot up to: index=12
2025-07-14T05:39:09.553Z [INFO]  creating new snapshot: path=/tmp/raft1693273509/snapshots/2-12-1752471549553.tmp
2025-07-14T05:39:09.601Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: snapshot complete up to: index=12
2025-07-14T05:39:09.602Z [INFO]  server-f07912e6-9e45-819f-dde3-cc2cc4f77216: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f07912e6-9e45-819f-dde3-cc2cc4f77216 Address:f07912e6-9e45-819f-dde3-cc2cc4f77216} {Suffrage:Voter ID:server-2b567f07-ade3-92e6-ef78-45c07f1df1ef Address:2b567f07-ade3-92e6-ef78-45c07f1df1ef} {Suffrage:Voter ID:server-c26764b1-e72b-7770-b257-31d31cc2a832 Address:c26764b1-e72b-7770-b257-31d31cc2a832}]"
2025-07-14T05:39:09.603Z [INFO]  server-f07912e6-9e45-819f-dde3-cc2cc4f77216: entering follower state: follower="Node at f07912e6-9e45-819f-dde3-cc2cc4f77216 [Follower]" leader-address= leader-id=
2025-07-14T05:39:09.603Z [INFO]  server-2b567f07-ade3-92e6-ef78-45c07f1df1ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f07912e6-9e45-819f-dde3-cc2cc4f77216 Address:f07912e6-9e45-819f-dde3-cc2cc4f77216} {Suffrage:Voter ID:server-2b567f07-ade3-92e6-ef78-45c07f1df1ef Address:2b567f07-ade3-92e6-ef78-45c07f1df1ef} {Suffrage:Voter ID:server-c26764b1-e72b-7770-b257-31d31cc2a832 Address:c26764b1-e72b-7770-b257-31d31cc2a832}]"
2025-07-14T05:39:09.603Z [INFO]  server-2b567f07-ade3-92e6-ef78-45c07f1df1ef: entering follower state: follower="Node at 2b567f07-ade3-92e6-ef78-45c07f1df1ef [Follower]" leader-address= leader-id=
2025-07-14T05:39:09.603Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f07912e6-9e45-819f-dde3-cc2cc4f77216 Address:f07912e6-9e45-819f-dde3-cc2cc4f77216} {Suffrage:Voter ID:server-2b567f07-ade3-92e6-ef78-45c07f1df1ef Address:2b567f07-ade3-92e6-ef78-45c07f1df1ef} {Suffrage:Voter ID:server-c26764b1-e72b-7770-b257-31d31cc2a832 Address:c26764b1-e72b-7770-b257-31d31cc2a832}]"
2025-07-14T05:39:09.603Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:09.603Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: entering follower state: follower="Node at c26764b1-e72b-7770-b257-31d31cc2a832 [Follower]" leader-address= leader-id=
2025-07-14T05:39:10.260Z [WARN]  server-c26764b1-e72b-7770-b257-31d31cc2a832: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:10.260Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: entering candidate state: node="Node at c26764b1-e72b-7770-b257-31d31cc2a832 [Candidate]" term=2
2025-07-14T05:39:10.261Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: election won: term=2 tally=2
2025-07-14T05:39:10.261Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: entering leader state: leader="Node at c26764b1-e72b-7770-b257-31d31cc2a832 [Leader]"
2025-07-14T05:39:10.261Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: added peer, starting replication: peer=server-f07912e6-9e45-819f-dde3-cc2cc4f77216
2025-07-14T05:39:10.261Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: added peer, starting replication: peer=server-2b567f07-ade3-92e6-ef78-45c07f1df1ef
2025-07-14T05:39:10.262Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: pipelining replication: peer="{Voter server-f07912e6-9e45-819f-dde3-cc2cc4f77216 f07912e6-9e45-819f-dde3-cc2cc4f77216}"
2025-07-14T05:39:10.262Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: pipelining replication: peer="{Voter server-2b567f07-ade3-92e6-ef78-45c07f1df1ef 2b567f07-ade3-92e6-ef78-45c07f1df1ef}"
2025-07-14T05:39:11.266Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:10.261206756 +0000 UTC m=+49.675667154 (1 nodes), 658.042043ms from start of poll, 658.35322ms from cluster start. Timeout at 2025-07-14 05:39:11.266425124 +0000 UTC m=+50.680885512, 1.005218358s after stability
2025-07-14T05:39:11.266Z [INFO]  creating new snapshot: path=/tmp/raft3345825512/snapshots/2-15-1752471551266.tmp
2025-07-14T05:39:11.285Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: copied to local snapshot: bytes=71
2025-07-14T05:39:11.285Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: snapshot restore progress: id=2-15-1752471551266 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:11.285Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: restored user snapshot: index=15
2025-07-14T05:39:11.285Z [ERROR] server-c26764b1-e72b-7770-b257-31d31cc2a832: failed to get log: index=3 error="log not found"
2025-07-14T05:39:11.285Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: aborting pipeline replication: peer="{Voter server-2b567f07-ade3-92e6-ef78-45c07f1df1ef 2b567f07-ade3-92e6-ef78-45c07f1df1ef}"
2025-07-14T05:39:11.285Z [ERROR] server-c26764b1-e72b-7770-b257-31d31cc2a832: failed to get log: index=3 error="log not found"
2025-07-14T05:39:11.285Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: aborting pipeline replication: peer="{Voter server-f07912e6-9e45-819f-dde3-cc2cc4f77216 f07912e6-9e45-819f-dde3-cc2cc4f77216}"
2025-07-14T05:39:11.295Z [ERROR] server-c26764b1-e72b-7770-b257-31d31cc2a832: failed to get log: index=3 error="log not found"
2025-07-14T05:39:11.295Z [INFO]  creating new snapshot: path=/tmp/raft2092965151/snapshots/2-15-1752471551295.tmp
2025-07-14T05:39:11.295Z [INFO]  server-2b567f07-ade3-92e6-ef78-45c07f1df1ef: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:11.295Z [ERROR] server-c26764b1-e72b-7770-b257-31d31cc2a832: failed to get log: index=3 error="log not found"
2025-07-14T05:39:11.295Z [INFO]  creating new snapshot: path=/tmp/raft190146267/snapshots/2-15-1752471551295.tmp
2025-07-14T05:39:11.296Z [INFO]  server-f07912e6-9e45-819f-dde3-cc2cc4f77216: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:11.412Z [INFO]  server-2b567f07-ade3-92e6-ef78-45c07f1df1ef: copied to local snapshot: bytes=71
2025-07-14T05:39:11.413Z [INFO]  server-2b567f07-ade3-92e6-ef78-45c07f1df1ef: snapshot restore progress: id=2-15-1752471551295 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:11.413Z [INFO]  server-2b567f07-ade3-92e6-ef78-45c07f1df1ef: Installed remote snapshot
2025-07-14T05:39:11.413Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: pipelining replication: peer="{Voter server-2b567f07-ade3-92e6-ef78-45c07f1df1ef 2b567f07-ade3-92e6-ef78-45c07f1df1ef}"
2025-07-14T05:39:11.413Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:11.413Z [INFO]  server-f07912e6-9e45-819f-dde3-cc2cc4f77216: copied to local snapshot: bytes=71
2025-07-14T05:39:11.413Z [INFO]  server-f07912e6-9e45-819f-dde3-cc2cc4f77216: snapshot restore progress: id=2-15-1752471551295 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:11.413Z [INFO]  server-f07912e6-9e45-819f-dde3-cc2cc4f77216: Installed remote snapshot
2025-07-14T05:39:11.413Z [INFO]  server-c26764b1-e72b-7770-b257-31d31cc2a832: pipelining replication: peer="{Voter server-f07912e6-9e45-819f-dde3-cc2cc4f77216 f07912e6-9e45-819f-dde3-cc2cc4f77216}"
2025-07-14T05:39:12.419Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:11.413198884 +0000 UTC m=+50.827659252 (2 nodes), 711ns from start of poll, 1.810345318s from cluster start. Timeout at 2025-07-14 05:39:12.419306676 +0000 UTC m=+51.833767074, 1.006107822s after stability
2025-07-14T05:39:12.430Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: aborting pipeline replication: peer="{Voter server-e6961522-27aa-91ca-2eed-290c03482e30 e6961522-27aa-91ca-2eed-290c03482e30}"
2025-07-14T05:39:12.430Z [INFO]  server-fe51cfd9-67d1-63e9-aa8f-10befcbd31f8: aborting pipeline replication: peer="{Voter server-bb81e8ff-1361-c259-4e4a-56b95c694f61 bb81e8ff-1361-c259-4e4a-56b95c694f61}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-07-14T05:39:12.432Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c2e9d59-3f02-bc93-a302-672f1945cc40 Address:9c2e9d59-3f02-bc93-a302-672f1945cc40} {Suffrage:Voter ID:server-7c5c19f0-24b9-5a27-7e74-711267e330cc Address:7c5c19f0-24b9-5a27-7e74-711267e330cc} {Suffrage:Voter ID:server-bb607b20-1b18-594f-d305-7d3e3e73467c Address:bb607b20-1b18-594f-d305-7d3e3e73467c}]"
2025-07-14T05:39:12.432Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: entering follower state: follower="Node at 9c2e9d59-3f02-bc93-a302-672f1945cc40 [Follower]" leader-address= leader-id=
2025-07-14T05:39:12.432Z [INFO]  server-7c5c19f0-24b9-5a27-7e74-711267e330cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c2e9d59-3f02-bc93-a302-672f1945cc40 Address:9c2e9d59-3f02-bc93-a302-672f1945cc40} {Suffrage:Voter ID:server-7c5c19f0-24b9-5a27-7e74-711267e330cc Address:7c5c19f0-24b9-5a27-7e74-711267e330cc} {Suffrage:Voter ID:server-bb607b20-1b18-594f-d305-7d3e3e73467c Address:bb607b20-1b18-594f-d305-7d3e3e73467c}]"
2025-07-14T05:39:12.432Z [INFO]  server-7c5c19f0-24b9-5a27-7e74-711267e330cc: entering follower state: follower="Node at 7c5c19f0-24b9-5a27-7e74-711267e330cc [Follower]" leader-address= leader-id=
2025-07-14T05:39:12.432Z [INFO]  server-bb607b20-1b18-594f-d305-7d3e3e73467c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c2e9d59-3f02-bc93-a302-672f1945cc40 Address:9c2e9d59-3f02-bc93-a302-672f1945cc40} {Suffrage:Voter ID:server-7c5c19f0-24b9-5a27-7e74-711267e330cc Address:7c5c19f0-24b9-5a27-7e74-711267e330cc} {Suffrage:Voter ID:server-bb607b20-1b18-594f-d305-7d3e3e73467c Address:bb607b20-1b18-594f-d305-7d3e3e73467c}]"
2025-07-14T05:39:12.432Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:12.432Z [INFO]  server-bb607b20-1b18-594f-d305-7d3e3e73467c: entering follower state: follower="Node at bb607b20-1b18-594f-d305-7d3e3e73467c [Follower]" leader-address= leader-id=
2025-07-14T05:39:12.958Z [WARN]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:12.958Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: entering candidate state: node="Node at 9c2e9d59-3f02-bc93-a302-672f1945cc40 [Candidate]" term=2
2025-07-14T05:39:12.958Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: election won: term=2 tally=2
2025-07-14T05:39:12.958Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: entering leader state: leader="Node at 9c2e9d59-3f02-bc93-a302-672f1945cc40 [Leader]"
2025-07-14T05:39:12.958Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: added peer, starting replication: peer=server-7c5c19f0-24b9-5a27-7e74-711267e330cc
2025-07-14T05:39:12.958Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: added peer, starting replication: peer=server-bb607b20-1b18-594f-d305-7d3e3e73467c
2025-07-14T05:39:12.959Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: pipelining replication: peer="{Voter server-bb607b20-1b18-594f-d305-7d3e3e73467c bb607b20-1b18-594f-d305-7d3e3e73467c}"
2025-07-14T05:39:12.959Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: pipelining replication: peer="{Voter server-7c5c19f0-24b9-5a27-7e74-711267e330cc 7c5c19f0-24b9-5a27-7e74-711267e330cc}"
2025-07-14T05:39:13.965Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:12.959704102 +0000 UTC m=+52.374164500 (1 nodes), 527.331397ms from start of poll, 527.739409ms from cluster start. Timeout at 2025-07-14 05:39:13.964970051 +0000 UTC m=+53.379430449, 1.005265949s after stability
2025-07-14T05:39:13.965Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: starting snapshot up to: index=12
2025-07-14T05:39:13.965Z [INFO]  creating new snapshot: path=/tmp/raft2545444954/snapshots/2-12-1752471553965.tmp
2025-07-14T05:39:13.974Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: snapshot complete up to: index=12
2025-07-14T05:39:13.975Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-001734e1-4faf-e53e-0e1f-a424d8414dbf Address:001734e1-4faf-e53e-0e1f-a424d8414dbf} {Suffrage:Voter ID:server-2f6e1359-c0df-e20e-be4c-624741604a1c Address:2f6e1359-c0df-e20e-be4c-624741604a1c} {Suffrage:Voter ID:server-d3739cf7-a92f-349e-f4dc-aad8140d7743 Address:d3739cf7-a92f-349e-f4dc-aad8140d7743}]"
2025-07-14T05:39:13.975Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: entering follower state: follower="Node at 001734e1-4faf-e53e-0e1f-a424d8414dbf [Follower]" leader-address= leader-id=
2025-07-14T05:39:13.975Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-001734e1-4faf-e53e-0e1f-a424d8414dbf Address:001734e1-4faf-e53e-0e1f-a424d8414dbf} {Suffrage:Voter ID:server-2f6e1359-c0df-e20e-be4c-624741604a1c Address:2f6e1359-c0df-e20e-be4c-624741604a1c} {Suffrage:Voter ID:server-d3739cf7-a92f-349e-f4dc-aad8140d7743 Address:d3739cf7-a92f-349e-f4dc-aad8140d7743}]"
2025-07-14T05:39:13.975Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: entering follower state: follower="Node at 2f6e1359-c0df-e20e-be4c-624741604a1c [Follower]" leader-address= leader-id=
2025-07-14T05:39:13.975Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-001734e1-4faf-e53e-0e1f-a424d8414dbf Address:001734e1-4faf-e53e-0e1f-a424d8414dbf} {Suffrage:Voter ID:server-2f6e1359-c0df-e20e-be4c-624741604a1c Address:2f6e1359-c0df-e20e-be4c-624741604a1c} {Suffrage:Voter ID:server-d3739cf7-a92f-349e-f4dc-aad8140d7743 Address:d3739cf7-a92f-349e-f4dc-aad8140d7743}]"
2025-07-14T05:39:13.975Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:13.975Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: entering follower state: follower="Node at d3739cf7-a92f-349e-f4dc-aad8140d7743 [Follower]" leader-address= leader-id=
2025-07-14T05:39:14.681Z [WARN]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:14.681Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: entering candidate state: node="Node at 001734e1-4faf-e53e-0e1f-a424d8414dbf [Candidate]" term=2
2025-07-14T05:39:14.681Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: election won: term=2 tally=2
2025-07-14T05:39:14.681Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: entering leader state: leader="Node at 001734e1-4faf-e53e-0e1f-a424d8414dbf [Leader]"
2025-07-14T05:39:14.681Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: added peer, starting replication: peer=server-2f6e1359-c0df-e20e-be4c-624741604a1c
2025-07-14T05:39:14.681Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: added peer, starting replication: peer=server-d3739cf7-a92f-349e-f4dc-aad8140d7743
2025-07-14T05:39:14.681Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: pipelining replication: peer="{Voter server-2f6e1359-c0df-e20e-be4c-624741604a1c 2f6e1359-c0df-e20e-be4c-624741604a1c}"
2025-07-14T05:39:14.681Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: pipelining replication: peer="{Voter server-d3739cf7-a92f-349e-f4dc-aad8140d7743 d3739cf7-a92f-349e-f4dc-aad8140d7743}"
2025-07-14T05:39:15.686Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:14.681312975 +0000 UTC m=+54.095773353 (1 nodes), 705.4132ms from start of poll, 705.739469ms from cluster start. Timeout at 2025-07-14 05:39:15.68665664 +0000 UTC m=+55.101117018, 1.005343665s after stability
2025-07-14T05:39:15.686Z [INFO]  creating new snapshot: path=/tmp/raft1960380390/snapshots/2-15-1752471555686.tmp
2025-07-14T05:39:15.715Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: copied to local snapshot: bytes=71
2025-07-14T05:39:15.717Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: snapshot restore progress: id=2-15-1752471555686 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:15.717Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: removing all old logs from log store
2025-07-14T05:39:15.717Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: compacting logs: from=1 to=2
2025-07-14T05:39:15.717Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: restored user snapshot: index=15
2025-07-14T05:39:15.717Z [ERROR] server-001734e1-4faf-e53e-0e1f-a424d8414dbf: failed to get log: index=2 error="log not found"
2025-07-14T05:39:15.717Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: aborting pipeline replication: peer="{Voter server-d3739cf7-a92f-349e-f4dc-aad8140d7743 d3739cf7-a92f-349e-f4dc-aad8140d7743}"
2025-07-14T05:39:15.717Z [ERROR] server-001734e1-4faf-e53e-0e1f-a424d8414dbf: failed to get log: index=2 error="log not found"
2025-07-14T05:39:15.717Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: aborting pipeline replication: peer="{Voter server-2f6e1359-c0df-e20e-be4c-624741604a1c 2f6e1359-c0df-e20e-be4c-624741604a1c}"
2025-07-14T05:39:15.726Z [ERROR] server-001734e1-4faf-e53e-0e1f-a424d8414dbf: failed to get log: index=2 error="log not found"
2025-07-14T05:39:15.726Z [INFO]  creating new snapshot: path=/tmp/raft2986823293/snapshots/2-15-1752471555726.tmp
2025-07-14T05:39:15.726Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:15.727Z [ERROR] server-001734e1-4faf-e53e-0e1f-a424d8414dbf: failed to get log: index=2 error="log not found"
2025-07-14T05:39:15.727Z [INFO]  creating new snapshot: path=/tmp/raft281499710/snapshots/2-15-1752471555727.tmp
2025-07-14T05:39:15.727Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:15.826Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: copied to local snapshot: bytes=71
2025-07-14T05:39:15.826Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: copied to local snapshot: bytes=71
2025-07-14T05:39:15.826Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: snapshot restore progress: id=2-15-1752471555727 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:15.826Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: removing all old logs from log store
2025-07-14T05:39:15.826Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: compacting logs: from=1 to=2
2025-07-14T05:39:15.826Z [INFO]  server-d3739cf7-a92f-349e-f4dc-aad8140d7743: Installed remote snapshot
2025-07-14T05:39:15.826Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: pipelining replication: peer="{Voter server-d3739cf7-a92f-349e-f4dc-aad8140d7743 d3739cf7-a92f-349e-f4dc-aad8140d7743}"
2025-07-14T05:39:15.826Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: snapshot restore progress: id=2-15-1752471555726 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:15.826Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:15.826Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: removing all old logs from log store
2025-07-14T05:39:15.826Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: compacting logs: from=1 to=2
2025-07-14T05:39:15.826Z [INFO]  server-2f6e1359-c0df-e20e-be4c-624741604a1c: Installed remote snapshot
2025-07-14T05:39:15.826Z [INFO]  server-001734e1-4faf-e53e-0e1f-a424d8414dbf: pipelining replication: peer="{Voter server-2f6e1359-c0df-e20e-be4c-624741604a1c 2f6e1359-c0df-e20e-be4c-624741604a1c}"
2025-07-14T05:39:16.832Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:15.826608127 +0000 UTC m=+55.241068495 (2 nodes), 831ns from start of poll, 1.851034611s from cluster start. Timeout at 2025-07-14 05:39:16.832465984 +0000 UTC m=+56.246926362, 1.005857867s after stability
2025-07-14T05:39:16.844Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: aborting pipeline replication: peer="{Voter server-bb607b20-1b18-594f-d305-7d3e3e73467c bb607b20-1b18-594f-d305-7d3e3e73467c}"
2025-07-14T05:39:16.844Z [INFO]  server-9c2e9d59-3f02-bc93-a302-672f1945cc40: aborting pipeline replication: peer="{Voter server-7c5c19f0-24b9-5a27-7e74-711267e330cc 7c5c19f0-24b9-5a27-7e74-711267e330cc}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-07-14T05:39:16.845Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d337307d-5103-549f-b92f-887cf40ad302 Address:d337307d-5103-549f-b92f-887cf40ad302} {Suffrage:Voter ID:server-0aad98df-1532-535b-2aec-151e5d6abfdf Address:0aad98df-1532-535b-2aec-151e5d6abfdf} {Suffrage:Voter ID:server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 Address:1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6}]"
2025-07-14T05:39:16.845Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: entering follower state: follower="Node at d337307d-5103-549f-b92f-887cf40ad302 [Follower]" leader-address= leader-id=
2025-07-14T05:39:16.845Z [INFO]  server-0aad98df-1532-535b-2aec-151e5d6abfdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d337307d-5103-549f-b92f-887cf40ad302 Address:d337307d-5103-549f-b92f-887cf40ad302} {Suffrage:Voter ID:server-0aad98df-1532-535b-2aec-151e5d6abfdf Address:0aad98df-1532-535b-2aec-151e5d6abfdf} {Suffrage:Voter ID:server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 Address:1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6}]"
2025-07-14T05:39:16.845Z [INFO]  server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d337307d-5103-549f-b92f-887cf40ad302 Address:d337307d-5103-549f-b92f-887cf40ad302} {Suffrage:Voter ID:server-0aad98df-1532-535b-2aec-151e5d6abfdf Address:0aad98df-1532-535b-2aec-151e5d6abfdf} {Suffrage:Voter ID:server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 Address:1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6}]"
2025-07-14T05:39:16.845Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:16.845Z [INFO]  server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6: entering follower state: follower="Node at 1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 [Follower]" leader-address= leader-id=
2025-07-14T05:39:16.845Z [INFO]  server-0aad98df-1532-535b-2aec-151e5d6abfdf: entering follower state: follower="Node at 0aad98df-1532-535b-2aec-151e5d6abfdf [Follower]" leader-address= leader-id=
2025-07-14T05:39:17.476Z [WARN]  server-d337307d-5103-549f-b92f-887cf40ad302: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:17.476Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: entering candidate state: node="Node at d337307d-5103-549f-b92f-887cf40ad302 [Candidate]" term=2
2025-07-14T05:39:17.476Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: election won: term=2 tally=2
2025-07-14T05:39:17.476Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: entering leader state: leader="Node at d337307d-5103-549f-b92f-887cf40ad302 [Leader]"
2025-07-14T05:39:17.476Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: added peer, starting replication: peer=server-0aad98df-1532-535b-2aec-151e5d6abfdf
2025-07-14T05:39:17.476Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: added peer, starting replication: peer=server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6
2025-07-14T05:39:17.476Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: pipelining replication: peer="{Voter server-0aad98df-1532-535b-2aec-151e5d6abfdf 0aad98df-1532-535b-2aec-151e5d6abfdf}"
2025-07-14T05:39:17.476Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: pipelining replication: peer="{Voter server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6}"
2025-07-14T05:39:18.482Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:17.476635769 +0000 UTC m=+56.891096137 (1 nodes), 630.86657ms from start of poll, 631.13973ms from cluster start. Timeout at 2025-07-14 05:39:18.482233195 +0000 UTC m=+57.896693563, 1.005597426s after stability
2025-07-14T05:39:18.482Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: starting snapshot up to: index=12
2025-07-14T05:39:18.482Z [INFO]  creating new snapshot: path=/tmp/raft1021376169/snapshots/2-12-1752471558482.tmp
2025-07-14T05:39:18.542Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: snapshot complete up to: index=12
2025-07-14T05:39:18.542Z [INFO]  creating new snapshot: path=/tmp/raft1021376169/snapshots/2-113-1752471558542.tmp
2025-07-14T05:39:18.677Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: copied to local snapshot: bytes=71
2025-07-14T05:39:18.678Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: snapshot restore progress: id=2-113-1752471558542 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:18.678Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: restored user snapshot: index=113
2025-07-14T05:39:18.678Z [ERROR] server-d337307d-5103-549f-b92f-887cf40ad302: failed to get log: index=23 error="log not found"
2025-07-14T05:39:18.678Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: aborting pipeline replication: peer="{Voter server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6}"
2025-07-14T05:39:18.678Z [ERROR] server-d337307d-5103-549f-b92f-887cf40ad302: failed to get log: index=23 error="log not found"
2025-07-14T05:39:18.678Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: aborting pipeline replication: peer="{Voter server-0aad98df-1532-535b-2aec-151e5d6abfdf 0aad98df-1532-535b-2aec-151e5d6abfdf}"
2025-07-14T05:39:18.685Z [ERROR] server-d337307d-5103-549f-b92f-887cf40ad302: failed to get log: index=23 error="log not found"
2025-07-14T05:39:18.686Z [ERROR] server-d337307d-5103-549f-b92f-887cf40ad302: failed to get log: index=23 error="log not found"
2025-07-14T05:39:18.686Z [INFO]  creating new snapshot: path=/tmp/raft9618337/snapshots/2-113-1752471558686.tmp
2025-07-14T05:39:18.686Z [INFO]  creating new snapshot: path=/tmp/raft1530691343/snapshots/2-113-1752471558686.tmp
2025-07-14T05:39:18.686Z [INFO]  server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:18.687Z [INFO]  server-0aad98df-1532-535b-2aec-151e5d6abfdf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:18.688Z [INFO]  server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6: copied to local snapshot: bytes=71
2025-07-14T05:39:18.688Z [INFO]  server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6: snapshot restore progress: id=2-113-1752471558686 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:18.688Z [INFO]  server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6: Installed remote snapshot
2025-07-14T05:39:18.688Z [INFO]  server-0aad98df-1532-535b-2aec-151e5d6abfdf: copied to local snapshot: bytes=71
2025-07-14T05:39:18.688Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: pipelining replication: peer="{Voter server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6}"
2025-07-14T05:39:18.688Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:18.688Z [INFO]  server-0aad98df-1532-535b-2aec-151e5d6abfdf: snapshot restore progress: id=2-113-1752471558686 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:18.688Z [INFO]  server-0aad98df-1532-535b-2aec-151e5d6abfdf: Installed remote snapshot
2025-07-14T05:39:18.688Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: pipelining replication: peer="{Voter server-0aad98df-1532-535b-2aec-151e5d6abfdf 0aad98df-1532-535b-2aec-151e5d6abfdf}"
2025-07-14T05:39:19.694Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:18.688414547 +0000 UTC m=+58.102874925 (2 nodes), 551ns from start of poll, 1.842918518s from cluster start. Timeout at 2025-07-14 05:39:19.694062889 +0000 UTC m=+59.108523287, 1.005648362s after stability
2025-07-14T05:39:19.706Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: aborting pipeline replication: peer="{Voter server-1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6 1e0ac9a0-fc2a-bb11-4b39-f5d407e848c6}"
2025-07-14T05:39:19.706Z [INFO]  server-d337307d-5103-549f-b92f-887cf40ad302: aborting pipeline replication: peer="{Voter server-0aad98df-1532-535b-2aec-151e5d6abfdf 0aad98df-1532-535b-2aec-151e5d6abfdf}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-07-14T05:39:19.708Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c Address:1c45ea50-7de0-1de0-e0b5-88e564c8ce7c} {Suffrage:Voter ID:server-8e523737-dd9e-d5da-8a91-11b92cd93c2a Address:8e523737-dd9e-d5da-8a91-11b92cd93c2a} {Suffrage:Voter ID:server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606 Address:e5ac093a-c6ac-95c0-8a86-ec10d84cb606}]"
2025-07-14T05:39:19.708Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: entering follower state: follower="Node at 1c45ea50-7de0-1de0-e0b5-88e564c8ce7c [Follower]" leader-address= leader-id=
2025-07-14T05:39:19.708Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c Address:1c45ea50-7de0-1de0-e0b5-88e564c8ce7c} {Suffrage:Voter ID:server-8e523737-dd9e-d5da-8a91-11b92cd93c2a Address:8e523737-dd9e-d5da-8a91-11b92cd93c2a} {Suffrage:Voter ID:server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606 Address:e5ac093a-c6ac-95c0-8a86-ec10d84cb606}]"
2025-07-14T05:39:19.708Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: entering follower state: follower="Node at 8e523737-dd9e-d5da-8a91-11b92cd93c2a [Follower]" leader-address= leader-id=
2025-07-14T05:39:19.708Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c Address:1c45ea50-7de0-1de0-e0b5-88e564c8ce7c} {Suffrage:Voter ID:server-8e523737-dd9e-d5da-8a91-11b92cd93c2a Address:8e523737-dd9e-d5da-8a91-11b92cd93c2a} {Suffrage:Voter ID:server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606 Address:e5ac093a-c6ac-95c0-8a86-ec10d84cb606}]"
2025-07-14T05:39:19.708Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:19.708Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: entering follower state: follower="Node at e5ac093a-c6ac-95c0-8a86-ec10d84cb606 [Follower]" leader-address= leader-id=
2025-07-14T05:39:20.286Z [WARN]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:20.286Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: entering candidate state: node="Node at 1c45ea50-7de0-1de0-e0b5-88e564c8ce7c [Candidate]" term=2
2025-07-14T05:39:20.286Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: election won: term=2 tally=2
2025-07-14T05:39:20.286Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: entering leader state: leader="Node at 1c45ea50-7de0-1de0-e0b5-88e564c8ce7c [Leader]"
2025-07-14T05:39:20.286Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: added peer, starting replication: peer=server-8e523737-dd9e-d5da-8a91-11b92cd93c2a
2025-07-14T05:39:20.286Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: added peer, starting replication: peer=server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606
2025-07-14T05:39:20.286Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: pipelining replication: peer="{Voter server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606 e5ac093a-c6ac-95c0-8a86-ec10d84cb606}"
2025-07-14T05:39:20.286Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: pipelining replication: peer="{Voter server-8e523737-dd9e-d5da-8a91-11b92cd93c2a 8e523737-dd9e-d5da-8a91-11b92cd93c2a}"
2025-07-14T05:39:21.292Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:20.286619153 +0000 UTC m=+59.701079521 (1 nodes), 577.849414ms from start of poll, 578.150575ms from cluster start. Timeout at 2025-07-14 05:39:21.292637741 +0000 UTC m=+60.707098109, 1.006018588s after stability
2025-07-14T05:39:21.292Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: starting snapshot up to: index=12
2025-07-14T05:39:21.293Z [INFO]  creating new snapshot: path=/tmp/raft3952426764/snapshots/2-12-1752471561293.tmp
2025-07-14T05:39:21.293Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: snapshot complete up to: index=12
2025-07-14T05:39:21.293Z [INFO]  creating new snapshot: path=/tmp/raft3952426764/snapshots/2-113-1752471561293.tmp
2025-07-14T05:39:21.294Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: copied to local snapshot: bytes=71
2025-07-14T05:39:21.294Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: snapshot restore progress: id=2-113-1752471561293 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:21.294Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: removing all old logs from log store
2025-07-14T05:39:21.294Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: compacting logs: from=1 to=22
2025-07-14T05:39:21.294Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: restored user snapshot: index=113
2025-07-14T05:39:21.294Z [ERROR] server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: failed to get log: index=22 error="log not found"
2025-07-14T05:39:21.294Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: aborting pipeline replication: peer="{Voter server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606 e5ac093a-c6ac-95c0-8a86-ec10d84cb606}"
2025-07-14T05:39:21.294Z [ERROR] server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: failed to get log: index=22 error="log not found"
2025-07-14T05:39:21.294Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: aborting pipeline replication: peer="{Voter server-8e523737-dd9e-d5da-8a91-11b92cd93c2a 8e523737-dd9e-d5da-8a91-11b92cd93c2a}"
2025-07-14T05:39:21.304Z [ERROR] server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: failed to get log: index=22 error="log not found"
2025-07-14T05:39:21.304Z [ERROR] server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: failed to get log: index=22 error="log not found"
2025-07-14T05:39:21.304Z [INFO]  creating new snapshot: path=/tmp/raft990139771/snapshots/2-113-1752471561304.tmp
2025-07-14T05:39:21.304Z [INFO]  creating new snapshot: path=/tmp/raft262098349/snapshots/2-113-1752471561304.tmp
2025-07-14T05:39:21.304Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:21.305Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:21.305Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: copied to local snapshot: bytes=71
2025-07-14T05:39:21.305Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: copied to local snapshot: bytes=71
2025-07-14T05:39:21.305Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: snapshot restore progress: id=2-113-1752471561304 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:21.305Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: removing all old logs from log store
2025-07-14T05:39:21.305Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: compacting logs: from=1 to=22
2025-07-14T05:39:21.305Z [INFO]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: Installed remote snapshot
2025-07-14T05:39:21.305Z [WARN]  server-8e523737-dd9e-d5da-8a91-11b92cd93c2a: skipping application of old log: index=22
2025-07-14T05:39:21.305Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: snapshot restore progress: id=2-113-1752471561304 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:21.305Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: removing all old logs from log store
2025-07-14T05:39:21.305Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: compacting logs: from=1 to=22
2025-07-14T05:39:21.305Z [INFO]  server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606: Installed remote snapshot
2025-07-14T05:39:21.305Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: pipelining replication: peer="{Voter server-8e523737-dd9e-d5da-8a91-11b92cd93c2a 8e523737-dd9e-d5da-8a91-11b92cd93c2a}"
2025-07-14T05:39:21.305Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:21.305Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: pipelining replication: peer="{Voter server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606 e5ac093a-c6ac-95c0-8a86-ec10d84cb606}"
2025-07-14T05:39:22.310Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:21.305419393 +0000 UTC m=+60.719879792 (2 nodes), 742ns from start of poll, 1.596950846s from cluster start. Timeout at 2025-07-14 05:39:22.31095035 +0000 UTC m=+61.725410718, 1.005530926s after stability
2025-07-14T05:39:22.322Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: aborting pipeline replication: peer="{Voter server-e5ac093a-c6ac-95c0-8a86-ec10d84cb606 e5ac093a-c6ac-95c0-8a86-ec10d84cb606}"
2025-07-14T05:39:22.322Z [INFO]  server-1c45ea50-7de0-1de0-e0b5-88e564c8ce7c: aborting pipeline replication: peer="{Voter server-8e523737-dd9e-d5da-8a91-11b92cd93c2a 8e523737-dd9e-d5da-8a91-11b92cd93c2a}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-07-14T05:39:22.323Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64284921-b081-1499-6ed6-da3f368a04a2 Address:64284921-b081-1499-6ed6-da3f368a04a2} {Suffrage:Voter ID:server-2d8fdcf4-2f65-3875-1d35-1aa189a46717 Address:2d8fdcf4-2f65-3875-1d35-1aa189a46717} {Suffrage:Voter ID:server-6a2ce9c8-a261-6430-0d7b-897efc824945 Address:6a2ce9c8-a261-6430-0d7b-897efc824945}]"
2025-07-14T05:39:22.323Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: entering follower state: follower="Node at 64284921-b081-1499-6ed6-da3f368a04a2 [Follower]" leader-address= leader-id=
2025-07-14T05:39:22.323Z [INFO]  server-2d8fdcf4-2f65-3875-1d35-1aa189a46717: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64284921-b081-1499-6ed6-da3f368a04a2 Address:64284921-b081-1499-6ed6-da3f368a04a2} {Suffrage:Voter ID:server-2d8fdcf4-2f65-3875-1d35-1aa189a46717 Address:2d8fdcf4-2f65-3875-1d35-1aa189a46717} {Suffrage:Voter ID:server-6a2ce9c8-a261-6430-0d7b-897efc824945 Address:6a2ce9c8-a261-6430-0d7b-897efc824945}]"
2025-07-14T05:39:22.324Z [INFO]  server-6a2ce9c8-a261-6430-0d7b-897efc824945: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64284921-b081-1499-6ed6-da3f368a04a2 Address:64284921-b081-1499-6ed6-da3f368a04a2} {Suffrage:Voter ID:server-2d8fdcf4-2f65-3875-1d35-1aa189a46717 Address:2d8fdcf4-2f65-3875-1d35-1aa189a46717} {Suffrage:Voter ID:server-6a2ce9c8-a261-6430-0d7b-897efc824945 Address:6a2ce9c8-a261-6430-0d7b-897efc824945}]"
2025-07-14T05:39:22.324Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:22.324Z [INFO]  server-6a2ce9c8-a261-6430-0d7b-897efc824945: entering follower state: follower="Node at 6a2ce9c8-a261-6430-0d7b-897efc824945 [Follower]" leader-address= leader-id=
2025-07-14T05:39:22.324Z [INFO]  server-2d8fdcf4-2f65-3875-1d35-1aa189a46717: entering follower state: follower="Node at 2d8fdcf4-2f65-3875-1d35-1aa189a46717 [Follower]" leader-address= leader-id=
2025-07-14T05:39:22.910Z [WARN]  server-64284921-b081-1499-6ed6-da3f368a04a2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:22.910Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: entering candidate state: node="Node at 64284921-b081-1499-6ed6-da3f368a04a2 [Candidate]" term=2
2025-07-14T05:39:22.910Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: election won: term=2 tally=2
2025-07-14T05:39:22.910Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: entering leader state: leader="Node at 64284921-b081-1499-6ed6-da3f368a04a2 [Leader]"
2025-07-14T05:39:22.910Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: added peer, starting replication: peer=server-2d8fdcf4-2f65-3875-1d35-1aa189a46717
2025-07-14T05:39:22.910Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: added peer, starting replication: peer=server-6a2ce9c8-a261-6430-0d7b-897efc824945
2025-07-14T05:39:22.910Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: pipelining replication: peer="{Voter server-2d8fdcf4-2f65-3875-1d35-1aa189a46717 2d8fdcf4-2f65-3875-1d35-1aa189a46717}"
2025-07-14T05:39:22.910Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: pipelining replication: peer="{Voter server-6a2ce9c8-a261-6430-0d7b-897efc824945 6a2ce9c8-a261-6430-0d7b-897efc824945}"
2025-07-14T05:39:23.915Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:22.910466203 +0000 UTC m=+62.324926581 (1 nodes), 585.912058ms from start of poll, 586.646219ms from cluster start. Timeout at 2025-07-14 05:39:23.915573745 +0000 UTC m=+63.330034143, 1.005107562s after stability
2025-07-14T05:39:23.917Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: starting snapshot up to: index=12
2025-07-14T05:39:23.917Z [INFO]  creating new snapshot: path=/tmp/raft3894075018/snapshots/2-12-1752471563917.tmp
2025-07-14T05:39:24.005Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: snapshot complete up to: index=12
2025-07-14T05:39:24.006Z [INFO]  server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6 Address:8c9aab11-b6aa-ea30-b845-a1b8f30e59e6} {Suffrage:Voter ID:server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9 Address:4ea5bb60-0745-c377-8d2f-ba6bb70f49f9} {Suffrage:Voter ID:server-cd963528-03ae-b228-6e0a-37f16433b6a3 Address:cd963528-03ae-b228-6e0a-37f16433b6a3}]"
2025-07-14T05:39:24.006Z [INFO]  server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6 Address:8c9aab11-b6aa-ea30-b845-a1b8f30e59e6} {Suffrage:Voter ID:server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9 Address:4ea5bb60-0745-c377-8d2f-ba6bb70f49f9} {Suffrage:Voter ID:server-cd963528-03ae-b228-6e0a-37f16433b6a3 Address:cd963528-03ae-b228-6e0a-37f16433b6a3}]"
2025-07-14T05:39:24.006Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6 Address:8c9aab11-b6aa-ea30-b845-a1b8f30e59e6} {Suffrage:Voter ID:server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9 Address:4ea5bb60-0745-c377-8d2f-ba6bb70f49f9} {Suffrage:Voter ID:server-cd963528-03ae-b228-6e0a-37f16433b6a3 Address:cd963528-03ae-b228-6e0a-37f16433b6a3}]"
2025-07-14T05:39:24.006Z [INFO]  server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6: entering follower state: follower="Node at 8c9aab11-b6aa-ea30-b845-a1b8f30e59e6 [Follower]" leader-address= leader-id=
2025-07-14T05:39:24.006Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: entering follower state: follower="Node at cd963528-03ae-b228-6e0a-37f16433b6a3 [Follower]" leader-address= leader-id=
2025-07-14T05:39:24.006Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:24.006Z [INFO]  server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9: entering follower state: follower="Node at 4ea5bb60-0745-c377-8d2f-ba6bb70f49f9 [Follower]" leader-address= leader-id=
2025-07-14T05:39:24.577Z [WARN]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:24.577Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: entering candidate state: node="Node at cd963528-03ae-b228-6e0a-37f16433b6a3 [Candidate]" term=2
2025-07-14T05:39:24.577Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: election won: term=2 tally=2
2025-07-14T05:39:24.577Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: entering leader state: leader="Node at cd963528-03ae-b228-6e0a-37f16433b6a3 [Leader]"
2025-07-14T05:39:24.577Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: added peer, starting replication: peer=server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6
2025-07-14T05:39:24.577Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: added peer, starting replication: peer=server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9
2025-07-14T05:39:24.577Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: pipelining replication: peer="{Voter server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9 4ea5bb60-0745-c377-8d2f-ba6bb70f49f9}"
2025-07-14T05:39:24.577Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: pipelining replication: peer="{Voter server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6 8c9aab11-b6aa-ea30-b845-a1b8f30e59e6}"
2025-07-14T05:39:25.583Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:24.577815457 +0000 UTC m=+63.992275835 (1 nodes), 570.834682ms from start of poll, 571.157105ms from cluster start. Timeout at 2025-07-14 05:39:25.58337053 +0000 UTC m=+64.997830918, 1.005555083s after stability
2025-07-14T05:39:25.583Z [INFO]  creating new snapshot: path=/tmp/raft2395538624/snapshots/2-113-1752471565583.tmp
2025-07-14T05:39:25.690Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: copied to local snapshot: bytes=71
2025-07-14T05:39:25.692Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: snapshot restore progress: id=2-113-1752471565583 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:25.692Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: restored user snapshot: index=113
2025-07-14T05:39:25.692Z [ERROR] server-cd963528-03ae-b228-6e0a-37f16433b6a3: failed to get log: index=3 error="log not found"
2025-07-14T05:39:25.692Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: aborting pipeline replication: peer="{Voter server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6 8c9aab11-b6aa-ea30-b845-a1b8f30e59e6}"
2025-07-14T05:39:25.692Z [ERROR] server-cd963528-03ae-b228-6e0a-37f16433b6a3: failed to get log: index=3 error="log not found"
2025-07-14T05:39:25.692Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: aborting pipeline replication: peer="{Voter server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9 4ea5bb60-0745-c377-8d2f-ba6bb70f49f9}"
2025-07-14T05:39:25.701Z [ERROR] server-cd963528-03ae-b228-6e0a-37f16433b6a3: failed to get log: index=3 error="log not found"
2025-07-14T05:39:25.701Z [INFO]  creating new snapshot: path=/tmp/raft2558215022/snapshots/2-113-1752471565701.tmp
2025-07-14T05:39:25.701Z [INFO]  server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:25.701Z [INFO]  server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6: copied to local snapshot: bytes=71
2025-07-14T05:39:25.701Z [INFO]  server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6: snapshot restore progress: id=2-113-1752471565701 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:25.701Z [INFO]  server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6: Installed remote snapshot
2025-07-14T05:39:25.702Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: pipelining replication: peer="{Voter server-8c9aab11-b6aa-ea30-b845-a1b8f30e59e6 8c9aab11-b6aa-ea30-b845-a1b8f30e59e6}"
2025-07-14T05:39:25.702Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:25.702Z [ERROR] server-cd963528-03ae-b228-6e0a-37f16433b6a3: failed to get log: index=3 error="log not found"
2025-07-14T05:39:25.702Z [INFO]  creating new snapshot: path=/tmp/raft3655258854/snapshots/2-113-1752471565702.tmp
2025-07-14T05:39:25.702Z [INFO]  server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:25.702Z [INFO]  server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9: copied to local snapshot: bytes=71
2025-07-14T05:39:25.703Z [INFO]  server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9: snapshot restore progress: id=2-113-1752471565702 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:25.703Z [INFO]  server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9: Installed remote snapshot
2025-07-14T05:39:25.703Z [INFO]  server-cd963528-03ae-b228-6e0a-37f16433b6a3: pipelining replication: peer="{Voter server-4ea5bb60-0745-c377-8d2f-ba6bb70f49f9 4ea5bb60-0745-c377-8d2f-ba6bb70f49f9}"
2025-07-14T05:39:26.707Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:25.702071992 +0000 UTC m=+65.116532380 (2 nodes), 841ns from start of poll, 1.69541365s from cluster start. Timeout at 2025-07-14 05:39:26.707860514 +0000 UTC m=+66.122320912, 1.005788532s after stability
2025-07-14T05:39:26.719Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: aborting pipeline replication: peer="{Voter server-6a2ce9c8-a261-6430-0d7b-897efc824945 6a2ce9c8-a261-6430-0d7b-897efc824945}"
2025-07-14T05:39:26.719Z [INFO]  server-64284921-b081-1499-6ed6-da3f368a04a2: aborting pipeline replication: peer="{Voter server-2d8fdcf4-2f65-3875-1d35-1aa189a46717 2d8fdcf4-2f65-3875-1d35-1aa189a46717}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-07-14T05:39:26.720Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91a7d9e5-d512-54bd-e493-f8900f1a358f Address:91a7d9e5-d512-54bd-e493-f8900f1a358f} {Suffrage:Voter ID:server-0cdb6082-2ff0-fc01-7962-64441e4b19f3 Address:0cdb6082-2ff0-fc01-7962-64441e4b19f3} {Suffrage:Voter ID:server-2a04d874-6c23-0a12-44d9-10df3d830650 Address:2a04d874-6c23-0a12-44d9-10df3d830650}]"
2025-07-14T05:39:26.720Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: entering follower state: follower="Node at 91a7d9e5-d512-54bd-e493-f8900f1a358f [Follower]" leader-address= leader-id=
2025-07-14T05:39:26.720Z [INFO]  server-0cdb6082-2ff0-fc01-7962-64441e4b19f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91a7d9e5-d512-54bd-e493-f8900f1a358f Address:91a7d9e5-d512-54bd-e493-f8900f1a358f} {Suffrage:Voter ID:server-0cdb6082-2ff0-fc01-7962-64441e4b19f3 Address:0cdb6082-2ff0-fc01-7962-64441e4b19f3} {Suffrage:Voter ID:server-2a04d874-6c23-0a12-44d9-10df3d830650 Address:2a04d874-6c23-0a12-44d9-10df3d830650}]"
2025-07-14T05:39:26.720Z [INFO]  server-0cdb6082-2ff0-fc01-7962-64441e4b19f3: entering follower state: follower="Node at 0cdb6082-2ff0-fc01-7962-64441e4b19f3 [Follower]" leader-address= leader-id=
2025-07-14T05:39:26.721Z [INFO]  server-2a04d874-6c23-0a12-44d9-10df3d830650: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91a7d9e5-d512-54bd-e493-f8900f1a358f Address:91a7d9e5-d512-54bd-e493-f8900f1a358f} {Suffrage:Voter ID:server-0cdb6082-2ff0-fc01-7962-64441e4b19f3 Address:0cdb6082-2ff0-fc01-7962-64441e4b19f3} {Suffrage:Voter ID:server-2a04d874-6c23-0a12-44d9-10df3d830650 Address:2a04d874-6c23-0a12-44d9-10df3d830650}]"
2025-07-14T05:39:26.721Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:26.721Z [INFO]  server-2a04d874-6c23-0a12-44d9-10df3d830650: entering follower state: follower="Node at 2a04d874-6c23-0a12-44d9-10df3d830650 [Follower]" leader-address= leader-id=
2025-07-14T05:39:27.303Z [WARN]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:27.303Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: entering candidate state: node="Node at 91a7d9e5-d512-54bd-e493-f8900f1a358f [Candidate]" term=2
2025-07-14T05:39:27.303Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: election won: term=2 tally=2
2025-07-14T05:39:27.303Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: entering leader state: leader="Node at 91a7d9e5-d512-54bd-e493-f8900f1a358f [Leader]"
2025-07-14T05:39:27.303Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: added peer, starting replication: peer=server-0cdb6082-2ff0-fc01-7962-64441e4b19f3
2025-07-14T05:39:27.303Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: added peer, starting replication: peer=server-2a04d874-6c23-0a12-44d9-10df3d830650
2025-07-14T05:39:27.303Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: pipelining replication: peer="{Voter server-2a04d874-6c23-0a12-44d9-10df3d830650 2a04d874-6c23-0a12-44d9-10df3d830650}"
2025-07-14T05:39:27.303Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: pipelining replication: peer="{Voter server-0cdb6082-2ff0-fc01-7962-64441e4b19f3 0cdb6082-2ff0-fc01-7962-64441e4b19f3}"
2025-07-14T05:39:28.308Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:27.303599102 +0000 UTC m=+66.718059470 (1 nodes), 582.568132ms from start of poll, 582.843054ms from cluster start. Timeout at 2025-07-14 05:39:28.308708486 +0000 UTC m=+67.723168884, 1.005109414s after stability
2025-07-14T05:39:28.309Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: starting snapshot up to: index=12
2025-07-14T05:39:28.309Z [INFO]  creating new snapshot: path=/tmp/raft3548878784/snapshots/2-12-1752471568309.tmp
2025-07-14T05:39:28.309Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: snapshot complete up to: index=12
2025-07-14T05:39:28.310Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad Address:f74d1e58-37ca-1e0e-ed9e-3d73337a99ad} {Suffrage:Voter ID:server-afdf13b6-2812-66ea-231c-fec6a5eed810 Address:afdf13b6-2812-66ea-231c-fec6a5eed810} {Suffrage:Voter ID:server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe Address:82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe}]"
2025-07-14T05:39:28.310Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad Address:f74d1e58-37ca-1e0e-ed9e-3d73337a99ad} {Suffrage:Voter ID:server-afdf13b6-2812-66ea-231c-fec6a5eed810 Address:afdf13b6-2812-66ea-231c-fec6a5eed810} {Suffrage:Voter ID:server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe Address:82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe}]"
2025-07-14T05:39:28.310Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: entering follower state: follower="Node at f74d1e58-37ca-1e0e-ed9e-3d73337a99ad [Follower]" leader-address= leader-id=
2025-07-14T05:39:28.310Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: entering follower state: follower="Node at afdf13b6-2812-66ea-231c-fec6a5eed810 [Follower]" leader-address= leader-id=
2025-07-14T05:39:28.310Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad Address:f74d1e58-37ca-1e0e-ed9e-3d73337a99ad} {Suffrage:Voter ID:server-afdf13b6-2812-66ea-231c-fec6a5eed810 Address:afdf13b6-2812-66ea-231c-fec6a5eed810} {Suffrage:Voter ID:server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe Address:82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe}]"
2025-07-14T05:39:28.310Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:28.310Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: entering follower state: follower="Node at 82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe [Follower]" leader-address= leader-id=
2025-07-14T05:39:29.013Z [WARN]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:29.013Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: entering candidate state: node="Node at 82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe [Candidate]" term=2
2025-07-14T05:39:29.013Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: election won: term=2 tally=2
2025-07-14T05:39:29.013Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: entering leader state: leader="Node at 82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe [Leader]"
2025-07-14T05:39:29.013Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: added peer, starting replication: peer=server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad
2025-07-14T05:39:29.013Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: added peer, starting replication: peer=server-afdf13b6-2812-66ea-231c-fec6a5eed810
2025-07-14T05:39:29.013Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: pipelining replication: peer="{Voter server-afdf13b6-2812-66ea-231c-fec6a5eed810 afdf13b6-2812-66ea-231c-fec6a5eed810}"
2025-07-14T05:39:29.013Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: pipelining replication: peer="{Voter server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad f74d1e58-37ca-1e0e-ed9e-3d73337a99ad}"
2025-07-14T05:39:30.018Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:29.013536298 +0000 UTC m=+68.427996676 (1 nodes), 702.711197ms from start of poll, 703.014903ms from cluster start. Timeout at 2025-07-14 05:39:30.018839452 +0000 UTC m=+69.433299820, 1.005303144s after stability
2025-07-14T05:39:30.019Z [INFO]  creating new snapshot: path=/tmp/raft487560096/snapshots/2-113-1752471570019.tmp
2025-07-14T05:39:30.040Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: copied to local snapshot: bytes=71
2025-07-14T05:39:30.041Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: snapshot restore progress: id=2-113-1752471570019 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:30.041Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: removing all old logs from log store
2025-07-14T05:39:30.041Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: compacting logs: from=1 to=2
2025-07-14T05:39:30.041Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: restored user snapshot: index=113
2025-07-14T05:39:30.041Z [ERROR] server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: failed to get log: index=2 error="log not found"
2025-07-14T05:39:30.041Z [ERROR] server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: failed to get log: index=2 error="log not found"
2025-07-14T05:39:30.041Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: aborting pipeline replication: peer="{Voter server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad f74d1e58-37ca-1e0e-ed9e-3d73337a99ad}"
2025-07-14T05:39:30.041Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: aborting pipeline replication: peer="{Voter server-afdf13b6-2812-66ea-231c-fec6a5eed810 afdf13b6-2812-66ea-231c-fec6a5eed810}"
2025-07-14T05:39:30.050Z [ERROR] server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: failed to get log: index=2 error="log not found"
2025-07-14T05:39:30.050Z [INFO]  creating new snapshot: path=/tmp/raft1195389667/snapshots/2-113-1752471570050.tmp
2025-07-14T05:39:30.050Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:30.051Z [ERROR] server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: failed to get log: index=2 error="log not found"
2025-07-14T05:39:30.051Z [INFO]  creating new snapshot: path=/tmp/raft458029533/snapshots/2-113-1752471570051.tmp
2025-07-14T05:39:30.052Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:30.133Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: copied to local snapshot: bytes=71
2025-07-14T05:39:30.134Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: snapshot restore progress: id=2-113-1752471570050 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:30.134Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: removing all old logs from log store
2025-07-14T05:39:30.134Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: compacting logs: from=1 to=2
2025-07-14T05:39:30.134Z [INFO]  server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad: Installed remote snapshot
2025-07-14T05:39:30.134Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: pipelining replication: peer="{Voter server-f74d1e58-37ca-1e0e-ed9e-3d73337a99ad f74d1e58-37ca-1e0e-ed9e-3d73337a99ad}"
2025-07-14T05:39:30.134Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:30.136Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: copied to local snapshot: bytes=71
2025-07-14T05:39:30.136Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: snapshot restore progress: id=2-113-1752471570051 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:30.136Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: removing all old logs from log store
2025-07-14T05:39:30.136Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: compacting logs: from=1 to=2
2025-07-14T05:39:30.136Z [INFO]  server-afdf13b6-2812-66ea-231c-fec6a5eed810: Installed remote snapshot
2025-07-14T05:39:30.136Z [INFO]  server-82942bd7-f85a-b8ae-fc9a-e52ccf2a80fe: pipelining replication: peer="{Voter server-afdf13b6-2812-66ea-231c-fec6a5eed810 afdf13b6-2812-66ea-231c-fec6a5eed810}"
2025-07-14T05:39:31.140Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:30.134592281 +0000 UTC m=+69.549052650 (2 nodes), 912ns from start of poll, 1.824070877s from cluster start. Timeout at 2025-07-14 05:39:31.140063678 +0000 UTC m=+70.554524056, 1.005471406s after stability
2025-07-14T05:39:31.154Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: aborting pipeline replication: peer="{Voter server-2a04d874-6c23-0a12-44d9-10df3d830650 2a04d874-6c23-0a12-44d9-10df3d830650}"
2025-07-14T05:39:31.154Z [INFO]  server-91a7d9e5-d512-54bd-e493-f8900f1a358f: aborting pipeline replication: peer="{Voter server-0cdb6082-2ff0-fc01-7962-64441e4b19f3 0cdb6082-2ff0-fc01-7962-64441e4b19f3}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-07-14T05:39:31.155Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4 Address:a5941047-4adc-5198-9cc5-9afdf6d2fbb4} {Suffrage:Voter ID:server-5cabe16e-24c1-5922-b118-9340a1669a73 Address:5cabe16e-24c1-5922-b118-9340a1669a73} {Suffrage:Voter ID:server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 Address:4ff196d7-038b-d4e5-dbad-eb7ea5d042a6}]"
2025-07-14T05:39:31.155Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: entering follower state: follower="Node at a5941047-4adc-5198-9cc5-9afdf6d2fbb4 [Follower]" leader-address= leader-id=
2025-07-14T05:39:31.156Z [INFO]  server-5cabe16e-24c1-5922-b118-9340a1669a73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4 Address:a5941047-4adc-5198-9cc5-9afdf6d2fbb4} {Suffrage:Voter ID:server-5cabe16e-24c1-5922-b118-9340a1669a73 Address:5cabe16e-24c1-5922-b118-9340a1669a73} {Suffrage:Voter ID:server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 Address:4ff196d7-038b-d4e5-dbad-eb7ea5d042a6}]"
2025-07-14T05:39:31.156Z [INFO]  server-5cabe16e-24c1-5922-b118-9340a1669a73: entering follower state: follower="Node at 5cabe16e-24c1-5922-b118-9340a1669a73 [Follower]" leader-address= leader-id=
2025-07-14T05:39:31.156Z [INFO]  server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4 Address:a5941047-4adc-5198-9cc5-9afdf6d2fbb4} {Suffrage:Voter ID:server-5cabe16e-24c1-5922-b118-9340a1669a73 Address:5cabe16e-24c1-5922-b118-9340a1669a73} {Suffrage:Voter ID:server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 Address:4ff196d7-038b-d4e5-dbad-eb7ea5d042a6}]"
2025-07-14T05:39:31.156Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:31.156Z [INFO]  server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6: entering follower state: follower="Node at 4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 [Follower]" leader-address= leader-id=
2025-07-14T05:39:31.685Z [WARN]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:31.685Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: entering candidate state: node="Node at a5941047-4adc-5198-9cc5-9afdf6d2fbb4 [Candidate]" term=2
2025-07-14T05:39:31.685Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: election won: term=2 tally=2
2025-07-14T05:39:31.685Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: entering leader state: leader="Node at a5941047-4adc-5198-9cc5-9afdf6d2fbb4 [Leader]"
2025-07-14T05:39:31.688Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: added peer, starting replication: peer=server-5cabe16e-24c1-5922-b118-9340a1669a73
2025-07-14T05:39:31.688Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: added peer, starting replication: peer=server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6
2025-07-14T05:39:31.688Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: pipelining replication: peer="{Voter server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 4ff196d7-038b-d4e5-dbad-eb7ea5d042a6}"
2025-07-14T05:39:31.688Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: pipelining replication: peer="{Voter server-5cabe16e-24c1-5922-b118-9340a1669a73 5cabe16e-24c1-5922-b118-9340a1669a73}"
2025-07-14T05:39:32.693Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:31.688229697 +0000 UTC m=+71.102690065 (1 nodes), 531.797516ms from start of poll, 532.517175ms from cluster start. Timeout at 2025-07-14 05:39:32.693235055 +0000 UTC m=+72.107695453, 1.005005388s after stability
2025-07-14T05:39:32.693Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: starting snapshot up to: index=12
2025-07-14T05:39:32.693Z [INFO]  creating new snapshot: path=/tmp/raft2578539787/snapshots/2-12-1752471572693.tmp
2025-07-14T05:39:32.721Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: snapshot complete up to: index=12
2025-07-14T05:39:32.721Z [INFO]  creating new snapshot: path=/tmp/raft2578539787/snapshots/2-1013-1752471572721.tmp
2025-07-14T05:39:32.722Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: copied to local snapshot: bytes=71
2025-07-14T05:39:32.722Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: snapshot restore progress: id=2-1013-1752471572721 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:32.722Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: restored user snapshot: index=1013
2025-07-14T05:39:32.722Z [ERROR] server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:32.722Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: aborting pipeline replication: peer="{Voter server-5cabe16e-24c1-5922-b118-9340a1669a73 5cabe16e-24c1-5922-b118-9340a1669a73}"
2025-07-14T05:39:32.722Z [ERROR] server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:32.722Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: aborting pipeline replication: peer="{Voter server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 4ff196d7-038b-d4e5-dbad-eb7ea5d042a6}"
2025-07-14T05:39:32.722Z [ERROR] server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:32.722Z [INFO]  creating new snapshot: path=/tmp/raft3618491468/snapshots/2-1013-1752471572722.tmp
2025-07-14T05:39:32.722Z [INFO]  server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:32.722Z [INFO]  server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6: copied to local snapshot: bytes=71
2025-07-14T05:39:32.730Z [ERROR] server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:32.731Z [INFO]  server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6: snapshot restore progress: id=2-1013-1752471572722 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:32.731Z [INFO]  server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6: Installed remote snapshot
2025-07-14T05:39:32.731Z [WARN]  server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6: skipping application of old log: index=22
2025-07-14T05:39:32.732Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: pipelining replication: peer="{Voter server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 4ff196d7-038b-d4e5-dbad-eb7ea5d042a6}"
2025-07-14T05:39:32.732Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:32.736Z [INFO]  creating new snapshot: path=/tmp/raft3563403251/snapshots/2-1013-1752471572736.tmp
2025-07-14T05:39:32.829Z [INFO]  server-5cabe16e-24c1-5922-b118-9340a1669a73: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:32.830Z [INFO]  server-5cabe16e-24c1-5922-b118-9340a1669a73: copied to local snapshot: bytes=71
2025-07-14T05:39:32.830Z [INFO]  server-5cabe16e-24c1-5922-b118-9340a1669a73: snapshot restore progress: id=2-1013-1752471572736 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:32.830Z [INFO]  server-5cabe16e-24c1-5922-b118-9340a1669a73: Installed remote snapshot
2025-07-14T05:39:32.830Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: pipelining replication: peer="{Voter server-5cabe16e-24c1-5922-b118-9340a1669a73 5cabe16e-24c1-5922-b118-9340a1669a73}"
2025-07-14T05:39:33.738Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:32.732525902 +0000 UTC m=+72.146986280 (2 nodes), 691ns from start of poll, 1.57681339s from cluster start. Timeout at 2025-07-14 05:39:33.738249056 +0000 UTC m=+73.152709424, 1.005723144s after stability
2025-07-14T05:39:33.754Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: aborting pipeline replication: peer="{Voter server-4ff196d7-038b-d4e5-dbad-eb7ea5d042a6 4ff196d7-038b-d4e5-dbad-eb7ea5d042a6}"
2025-07-14T05:39:33.754Z [INFO]  server-a5941047-4adc-5198-9cc5-9afdf6d2fbb4: aborting pipeline replication: peer="{Voter server-5cabe16e-24c1-5922-b118-9340a1669a73 5cabe16e-24c1-5922-b118-9340a1669a73}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-07-14T05:39:33.756Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea Address:ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea} {Suffrage:Voter ID:server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78 Address:aae8b3fa-58de-1fd0-13c6-c047e55f1f78} {Suffrage:Voter ID:server-4f53451d-6b14-4428-dc22-bc2224a27851 Address:4f53451d-6b14-4428-dc22-bc2224a27851}]"
2025-07-14T05:39:33.756Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: entering follower state: follower="Node at ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea [Follower]" leader-address= leader-id=
2025-07-14T05:39:33.757Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea Address:ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea} {Suffrage:Voter ID:server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78 Address:aae8b3fa-58de-1fd0-13c6-c047e55f1f78} {Suffrage:Voter ID:server-4f53451d-6b14-4428-dc22-bc2224a27851 Address:4f53451d-6b14-4428-dc22-bc2224a27851}]"
2025-07-14T05:39:33.757Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: entering follower state: follower="Node at aae8b3fa-58de-1fd0-13c6-c047e55f1f78 [Follower]" leader-address= leader-id=
2025-07-14T05:39:33.757Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea Address:ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea} {Suffrage:Voter ID:server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78 Address:aae8b3fa-58de-1fd0-13c6-c047e55f1f78} {Suffrage:Voter ID:server-4f53451d-6b14-4428-dc22-bc2224a27851 Address:4f53451d-6b14-4428-dc22-bc2224a27851}]"
2025-07-14T05:39:33.757Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:33.757Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: entering follower state: follower="Node at 4f53451d-6b14-4428-dc22-bc2224a27851 [Follower]" leader-address= leader-id=
2025-07-14T05:39:34.322Z [WARN]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:34.322Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: entering candidate state: node="Node at ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea [Candidate]" term=2
2025-07-14T05:39:34.322Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: election won: term=2 tally=2
2025-07-14T05:39:34.322Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: entering leader state: leader="Node at ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea [Leader]"
2025-07-14T05:39:34.322Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: added peer, starting replication: peer=server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78
2025-07-14T05:39:34.322Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: added peer, starting replication: peer=server-4f53451d-6b14-4428-dc22-bc2224a27851
2025-07-14T05:39:34.322Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: pipelining replication: peer="{Voter server-4f53451d-6b14-4428-dc22-bc2224a27851 4f53451d-6b14-4428-dc22-bc2224a27851}"
2025-07-14T05:39:34.322Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: pipelining replication: peer="{Voter server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78 aae8b3fa-58de-1fd0-13c6-c047e55f1f78}"
2025-07-14T05:39:35.328Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:34.32236493 +0000 UTC m=+73.736825298 (1 nodes), 565.10001ms from start of poll, 565.744797ms from cluster start. Timeout at 2025-07-14 05:39:35.328925831 +0000 UTC m=+74.743386219, 1.006560921s after stability
2025-07-14T05:39:35.329Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: starting snapshot up to: index=12
2025-07-14T05:39:35.329Z [INFO]  creating new snapshot: path=/tmp/raft3768769256/snapshots/2-12-1752471575329.tmp
2025-07-14T05:39:35.330Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: snapshot complete up to: index=12
2025-07-14T05:39:35.330Z [INFO]  creating new snapshot: path=/tmp/raft3768769256/snapshots/2-1013-1752471575330.tmp
2025-07-14T05:39:35.331Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: copied to local snapshot: bytes=71
2025-07-14T05:39:35.331Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: snapshot restore progress: id=2-1013-1752471575330 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:35.331Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: removing all old logs from log store
2025-07-14T05:39:35.331Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: compacting logs: from=1 to=22
2025-07-14T05:39:35.331Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: restored user snapshot: index=1013
2025-07-14T05:39:35.331Z [ERROR] server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: failed to get log: index=22 error="log not found"
2025-07-14T05:39:35.332Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: aborting pipeline replication: peer="{Voter server-4f53451d-6b14-4428-dc22-bc2224a27851 4f53451d-6b14-4428-dc22-bc2224a27851}"
2025-07-14T05:39:35.332Z [ERROR] server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: failed to get log: index=22 error="log not found"
2025-07-14T05:39:35.332Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: aborting pipeline replication: peer="{Voter server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78 aae8b3fa-58de-1fd0-13c6-c047e55f1f78}"
2025-07-14T05:39:35.340Z [ERROR] server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: failed to get log: index=22 error="log not found"
2025-07-14T05:39:35.341Z [INFO]  creating new snapshot: path=/tmp/raft3410556581/snapshots/2-1013-1752471575341.tmp
2025-07-14T05:39:35.341Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:35.341Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: copied to local snapshot: bytes=71
2025-07-14T05:39:35.341Z [ERROR] server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: failed to get log: index=22 error="log not found"
2025-07-14T05:39:35.342Z [INFO]  creating new snapshot: path=/tmp/raft3410915839/snapshots/2-1013-1752471575342.tmp
2025-07-14T05:39:35.342Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:35.342Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: copied to local snapshot: bytes=71
2025-07-14T05:39:35.342Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: snapshot restore progress: id=2-1013-1752471575342 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:35.342Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: removing all old logs from log store
2025-07-14T05:39:35.342Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: compacting logs: from=1 to=22
2025-07-14T05:39:35.342Z [INFO]  server-4f53451d-6b14-4428-dc22-bc2224a27851: Installed remote snapshot
2025-07-14T05:39:35.342Z [WARN]  server-4f53451d-6b14-4428-dc22-bc2224a27851: skipping application of old log: index=22
2025-07-14T05:39:35.342Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: pipelining replication: peer="{Voter server-4f53451d-6b14-4428-dc22-bc2224a27851 4f53451d-6b14-4428-dc22-bc2224a27851}"
2025-07-14T05:39:35.342Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: snapshot restore progress: id=2-1013-1752471575341 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:35.342Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: removing all old logs from log store
2025-07-14T05:39:35.342Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: compacting logs: from=1 to=22
2025-07-14T05:39:35.342Z [INFO]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: Installed remote snapshot
2025-07-14T05:39:35.342Z [WARN]  server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78: skipping application of old log: index=22
2025-07-14T05:39:35.342Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: pipelining replication: peer="{Voter server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78 aae8b3fa-58de-1fd0-13c6-c047e55f1f78}"
2025-07-14T05:39:35.342Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:36.348Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:35.342993753 +0000 UTC m=+74.757454131 (2 nodes), 941ns from start of poll, 1.58637363s from cluster start. Timeout at 2025-07-14 05:39:36.348622947 +0000 UTC m=+75.763083315, 1.005629184s after stability
2025-07-14T05:39:36.362Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: aborting pipeline replication: peer="{Voter server-aae8b3fa-58de-1fd0-13c6-c047e55f1f78 aae8b3fa-58de-1fd0-13c6-c047e55f1f78}"
2025-07-14T05:39:36.362Z [INFO]  server-ee471659-c0ab-ddbd-ce16-fa3e4c3b71ea: aborting pipeline replication: peer="{Voter server-4f53451d-6b14-4428-dc22-bc2224a27851 4f53451d-6b14-4428-dc22-bc2224a27851}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-07-14T05:39:36.363Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b782aa-316d-adb5-de9a-173a6e7edff9 Address:20b782aa-316d-adb5-de9a-173a6e7edff9} {Suffrage:Voter ID:server-45c5db37-fa0e-1cd5-172d-316b508e8f45 Address:45c5db37-fa0e-1cd5-172d-316b508e8f45} {Suffrage:Voter ID:server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0 Address:c2dabdc1-0849-a87f-8fe5-b068acd14ad0}]"
2025-07-14T05:39:36.363Z [INFO]  server-45c5db37-fa0e-1cd5-172d-316b508e8f45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b782aa-316d-adb5-de9a-173a6e7edff9 Address:20b782aa-316d-adb5-de9a-173a6e7edff9} {Suffrage:Voter ID:server-45c5db37-fa0e-1cd5-172d-316b508e8f45 Address:45c5db37-fa0e-1cd5-172d-316b508e8f45} {Suffrage:Voter ID:server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0 Address:c2dabdc1-0849-a87f-8fe5-b068acd14ad0}]"
2025-07-14T05:39:36.363Z [INFO]  server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20b782aa-316d-adb5-de9a-173a6e7edff9 Address:20b782aa-316d-adb5-de9a-173a6e7edff9} {Suffrage:Voter ID:server-45c5db37-fa0e-1cd5-172d-316b508e8f45 Address:45c5db37-fa0e-1cd5-172d-316b508e8f45} {Suffrage:Voter ID:server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0 Address:c2dabdc1-0849-a87f-8fe5-b068acd14ad0}]"
2025-07-14T05:39:36.363Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:36.363Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: entering follower state: follower="Node at 20b782aa-316d-adb5-de9a-173a6e7edff9 [Follower]" leader-address= leader-id=
2025-07-14T05:39:36.363Z [INFO]  server-45c5db37-fa0e-1cd5-172d-316b508e8f45: entering follower state: follower="Node at 45c5db37-fa0e-1cd5-172d-316b508e8f45 [Follower]" leader-address= leader-id=
2025-07-14T05:39:36.363Z [INFO]  server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0: entering follower state: follower="Node at c2dabdc1-0849-a87f-8fe5-b068acd14ad0 [Follower]" leader-address= leader-id=
2025-07-14T05:39:37.076Z [WARN]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:37.076Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: entering candidate state: node="Node at 20b782aa-316d-adb5-de9a-173a6e7edff9 [Candidate]" term=2
2025-07-14T05:39:37.076Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: election won: term=2 tally=2
2025-07-14T05:39:37.076Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: entering leader state: leader="Node at 20b782aa-316d-adb5-de9a-173a6e7edff9 [Leader]"
2025-07-14T05:39:37.076Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: added peer, starting replication: peer=server-45c5db37-fa0e-1cd5-172d-316b508e8f45
2025-07-14T05:39:37.076Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: added peer, starting replication: peer=server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0
2025-07-14T05:39:37.076Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: pipelining replication: peer="{Voter server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0 c2dabdc1-0849-a87f-8fe5-b068acd14ad0}"
2025-07-14T05:39:37.076Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: pipelining replication: peer="{Voter server-45c5db37-fa0e-1cd5-172d-316b508e8f45 45c5db37-fa0e-1cd5-172d-316b508e8f45}"
2025-07-14T05:39:38.081Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:37.076788104 +0000 UTC m=+76.491248472 (1 nodes), 713.062013ms from start of poll, 713.292739ms from cluster start. Timeout at 2025-07-14 05:39:38.081844038 +0000 UTC m=+77.496304406, 1.005055934s after stability
2025-07-14T05:39:38.082Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: starting snapshot up to: index=12
2025-07-14T05:39:38.082Z [INFO]  creating new snapshot: path=/tmp/raft3126612231/snapshots/2-12-1752471578082.tmp
2025-07-14T05:39:38.083Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: snapshot complete up to: index=12
2025-07-14T05:39:38.084Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a2b1e88-8303-c429-406d-5905aa40b3e6 Address:1a2b1e88-8303-c429-406d-5905aa40b3e6} {Suffrage:Voter ID:server-65283db9-ec26-dd98-1328-395f9ecc405b Address:65283db9-ec26-dd98-1328-395f9ecc405b} {Suffrage:Voter ID:server-22aafa5d-f0a7-aa24-981c-11760d72ce23 Address:22aafa5d-f0a7-aa24-981c-11760d72ce23}]"
2025-07-14T05:39:38.084Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: entering follower state: follower="Node at 1a2b1e88-8303-c429-406d-5905aa40b3e6 [Follower]" leader-address= leader-id=
2025-07-14T05:39:38.084Z [INFO]  server-65283db9-ec26-dd98-1328-395f9ecc405b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a2b1e88-8303-c429-406d-5905aa40b3e6 Address:1a2b1e88-8303-c429-406d-5905aa40b3e6} {Suffrage:Voter ID:server-65283db9-ec26-dd98-1328-395f9ecc405b Address:65283db9-ec26-dd98-1328-395f9ecc405b} {Suffrage:Voter ID:server-22aafa5d-f0a7-aa24-981c-11760d72ce23 Address:22aafa5d-f0a7-aa24-981c-11760d72ce23}]"
2025-07-14T05:39:38.084Z [INFO]  server-65283db9-ec26-dd98-1328-395f9ecc405b: entering follower state: follower="Node at 65283db9-ec26-dd98-1328-395f9ecc405b [Follower]" leader-address= leader-id=
2025-07-14T05:39:38.084Z [INFO]  server-22aafa5d-f0a7-aa24-981c-11760d72ce23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a2b1e88-8303-c429-406d-5905aa40b3e6 Address:1a2b1e88-8303-c429-406d-5905aa40b3e6} {Suffrage:Voter ID:server-65283db9-ec26-dd98-1328-395f9ecc405b Address:65283db9-ec26-dd98-1328-395f9ecc405b} {Suffrage:Voter ID:server-22aafa5d-f0a7-aa24-981c-11760d72ce23 Address:22aafa5d-f0a7-aa24-981c-11760d72ce23}]"
2025-07-14T05:39:38.084Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:38.084Z [INFO]  server-22aafa5d-f0a7-aa24-981c-11760d72ce23: entering follower state: follower="Node at 22aafa5d-f0a7-aa24-981c-11760d72ce23 [Follower]" leader-address= leader-id=
2025-07-14T05:39:38.604Z [WARN]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:38.604Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: entering candidate state: node="Node at 1a2b1e88-8303-c429-406d-5905aa40b3e6 [Candidate]" term=2
2025-07-14T05:39:38.604Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: election won: term=2 tally=2
2025-07-14T05:39:38.604Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: entering leader state: leader="Node at 1a2b1e88-8303-c429-406d-5905aa40b3e6 [Leader]"
2025-07-14T05:39:38.604Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: added peer, starting replication: peer=server-65283db9-ec26-dd98-1328-395f9ecc405b
2025-07-14T05:39:38.604Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: added peer, starting replication: peer=server-22aafa5d-f0a7-aa24-981c-11760d72ce23
2025-07-14T05:39:38.604Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: pipelining replication: peer="{Voter server-22aafa5d-f0a7-aa24-981c-11760d72ce23 22aafa5d-f0a7-aa24-981c-11760d72ce23}"
2025-07-14T05:39:38.604Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: pipelining replication: peer="{Voter server-65283db9-ec26-dd98-1328-395f9ecc405b 65283db9-ec26-dd98-1328-395f9ecc405b}"
2025-07-14T05:39:39.616Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:38.604483327 +0000 UTC m=+78.018943725 (1 nodes), 519.744637ms from start of poll, 520.316214ms from cluster start. Timeout at 2025-07-14 05:39:39.616623387 +0000 UTC m=+79.031083775, 1.01214005s after stability
2025-07-14T05:39:39.616Z [INFO]  creating new snapshot: path=/tmp/raft3296573837/snapshots/2-1013-1752471579616.tmp
2025-07-14T05:39:39.617Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: copied to local snapshot: bytes=71
2025-07-14T05:39:39.617Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: snapshot restore progress: id=2-1013-1752471579616 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:39.617Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: restored user snapshot: index=1013
2025-07-14T05:39:39.617Z [ERROR] server-1a2b1e88-8303-c429-406d-5905aa40b3e6: failed to get log: index=3 error="log not found"
2025-07-14T05:39:39.617Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: aborting pipeline replication: peer="{Voter server-22aafa5d-f0a7-aa24-981c-11760d72ce23 22aafa5d-f0a7-aa24-981c-11760d72ce23}"
2025-07-14T05:39:39.617Z [ERROR] server-1a2b1e88-8303-c429-406d-5905aa40b3e6: failed to get log: index=3 error="log not found"
2025-07-14T05:39:39.617Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: aborting pipeline replication: peer="{Voter server-65283db9-ec26-dd98-1328-395f9ecc405b 65283db9-ec26-dd98-1328-395f9ecc405b}"
2025-07-14T05:39:39.624Z [ERROR] server-1a2b1e88-8303-c429-406d-5905aa40b3e6: failed to get log: index=3 error="log not found"
2025-07-14T05:39:39.624Z [INFO]  creating new snapshot: path=/tmp/raft7603114/snapshots/2-1013-1752471579624.tmp
2025-07-14T05:39:39.624Z [INFO]  server-65283db9-ec26-dd98-1328-395f9ecc405b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:39.624Z [INFO]  server-65283db9-ec26-dd98-1328-395f9ecc405b: copied to local snapshot: bytes=71
2025-07-14T05:39:39.624Z [INFO]  server-65283db9-ec26-dd98-1328-395f9ecc405b: snapshot restore progress: id=2-1013-1752471579624 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:39.625Z [INFO]  server-65283db9-ec26-dd98-1328-395f9ecc405b: Installed remote snapshot
2025-07-14T05:39:39.625Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: pipelining replication: peer="{Voter server-65283db9-ec26-dd98-1328-395f9ecc405b 65283db9-ec26-dd98-1328-395f9ecc405b}"
2025-07-14T05:39:39.625Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:39.626Z [ERROR] server-1a2b1e88-8303-c429-406d-5905aa40b3e6: failed to get log: index=3 error="log not found"
2025-07-14T05:39:39.626Z [INFO]  creating new snapshot: path=/tmp/raft414208905/snapshots/2-1013-1752471579626.tmp
2025-07-14T05:39:39.626Z [INFO]  server-22aafa5d-f0a7-aa24-981c-11760d72ce23: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:39.626Z [INFO]  server-22aafa5d-f0a7-aa24-981c-11760d72ce23: copied to local snapshot: bytes=71
2025-07-14T05:39:39.627Z [INFO]  server-22aafa5d-f0a7-aa24-981c-11760d72ce23: snapshot restore progress: id=2-1013-1752471579626 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:39.627Z [INFO]  server-22aafa5d-f0a7-aa24-981c-11760d72ce23: Installed remote snapshot
2025-07-14T05:39:39.627Z [INFO]  server-1a2b1e88-8303-c429-406d-5905aa40b3e6: pipelining replication: peer="{Voter server-22aafa5d-f0a7-aa24-981c-11760d72ce23 22aafa5d-f0a7-aa24-981c-11760d72ce23}"
2025-07-14T05:39:40.630Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:39.625137498 +0000 UTC m=+79.039597876 (2 nodes), 601ns from start of poll, 1.540970365s from cluster start. Timeout at 2025-07-14 05:39:40.630755955 +0000 UTC m=+80.045216323, 1.005618447s after stability
2025-07-14T05:39:40.642Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: aborting pipeline replication: peer="{Voter server-c2dabdc1-0849-a87f-8fe5-b068acd14ad0 c2dabdc1-0849-a87f-8fe5-b068acd14ad0}"
2025-07-14T05:39:40.642Z [INFO]  server-20b782aa-316d-adb5-de9a-173a6e7edff9: aborting pipeline replication: peer="{Voter server-45c5db37-fa0e-1cd5-172d-316b508e8f45 45c5db37-fa0e-1cd5-172d-316b508e8f45}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-07-14T05:39:40.643Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847 Address:f98bd965-9d3e-22f6-3c63-6bc8bdb55847} {Suffrage:Voter ID:server-091e13c4-0fa3-656b-1d88-a352ed491772 Address:091e13c4-0fa3-656b-1d88-a352ed491772} {Suffrage:Voter ID:server-eaaae432-a4cf-7e29-02ca-701b9d537ebc Address:eaaae432-a4cf-7e29-02ca-701b9d537ebc}]"
2025-07-14T05:39:40.643Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: entering follower state: follower="Node at f98bd965-9d3e-22f6-3c63-6bc8bdb55847 [Follower]" leader-address= leader-id=
2025-07-14T05:39:40.643Z [INFO]  server-091e13c4-0fa3-656b-1d88-a352ed491772: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847 Address:f98bd965-9d3e-22f6-3c63-6bc8bdb55847} {Suffrage:Voter ID:server-091e13c4-0fa3-656b-1d88-a352ed491772 Address:091e13c4-0fa3-656b-1d88-a352ed491772} {Suffrage:Voter ID:server-eaaae432-a4cf-7e29-02ca-701b9d537ebc Address:eaaae432-a4cf-7e29-02ca-701b9d537ebc}]"
2025-07-14T05:39:40.643Z [INFO]  server-091e13c4-0fa3-656b-1d88-a352ed491772: entering follower state: follower="Node at 091e13c4-0fa3-656b-1d88-a352ed491772 [Follower]" leader-address= leader-id=
2025-07-14T05:39:40.643Z [INFO]  server-eaaae432-a4cf-7e29-02ca-701b9d537ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847 Address:f98bd965-9d3e-22f6-3c63-6bc8bdb55847} {Suffrage:Voter ID:server-091e13c4-0fa3-656b-1d88-a352ed491772 Address:091e13c4-0fa3-656b-1d88-a352ed491772} {Suffrage:Voter ID:server-eaaae432-a4cf-7e29-02ca-701b9d537ebc Address:eaaae432-a4cf-7e29-02ca-701b9d537ebc}]"
2025-07-14T05:39:40.643Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:40.643Z [INFO]  server-eaaae432-a4cf-7e29-02ca-701b9d537ebc: entering follower state: follower="Node at eaaae432-a4cf-7e29-02ca-701b9d537ebc [Follower]" leader-address= leader-id=
2025-07-14T05:39:41.214Z [WARN]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:41.214Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: entering candidate state: node="Node at f98bd965-9d3e-22f6-3c63-6bc8bdb55847 [Candidate]" term=2
2025-07-14T05:39:41.214Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: election won: term=2 tally=2
2025-07-14T05:39:41.214Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: entering leader state: leader="Node at f98bd965-9d3e-22f6-3c63-6bc8bdb55847 [Leader]"
2025-07-14T05:39:41.214Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: added peer, starting replication: peer=server-091e13c4-0fa3-656b-1d88-a352ed491772
2025-07-14T05:39:41.214Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: added peer, starting replication: peer=server-eaaae432-a4cf-7e29-02ca-701b9d537ebc
2025-07-14T05:39:41.214Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: pipelining replication: peer="{Voter server-eaaae432-a4cf-7e29-02ca-701b9d537ebc eaaae432-a4cf-7e29-02ca-701b9d537ebc}"
2025-07-14T05:39:41.214Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: pipelining replication: peer="{Voter server-091e13c4-0fa3-656b-1d88-a352ed491772 091e13c4-0fa3-656b-1d88-a352ed491772}"
2025-07-14T05:39:42.219Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:41.214349566 +0000 UTC m=+80.628809964 (1 nodes), 570.662033ms from start of poll, 570.945208ms from cluster start. Timeout at 2025-07-14 05:39:42.219496196 +0000 UTC m=+81.633956574, 1.00514661s after stability
2025-07-14T05:39:42.219Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: starting snapshot up to: index=12
2025-07-14T05:39:42.219Z [INFO]  creating new snapshot: path=/tmp/raft1501792001/snapshots/2-12-1752471582219.tmp
2025-07-14T05:39:42.220Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: snapshot complete up to: index=12
2025-07-14T05:39:42.221Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c058df63-de19-13a4-b2df-1a340a9e6a1f Address:c058df63-de19-13a4-b2df-1a340a9e6a1f} {Suffrage:Voter ID:server-1b3f21f6-8a1c-c872-7d0f-8621572e3119 Address:1b3f21f6-8a1c-c872-7d0f-8621572e3119} {Suffrage:Voter ID:server-bb9d2a65-4352-85fb-6540-fd1580f66423 Address:bb9d2a65-4352-85fb-6540-fd1580f66423}]"
2025-07-14T05:39:42.221Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c058df63-de19-13a4-b2df-1a340a9e6a1f Address:c058df63-de19-13a4-b2df-1a340a9e6a1f} {Suffrage:Voter ID:server-1b3f21f6-8a1c-c872-7d0f-8621572e3119 Address:1b3f21f6-8a1c-c872-7d0f-8621572e3119} {Suffrage:Voter ID:server-bb9d2a65-4352-85fb-6540-fd1580f66423 Address:bb9d2a65-4352-85fb-6540-fd1580f66423}]"
2025-07-14T05:39:42.221Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c058df63-de19-13a4-b2df-1a340a9e6a1f Address:c058df63-de19-13a4-b2df-1a340a9e6a1f} {Suffrage:Voter ID:server-1b3f21f6-8a1c-c872-7d0f-8621572e3119 Address:1b3f21f6-8a1c-c872-7d0f-8621572e3119} {Suffrage:Voter ID:server-bb9d2a65-4352-85fb-6540-fd1580f66423 Address:bb9d2a65-4352-85fb-6540-fd1580f66423}]"
2025-07-14T05:39:42.221Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:42.221Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: entering follower state: follower="Node at c058df63-de19-13a4-b2df-1a340a9e6a1f [Follower]" leader-address= leader-id=
2025-07-14T05:39:42.221Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: entering follower state: follower="Node at 1b3f21f6-8a1c-c872-7d0f-8621572e3119 [Follower]" leader-address= leader-id=
2025-07-14T05:39:42.221Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: entering follower state: follower="Node at bb9d2a65-4352-85fb-6540-fd1580f66423 [Follower]" leader-address= leader-id=
2025-07-14T05:39:42.775Z [WARN]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:42.775Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: entering candidate state: node="Node at c058df63-de19-13a4-b2df-1a340a9e6a1f [Candidate]" term=2
2025-07-14T05:39:42.775Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: election won: term=2 tally=2
2025-07-14T05:39:42.775Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: entering leader state: leader="Node at c058df63-de19-13a4-b2df-1a340a9e6a1f [Leader]"
2025-07-14T05:39:42.775Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: added peer, starting replication: peer=server-1b3f21f6-8a1c-c872-7d0f-8621572e3119
2025-07-14T05:39:42.775Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: added peer, starting replication: peer=server-bb9d2a65-4352-85fb-6540-fd1580f66423
2025-07-14T05:39:42.775Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: pipelining replication: peer="{Voter server-bb9d2a65-4352-85fb-6540-fd1580f66423 bb9d2a65-4352-85fb-6540-fd1580f66423}"
2025-07-14T05:39:42.775Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: pipelining replication: peer="{Voter server-1b3f21f6-8a1c-c872-7d0f-8621572e3119 1b3f21f6-8a1c-c872-7d0f-8621572e3119}"
2025-07-14T05:39:43.781Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:42.775958379 +0000 UTC m=+82.190418757 (1 nodes), 554.598668ms from start of poll, 554.829625ms from cluster start. Timeout at 2025-07-14 05:39:43.781214222 +0000 UTC m=+83.195674620, 1.005255863s after stability
2025-07-14T05:39:43.781Z [INFO]  creating new snapshot: path=/tmp/raft1790788697/snapshots/2-1013-1752471583781.tmp
2025-07-14T05:39:43.781Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: copied to local snapshot: bytes=71
2025-07-14T05:39:43.782Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: snapshot restore progress: id=2-1013-1752471583781 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:43.782Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: removing all old logs from log store
2025-07-14T05:39:43.782Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: compacting logs: from=1 to=2
2025-07-14T05:39:43.782Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: restored user snapshot: index=1013
2025-07-14T05:39:43.782Z [ERROR] server-c058df63-de19-13a4-b2df-1a340a9e6a1f: failed to get log: index=2 error="log not found"
2025-07-14T05:39:43.782Z [ERROR] server-c058df63-de19-13a4-b2df-1a340a9e6a1f: failed to get log: index=2 error="log not found"
2025-07-14T05:39:43.782Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: aborting pipeline replication: peer="{Voter server-1b3f21f6-8a1c-c872-7d0f-8621572e3119 1b3f21f6-8a1c-c872-7d0f-8621572e3119}"
2025-07-14T05:39:43.782Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: aborting pipeline replication: peer="{Voter server-bb9d2a65-4352-85fb-6540-fd1580f66423 bb9d2a65-4352-85fb-6540-fd1580f66423}"
2025-07-14T05:39:43.790Z [ERROR] server-c058df63-de19-13a4-b2df-1a340a9e6a1f: failed to get log: index=2 error="log not found"
2025-07-14T05:39:43.791Z [INFO]  creating new snapshot: path=/tmp/raft1586880591/snapshots/2-1013-1752471583791.tmp
2025-07-14T05:39:43.791Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:43.791Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: copied to local snapshot: bytes=71
2025-07-14T05:39:43.791Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: snapshot restore progress: id=2-1013-1752471583791 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:43.791Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: removing all old logs from log store
2025-07-14T05:39:43.791Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: compacting logs: from=1 to=2
2025-07-14T05:39:43.791Z [INFO]  server-1b3f21f6-8a1c-c872-7d0f-8621572e3119: Installed remote snapshot
2025-07-14T05:39:43.791Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: pipelining replication: peer="{Voter server-1b3f21f6-8a1c-c872-7d0f-8621572e3119 1b3f21f6-8a1c-c872-7d0f-8621572e3119}"
2025-07-14T05:39:43.791Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:43.792Z [ERROR] server-c058df63-de19-13a4-b2df-1a340a9e6a1f: failed to get log: index=2 error="log not found"
2025-07-14T05:39:43.792Z [INFO]  creating new snapshot: path=/tmp/raft2992751766/snapshots/2-1013-1752471583792.tmp
2025-07-14T05:39:43.792Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:43.792Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: copied to local snapshot: bytes=71
2025-07-14T05:39:43.792Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: snapshot restore progress: id=2-1013-1752471583792 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:43.792Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: removing all old logs from log store
2025-07-14T05:39:43.792Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: compacting logs: from=1 to=2
2025-07-14T05:39:43.792Z [INFO]  server-bb9d2a65-4352-85fb-6540-fd1580f66423: Installed remote snapshot
2025-07-14T05:39:43.793Z [INFO]  server-c058df63-de19-13a4-b2df-1a340a9e6a1f: pipelining replication: peer="{Voter server-bb9d2a65-4352-85fb-6540-fd1580f66423 bb9d2a65-4352-85fb-6540-fd1580f66423}"
2025-07-14T05:39:44.797Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:43.79193942 +0000 UTC m=+83.206399788 (2 nodes), 641ns from start of poll, 1.570810656s from cluster start. Timeout at 2025-07-14 05:39:44.797644598 +0000 UTC m=+84.212104986, 1.005705198s after stability
2025-07-14T05:39:44.809Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: aborting pipeline replication: peer="{Voter server-eaaae432-a4cf-7e29-02ca-701b9d537ebc eaaae432-a4cf-7e29-02ca-701b9d537ebc}"
2025-07-14T05:39:44.809Z [INFO]  server-f98bd965-9d3e-22f6-3c63-6bc8bdb55847: aborting pipeline replication: peer="{Voter server-091e13c4-0fa3-656b-1d88-a352ed491772 091e13c4-0fa3-656b-1d88-a352ed491772}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-07-14T05:39:44.811Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4 Address:80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4} {Suffrage:Voter ID:server-61f05e48-bda8-98c9-3a60-a45c59fc11d4 Address:61f05e48-bda8-98c9-3a60-a45c59fc11d4} {Suffrage:Voter ID:server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0 Address:b38e5241-bdfe-eed8-3bf8-2600c8f587d0}]"
2025-07-14T05:39:44.811Z [INFO]  server-61f05e48-bda8-98c9-3a60-a45c59fc11d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4 Address:80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4} {Suffrage:Voter ID:server-61f05e48-bda8-98c9-3a60-a45c59fc11d4 Address:61f05e48-bda8-98c9-3a60-a45c59fc11d4} {Suffrage:Voter ID:server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0 Address:b38e5241-bdfe-eed8-3bf8-2600c8f587d0}]"
2025-07-14T05:39:44.811Z [INFO]  server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4 Address:80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4} {Suffrage:Voter ID:server-61f05e48-bda8-98c9-3a60-a45c59fc11d4 Address:61f05e48-bda8-98c9-3a60-a45c59fc11d4} {Suffrage:Voter ID:server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0 Address:b38e5241-bdfe-eed8-3bf8-2600c8f587d0}]"
2025-07-14T05:39:44.811Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:44.811Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: entering follower state: follower="Node at 80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4 [Follower]" leader-address= leader-id=
2025-07-14T05:39:44.811Z [INFO]  server-61f05e48-bda8-98c9-3a60-a45c59fc11d4: entering follower state: follower="Node at 61f05e48-bda8-98c9-3a60-a45c59fc11d4 [Follower]" leader-address= leader-id=
2025-07-14T05:39:44.811Z [INFO]  server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0: entering follower state: follower="Node at b38e5241-bdfe-eed8-3bf8-2600c8f587d0 [Follower]" leader-address= leader-id=
2025-07-14T05:39:45.316Z [WARN]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:45.316Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: entering candidate state: node="Node at 80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4 [Candidate]" term=2
2025-07-14T05:39:45.316Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: election won: term=2 tally=2
2025-07-14T05:39:45.316Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: entering leader state: leader="Node at 80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4 [Leader]"
2025-07-14T05:39:45.316Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: added peer, starting replication: peer=server-61f05e48-bda8-98c9-3a60-a45c59fc11d4
2025-07-14T05:39:45.316Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: added peer, starting replication: peer=server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0
2025-07-14T05:39:45.316Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: pipelining replication: peer="{Voter server-61f05e48-bda8-98c9-3a60-a45c59fc11d4 61f05e48-bda8-98c9-3a60-a45c59fc11d4}"
2025-07-14T05:39:45.316Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: pipelining replication: peer="{Voter server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0 b38e5241-bdfe-eed8-3bf8-2600c8f587d0}"
2025-07-14T05:39:46.322Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:45.316882104 +0000 UTC m=+84.731342472 (1 nodes), 505.103505ms from start of poll, 505.922003ms from cluster start. Timeout at 2025-07-14 05:39:46.3222926 +0000 UTC m=+85.736752978, 1.005410506s after stability
2025-07-14T05:39:46.322Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: starting snapshot up to: index=12
2025-07-14T05:39:46.322Z [INFO]  creating new snapshot: path=/tmp/raft2279275634/snapshots/2-12-1752471586322.tmp
2025-07-14T05:39:46.323Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: snapshot complete up to: index=12
2025-07-14T05:39:46.323Z [INFO]  creating new snapshot: path=/tmp/raft2279275634/snapshots/2-10013-1752471586323.tmp
2025-07-14T05:39:46.324Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: copied to local snapshot: bytes=71
2025-07-14T05:39:46.324Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: snapshot restore progress: id=2-10013-1752471586323 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:46.324Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: restored user snapshot: index=10013
2025-07-14T05:39:46.324Z [ERROR] server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:46.324Z [ERROR] server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:46.324Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: aborting pipeline replication: peer="{Voter server-61f05e48-bda8-98c9-3a60-a45c59fc11d4 61f05e48-bda8-98c9-3a60-a45c59fc11d4}"
2025-07-14T05:39:46.324Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: aborting pipeline replication: peer="{Voter server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0 b38e5241-bdfe-eed8-3bf8-2600c8f587d0}"
2025-07-14T05:39:46.331Z [ERROR] server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:46.332Z [INFO]  creating new snapshot: path=/tmp/raft1893809161/snapshots/2-10013-1752471586332.tmp
2025-07-14T05:39:46.332Z [INFO]  server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:46.332Z [INFO]  server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0: copied to local snapshot: bytes=71
2025-07-14T05:39:46.332Z [ERROR] server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: failed to get log: index=23 error="log not found"
2025-07-14T05:39:46.332Z [INFO]  creating new snapshot: path=/tmp/raft1734692202/snapshots/2-10013-1752471586332.tmp
2025-07-14T05:39:46.332Z [INFO]  server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0: snapshot restore progress: id=2-10013-1752471586332 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:46.332Z [INFO]  server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0: Installed remote snapshot
2025-07-14T05:39:46.332Z [WARN]  server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0: skipping application of old log: index=22
2025-07-14T05:39:46.332Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: pipelining replication: peer="{Voter server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0 b38e5241-bdfe-eed8-3bf8-2600c8f587d0}"
2025-07-14T05:39:46.332Z [INFO]  server-61f05e48-bda8-98c9-3a60-a45c59fc11d4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:46.332Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:46.333Z [INFO]  server-61f05e48-bda8-98c9-3a60-a45c59fc11d4: copied to local snapshot: bytes=71
2025-07-14T05:39:46.333Z [INFO]  server-61f05e48-bda8-98c9-3a60-a45c59fc11d4: snapshot restore progress: id=2-10013-1752471586332 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:46.333Z [INFO]  server-61f05e48-bda8-98c9-3a60-a45c59fc11d4: Installed remote snapshot
2025-07-14T05:39:46.333Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: pipelining replication: peer="{Voter server-61f05e48-bda8-98c9-3a60-a45c59fc11d4 61f05e48-bda8-98c9-3a60-a45c59fc11d4}"
2025-07-14T05:39:47.338Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:46.332961803 +0000 UTC m=+85.747422181 (2 nodes), 951ns from start of poll, 1.522001712s from cluster start. Timeout at 2025-07-14 05:39:47.338353531 +0000 UTC m=+86.752813929, 1.005391748s after stability
2025-07-14T05:39:47.350Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: aborting pipeline replication: peer="{Voter server-b38e5241-bdfe-eed8-3bf8-2600c8f587d0 b38e5241-bdfe-eed8-3bf8-2600c8f587d0}"
2025-07-14T05:39:47.350Z [INFO]  server-80f231b9-43b3-d4a6-e4b3-3e6a8243b1f4: aborting pipeline replication: peer="{Voter server-61f05e48-bda8-98c9-3a60-a45c59fc11d4 61f05e48-bda8-98c9-3a60-a45c59fc11d4}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-07-14T05:39:47.351Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c472f03b-3610-9686-6ad5-81241d7d051c Address:c472f03b-3610-9686-6ad5-81241d7d051c} {Suffrage:Voter ID:server-cda32f1e-ca95-6410-71fc-d5abe4e327c6 Address:cda32f1e-ca95-6410-71fc-d5abe4e327c6} {Suffrage:Voter ID:server-7b31de1c-70be-df1e-37df-61f03aebf6fa Address:7b31de1c-70be-df1e-37df-61f03aebf6fa}]"
2025-07-14T05:39:47.352Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: entering follower state: follower="Node at c472f03b-3610-9686-6ad5-81241d7d051c [Follower]" leader-address= leader-id=
2025-07-14T05:39:47.352Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c472f03b-3610-9686-6ad5-81241d7d051c Address:c472f03b-3610-9686-6ad5-81241d7d051c} {Suffrage:Voter ID:server-cda32f1e-ca95-6410-71fc-d5abe4e327c6 Address:cda32f1e-ca95-6410-71fc-d5abe4e327c6} {Suffrage:Voter ID:server-7b31de1c-70be-df1e-37df-61f03aebf6fa Address:7b31de1c-70be-df1e-37df-61f03aebf6fa}]"
2025-07-14T05:39:47.352Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: entering follower state: follower="Node at cda32f1e-ca95-6410-71fc-d5abe4e327c6 [Follower]" leader-address= leader-id=
2025-07-14T05:39:47.352Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c472f03b-3610-9686-6ad5-81241d7d051c Address:c472f03b-3610-9686-6ad5-81241d7d051c} {Suffrage:Voter ID:server-cda32f1e-ca95-6410-71fc-d5abe4e327c6 Address:cda32f1e-ca95-6410-71fc-d5abe4e327c6} {Suffrage:Voter ID:server-7b31de1c-70be-df1e-37df-61f03aebf6fa Address:7b31de1c-70be-df1e-37df-61f03aebf6fa}]"
2025-07-14T05:39:47.352Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:47.352Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: entering follower state: follower="Node at 7b31de1c-70be-df1e-37df-61f03aebf6fa [Follower]" leader-address= leader-id=
2025-07-14T05:39:47.880Z [WARN]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:47.880Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: entering candidate state: node="Node at 7b31de1c-70be-df1e-37df-61f03aebf6fa [Candidate]" term=2
2025-07-14T05:39:47.880Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: election won: term=2 tally=2
2025-07-14T05:39:47.880Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: entering leader state: leader="Node at 7b31de1c-70be-df1e-37df-61f03aebf6fa [Leader]"
2025-07-14T05:39:47.880Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: added peer, starting replication: peer=server-c472f03b-3610-9686-6ad5-81241d7d051c
2025-07-14T05:39:47.880Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: added peer, starting replication: peer=server-cda32f1e-ca95-6410-71fc-d5abe4e327c6
2025-07-14T05:39:47.880Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: pipelining replication: peer="{Voter server-c472f03b-3610-9686-6ad5-81241d7d051c c472f03b-3610-9686-6ad5-81241d7d051c}"
2025-07-14T05:39:47.880Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: pipelining replication: peer="{Voter server-cda32f1e-ca95-6410-71fc-d5abe4e327c6 cda32f1e-ca95-6410-71fc-d5abe4e327c6}"
2025-07-14T05:39:48.888Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:47.880861002 +0000 UTC m=+87.295321380 (1 nodes), 528.545879ms from start of poll, 528.972429ms from cluster start. Timeout at 2025-07-14 05:39:48.88815985 +0000 UTC m=+88.302620248, 1.007298868s after stability
2025-07-14T05:39:48.889Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: starting snapshot up to: index=12
2025-07-14T05:39:48.889Z [INFO]  creating new snapshot: path=/tmp/raft2059675015/snapshots/2-12-1752471588889.tmp
2025-07-14T05:39:48.889Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: snapshot complete up to: index=12
2025-07-14T05:39:48.890Z [INFO]  creating new snapshot: path=/tmp/raft2059675015/snapshots/2-10013-1752471588890.tmp
2025-07-14T05:39:48.890Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: copied to local snapshot: bytes=71
2025-07-14T05:39:48.890Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: snapshot restore progress: id=2-10013-1752471588890 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:48.890Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: removing all old logs from log store
2025-07-14T05:39:48.890Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: compacting logs: from=1 to=22
2025-07-14T05:39:48.890Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: restored user snapshot: index=10013
2025-07-14T05:39:48.890Z [ERROR] server-7b31de1c-70be-df1e-37df-61f03aebf6fa: failed to get log: index=22 error="log not found"
2025-07-14T05:39:48.890Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: aborting pipeline replication: peer="{Voter server-cda32f1e-ca95-6410-71fc-d5abe4e327c6 cda32f1e-ca95-6410-71fc-d5abe4e327c6}"
2025-07-14T05:39:48.891Z [ERROR] server-7b31de1c-70be-df1e-37df-61f03aebf6fa: failed to get log: index=22 error="log not found"
2025-07-14T05:39:48.891Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: aborting pipeline replication: peer="{Voter server-c472f03b-3610-9686-6ad5-81241d7d051c c472f03b-3610-9686-6ad5-81241d7d051c}"
2025-07-14T05:39:48.899Z [ERROR] server-7b31de1c-70be-df1e-37df-61f03aebf6fa: failed to get log: index=22 error="log not found"
2025-07-14T05:39:48.899Z [INFO]  creating new snapshot: path=/tmp/raft338172885/snapshots/2-10013-1752471588899.tmp
2025-07-14T05:39:48.900Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:48.900Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: copied to local snapshot: bytes=71
2025-07-14T05:39:48.900Z [ERROR] server-7b31de1c-70be-df1e-37df-61f03aebf6fa: failed to get log: index=22 error="log not found"
2025-07-14T05:39:48.900Z [INFO]  creating new snapshot: path=/tmp/raft1470857437/snapshots/2-10013-1752471588900.tmp
2025-07-14T05:39:48.900Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:48.900Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: copied to local snapshot: bytes=71
2025-07-14T05:39:48.900Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: snapshot restore progress: id=2-10013-1752471588900 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:48.900Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: removing all old logs from log store
2025-07-14T05:39:48.900Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: compacting logs: from=1 to=22
2025-07-14T05:39:48.900Z [INFO]  server-c472f03b-3610-9686-6ad5-81241d7d051c: Installed remote snapshot
2025-07-14T05:39:48.901Z [WARN]  server-c472f03b-3610-9686-6ad5-81241d7d051c: skipping application of old log: index=22
2025-07-14T05:39:48.901Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: pipelining replication: peer="{Voter server-c472f03b-3610-9686-6ad5-81241d7d051c c472f03b-3610-9686-6ad5-81241d7d051c}"
2025-07-14T05:39:48.901Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: snapshot restore progress: id=2-10013-1752471588899 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:48.901Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: removing all old logs from log store
2025-07-14T05:39:48.901Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: compacting logs: from=1 to=22
2025-07-14T05:39:48.901Z [INFO]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: Installed remote snapshot
2025-07-14T05:39:48.901Z [WARN]  server-cda32f1e-ca95-6410-71fc-d5abe4e327c6: skipping application of old log: index=22
2025-07-14T05:39:48.904Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: pipelining replication: peer="{Voter server-cda32f1e-ca95-6410-71fc-d5abe4e327c6 cda32f1e-ca95-6410-71fc-d5abe4e327c6}"
2025-07-14T05:39:48.903Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:49.910Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:48.90409315 +0000 UTC m=+88.318553528 (2 nodes), 571ns from start of poll, 1.552204577s from cluster start. Timeout at 2025-07-14 05:39:49.910202955 +0000 UTC m=+89.324663323, 1.006109795s after stability
2025-07-14T05:39:49.921Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: aborting pipeline replication: peer="{Voter server-cda32f1e-ca95-6410-71fc-d5abe4e327c6 cda32f1e-ca95-6410-71fc-d5abe4e327c6}"
2025-07-14T05:39:49.921Z [INFO]  server-7b31de1c-70be-df1e-37df-61f03aebf6fa: aborting pipeline replication: peer="{Voter server-c472f03b-3610-9686-6ad5-81241d7d051c c472f03b-3610-9686-6ad5-81241d7d051c}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-07-14T05:39:49.923Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fb25d70-14e1-a159-9211-508c9059a3cd Address:2fb25d70-14e1-a159-9211-508c9059a3cd} {Suffrage:Voter ID:server-5d43e4ad-ac33-2552-95e6-7a183baf891b Address:5d43e4ad-ac33-2552-95e6-7a183baf891b} {Suffrage:Voter ID:server-297af766-b215-24c4-0d04-5f4a619ac9fd Address:297af766-b215-24c4-0d04-5f4a619ac9fd}]"
2025-07-14T05:39:49.923Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: entering follower state: follower="Node at 2fb25d70-14e1-a159-9211-508c9059a3cd [Follower]" leader-address= leader-id=
2025-07-14T05:39:49.923Z [INFO]  server-5d43e4ad-ac33-2552-95e6-7a183baf891b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fb25d70-14e1-a159-9211-508c9059a3cd Address:2fb25d70-14e1-a159-9211-508c9059a3cd} {Suffrage:Voter ID:server-5d43e4ad-ac33-2552-95e6-7a183baf891b Address:5d43e4ad-ac33-2552-95e6-7a183baf891b} {Suffrage:Voter ID:server-297af766-b215-24c4-0d04-5f4a619ac9fd Address:297af766-b215-24c4-0d04-5f4a619ac9fd}]"
2025-07-14T05:39:49.923Z [INFO]  server-5d43e4ad-ac33-2552-95e6-7a183baf891b: entering follower state: follower="Node at 5d43e4ad-ac33-2552-95e6-7a183baf891b [Follower]" leader-address= leader-id=
2025-07-14T05:39:49.923Z [INFO]  server-297af766-b215-24c4-0d04-5f4a619ac9fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fb25d70-14e1-a159-9211-508c9059a3cd Address:2fb25d70-14e1-a159-9211-508c9059a3cd} {Suffrage:Voter ID:server-5d43e4ad-ac33-2552-95e6-7a183baf891b Address:5d43e4ad-ac33-2552-95e6-7a183baf891b} {Suffrage:Voter ID:server-297af766-b215-24c4-0d04-5f4a619ac9fd Address:297af766-b215-24c4-0d04-5f4a619ac9fd}]"
2025-07-14T05:39:49.923Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:49.923Z [INFO]  server-297af766-b215-24c4-0d04-5f4a619ac9fd: entering follower state: follower="Node at 297af766-b215-24c4-0d04-5f4a619ac9fd [Follower]" leader-address= leader-id=
2025-07-14T05:39:50.567Z [WARN]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:50.567Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: entering candidate state: node="Node at 2fb25d70-14e1-a159-9211-508c9059a3cd [Candidate]" term=2
2025-07-14T05:39:50.567Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: election won: term=2 tally=2
2025-07-14T05:39:50.567Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: entering leader state: leader="Node at 2fb25d70-14e1-a159-9211-508c9059a3cd [Leader]"
2025-07-14T05:39:50.567Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: added peer, starting replication: peer=server-5d43e4ad-ac33-2552-95e6-7a183baf891b
2025-07-14T05:39:50.567Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: added peer, starting replication: peer=server-297af766-b215-24c4-0d04-5f4a619ac9fd
2025-07-14T05:39:50.567Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: pipelining replication: peer="{Voter server-297af766-b215-24c4-0d04-5f4a619ac9fd 297af766-b215-24c4-0d04-5f4a619ac9fd}"
2025-07-14T05:39:50.568Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: pipelining replication: peer="{Voter server-5d43e4ad-ac33-2552-95e6-7a183baf891b 5d43e4ad-ac33-2552-95e6-7a183baf891b}"
2025-07-14T05:39:51.573Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:50.567947772 +0000 UTC m=+89.982408170 (1 nodes), 644.287852ms from start of poll, 644.570896ms from cluster start. Timeout at 2025-07-14 05:39:51.573276775 +0000 UTC m=+90.987737163, 1.005328993s after stability
2025-07-14T05:39:51.576Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: starting snapshot up to: index=12
2025-07-14T05:39:51.576Z [INFO]  creating new snapshot: path=/tmp/raft496162450/snapshots/2-12-1752471591576.tmp
2025-07-14T05:39:51.576Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: snapshot complete up to: index=12
2025-07-14T05:39:51.578Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e086c42-81d1-71fa-748c-088da292bf59 Address:4e086c42-81d1-71fa-748c-088da292bf59} {Suffrage:Voter ID:server-95488cfa-1883-7837-5eea-def598dc7121 Address:95488cfa-1883-7837-5eea-def598dc7121} {Suffrage:Voter ID:server-3ff6ac97-1d3e-ca45-9af6-5764446a4703 Address:3ff6ac97-1d3e-ca45-9af6-5764446a4703}]"
2025-07-14T05:39:51.578Z [INFO]  server-95488cfa-1883-7837-5eea-def598dc7121: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e086c42-81d1-71fa-748c-088da292bf59 Address:4e086c42-81d1-71fa-748c-088da292bf59} {Suffrage:Voter ID:server-95488cfa-1883-7837-5eea-def598dc7121 Address:95488cfa-1883-7837-5eea-def598dc7121} {Suffrage:Voter ID:server-3ff6ac97-1d3e-ca45-9af6-5764446a4703 Address:3ff6ac97-1d3e-ca45-9af6-5764446a4703}]"
2025-07-14T05:39:51.578Z [INFO]  server-3ff6ac97-1d3e-ca45-9af6-5764446a4703: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e086c42-81d1-71fa-748c-088da292bf59 Address:4e086c42-81d1-71fa-748c-088da292bf59} {Suffrage:Voter ID:server-95488cfa-1883-7837-5eea-def598dc7121 Address:95488cfa-1883-7837-5eea-def598dc7121} {Suffrage:Voter ID:server-3ff6ac97-1d3e-ca45-9af6-5764446a4703 Address:3ff6ac97-1d3e-ca45-9af6-5764446a4703}]"
2025-07-14T05:39:51.578Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:51.578Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: entering follower state: follower="Node at 4e086c42-81d1-71fa-748c-088da292bf59 [Follower]" leader-address= leader-id=
2025-07-14T05:39:51.578Z [INFO]  server-3ff6ac97-1d3e-ca45-9af6-5764446a4703: entering follower state: follower="Node at 3ff6ac97-1d3e-ca45-9af6-5764446a4703 [Follower]" leader-address= leader-id=
2025-07-14T05:39:51.578Z [INFO]  server-95488cfa-1883-7837-5eea-def598dc7121: entering follower state: follower="Node at 95488cfa-1883-7837-5eea-def598dc7121 [Follower]" leader-address= leader-id=
2025-07-14T05:39:52.082Z [WARN]  server-4e086c42-81d1-71fa-748c-088da292bf59: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:52.082Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: entering candidate state: node="Node at 4e086c42-81d1-71fa-748c-088da292bf59 [Candidate]" term=2
2025-07-14T05:39:52.082Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: election won: term=2 tally=2
2025-07-14T05:39:52.082Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: entering leader state: leader="Node at 4e086c42-81d1-71fa-748c-088da292bf59 [Leader]"
2025-07-14T05:39:52.082Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: added peer, starting replication: peer=server-95488cfa-1883-7837-5eea-def598dc7121
2025-07-14T05:39:52.082Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: added peer, starting replication: peer=server-3ff6ac97-1d3e-ca45-9af6-5764446a4703
2025-07-14T05:39:52.082Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: pipelining replication: peer="{Voter server-95488cfa-1883-7837-5eea-def598dc7121 95488cfa-1883-7837-5eea-def598dc7121}"
2025-07-14T05:39:52.082Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: pipelining replication: peer="{Voter server-3ff6ac97-1d3e-ca45-9af6-5764446a4703 3ff6ac97-1d3e-ca45-9af6-5764446a4703}"
2025-07-14T05:39:53.088Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:52.082421782 +0000 UTC m=+91.496882181 (1 nodes), 504.203546ms from start of poll, 504.475614ms from cluster start. Timeout at 2025-07-14 05:39:53.088235823 +0000 UTC m=+92.502696191, 1.00581401s after stability
2025-07-14T05:39:53.088Z [INFO]  creating new snapshot: path=/tmp/raft3502526945/snapshots/2-10013-1752471593088.tmp
2025-07-14T05:39:53.088Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: copied to local snapshot: bytes=71
2025-07-14T05:39:53.089Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: snapshot restore progress: id=2-10013-1752471593088 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:53.089Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: restored user snapshot: index=10013
2025-07-14T05:39:53.089Z [ERROR] server-4e086c42-81d1-71fa-748c-088da292bf59: failed to get log: index=3 error="log not found"
2025-07-14T05:39:53.089Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: aborting pipeline replication: peer="{Voter server-95488cfa-1883-7837-5eea-def598dc7121 95488cfa-1883-7837-5eea-def598dc7121}"
2025-07-14T05:39:53.089Z [ERROR] server-4e086c42-81d1-71fa-748c-088da292bf59: failed to get log: index=3 error="log not found"
2025-07-14T05:39:53.089Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: aborting pipeline replication: peer="{Voter server-3ff6ac97-1d3e-ca45-9af6-5764446a4703 3ff6ac97-1d3e-ca45-9af6-5764446a4703}"
2025-07-14T05:39:53.098Z [ERROR] server-4e086c42-81d1-71fa-748c-088da292bf59: failed to get log: index=3 error="log not found"
2025-07-14T05:39:53.098Z [ERROR] server-4e086c42-81d1-71fa-748c-088da292bf59: failed to get log: index=3 error="log not found"
2025-07-14T05:39:53.099Z [INFO]  creating new snapshot: path=/tmp/raft2141354719/snapshots/2-10013-1752471593099.tmp
2025-07-14T05:39:53.099Z [INFO]  server-95488cfa-1883-7837-5eea-def598dc7121: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:53.099Z [INFO]  creating new snapshot: path=/tmp/raft1768304821/snapshots/2-10013-1752471593099.tmp
2025-07-14T05:39:53.099Z [INFO]  server-3ff6ac97-1d3e-ca45-9af6-5764446a4703: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:53.099Z [INFO]  server-95488cfa-1883-7837-5eea-def598dc7121: copied to local snapshot: bytes=71
2025-07-14T05:39:53.099Z [INFO]  server-95488cfa-1883-7837-5eea-def598dc7121: snapshot restore progress: id=2-10013-1752471593099 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:53.100Z [INFO]  server-95488cfa-1883-7837-5eea-def598dc7121: Installed remote snapshot
2025-07-14T05:39:53.100Z [INFO]  server-3ff6ac97-1d3e-ca45-9af6-5764446a4703: copied to local snapshot: bytes=71
2025-07-14T05:39:53.100Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: pipelining replication: peer="{Voter server-95488cfa-1883-7837-5eea-def598dc7121 95488cfa-1883-7837-5eea-def598dc7121}"
2025-07-14T05:39:53.100Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:53.100Z [INFO]  server-3ff6ac97-1d3e-ca45-9af6-5764446a4703: snapshot restore progress: id=2-10013-1752471593099 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:53.100Z [INFO]  server-3ff6ac97-1d3e-ca45-9af6-5764446a4703: Installed remote snapshot
2025-07-14T05:39:53.100Z [INFO]  server-4e086c42-81d1-71fa-748c-088da292bf59: pipelining replication: peer="{Voter server-3ff6ac97-1d3e-ca45-9af6-5764446a4703 3ff6ac97-1d3e-ca45-9af6-5764446a4703}"
2025-07-14T05:39:54.105Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:53.100216695 +0000 UTC m=+92.514677073 (2 nodes), 751ns from start of poll, 1.522270506s from cluster start. Timeout at 2025-07-14 05:39:54.105784326 +0000 UTC m=+93.520244694, 1.005567621s after stability
2025-07-14T05:39:54.117Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: aborting pipeline replication: peer="{Voter server-297af766-b215-24c4-0d04-5f4a619ac9fd 297af766-b215-24c4-0d04-5f4a619ac9fd}"
2025-07-14T05:39:54.117Z [INFO]  server-2fb25d70-14e1-a159-9211-508c9059a3cd: aborting pipeline replication: peer="{Voter server-5d43e4ad-ac33-2552-95e6-7a183baf891b 5d43e4ad-ac33-2552-95e6-7a183baf891b}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-07-14T05:39:54.118Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c Address:4f1eb574-0212-83e0-7783-5eef6bd3cf5c} {Suffrage:Voter ID:server-d24f56e8-ff92-edae-d1b0-5277300e2622 Address:d24f56e8-ff92-edae-d1b0-5277300e2622} {Suffrage:Voter ID:server-3c06f112-6bac-74a5-68bb-68edbfc22c4e Address:3c06f112-6bac-74a5-68bb-68edbfc22c4e}]"
2025-07-14T05:39:54.118Z [INFO]  server-d24f56e8-ff92-edae-d1b0-5277300e2622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c Address:4f1eb574-0212-83e0-7783-5eef6bd3cf5c} {Suffrage:Voter ID:server-d24f56e8-ff92-edae-d1b0-5277300e2622 Address:d24f56e8-ff92-edae-d1b0-5277300e2622} {Suffrage:Voter ID:server-3c06f112-6bac-74a5-68bb-68edbfc22c4e Address:3c06f112-6bac-74a5-68bb-68edbfc22c4e}]"
2025-07-14T05:39:54.118Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: entering follower state: follower="Node at 4f1eb574-0212-83e0-7783-5eef6bd3cf5c [Follower]" leader-address= leader-id=
2025-07-14T05:39:54.118Z [INFO]  server-d24f56e8-ff92-edae-d1b0-5277300e2622: entering follower state: follower="Node at d24f56e8-ff92-edae-d1b0-5277300e2622 [Follower]" leader-address= leader-id=
2025-07-14T05:39:54.118Z [INFO]  server-3c06f112-6bac-74a5-68bb-68edbfc22c4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c Address:4f1eb574-0212-83e0-7783-5eef6bd3cf5c} {Suffrage:Voter ID:server-d24f56e8-ff92-edae-d1b0-5277300e2622 Address:d24f56e8-ff92-edae-d1b0-5277300e2622} {Suffrage:Voter ID:server-3c06f112-6bac-74a5-68bb-68edbfc22c4e Address:3c06f112-6bac-74a5-68bb-68edbfc22c4e}]"
2025-07-14T05:39:54.118Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:54.118Z [INFO]  server-3c06f112-6bac-74a5-68bb-68edbfc22c4e: entering follower state: follower="Node at 3c06f112-6bac-74a5-68bb-68edbfc22c4e [Follower]" leader-address= leader-id=
2025-07-14T05:39:54.687Z [WARN]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:54.687Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: entering candidate state: node="Node at 4f1eb574-0212-83e0-7783-5eef6bd3cf5c [Candidate]" term=2
2025-07-14T05:39:54.687Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: election won: term=2 tally=2
2025-07-14T05:39:54.687Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: entering leader state: leader="Node at 4f1eb574-0212-83e0-7783-5eef6bd3cf5c [Leader]"
2025-07-14T05:39:54.687Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: added peer, starting replication: peer=server-d24f56e8-ff92-edae-d1b0-5277300e2622
2025-07-14T05:39:54.687Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: added peer, starting replication: peer=server-3c06f112-6bac-74a5-68bb-68edbfc22c4e
2025-07-14T05:39:54.687Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: pipelining replication: peer="{Voter server-3c06f112-6bac-74a5-68bb-68edbfc22c4e 3c06f112-6bac-74a5-68bb-68edbfc22c4e}"
2025-07-14T05:39:54.687Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: pipelining replication: peer="{Voter server-d24f56e8-ff92-edae-d1b0-5277300e2622 d24f56e8-ff92-edae-d1b0-5277300e2622}"
2025-07-14T05:39:55.693Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:54.687337145 +0000 UTC m=+94.101797523 (1 nodes), 568.481973ms from start of poll, 568.750966ms from cluster start. Timeout at 2025-07-14 05:39:55.693077536 +0000 UTC m=+95.107537904, 1.005740381s after stability
2025-07-14T05:39:55.693Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: starting snapshot up to: index=12
2025-07-14T05:39:55.693Z [INFO]  creating new snapshot: path=/tmp/raft126202334/snapshots/2-12-1752471595693.tmp
2025-07-14T05:39:55.701Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: snapshot complete up to: index=12
2025-07-14T05:39:55.703Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca855e6f-01ae-2440-f8f0-5b053788ce5e Address:ca855e6f-01ae-2440-f8f0-5b053788ce5e} {Suffrage:Voter ID:server-149faa5c-a687-0de1-6ddf-91f44cc16fbb Address:149faa5c-a687-0de1-6ddf-91f44cc16fbb} {Suffrage:Voter ID:server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4 Address:ee978c77-b1a0-b03c-55ce-f426b7e5bcb4}]"
2025-07-14T05:39:55.703Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca855e6f-01ae-2440-f8f0-5b053788ce5e Address:ca855e6f-01ae-2440-f8f0-5b053788ce5e} {Suffrage:Voter ID:server-149faa5c-a687-0de1-6ddf-91f44cc16fbb Address:149faa5c-a687-0de1-6ddf-91f44cc16fbb} {Suffrage:Voter ID:server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4 Address:ee978c77-b1a0-b03c-55ce-f426b7e5bcb4}]"
2025-07-14T05:39:55.703Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca855e6f-01ae-2440-f8f0-5b053788ce5e Address:ca855e6f-01ae-2440-f8f0-5b053788ce5e} {Suffrage:Voter ID:server-149faa5c-a687-0de1-6ddf-91f44cc16fbb Address:149faa5c-a687-0de1-6ddf-91f44cc16fbb} {Suffrage:Voter ID:server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4 Address:ee978c77-b1a0-b03c-55ce-f426b7e5bcb4}]"
2025-07-14T05:39:55.703Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:55.703Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: entering follower state: follower="Node at ca855e6f-01ae-2440-f8f0-5b053788ce5e [Follower]" leader-address= leader-id=
2025-07-14T05:39:55.703Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: entering follower state: follower="Node at 149faa5c-a687-0de1-6ddf-91f44cc16fbb [Follower]" leader-address= leader-id=
2025-07-14T05:39:55.703Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: entering follower state: follower="Node at ee978c77-b1a0-b03c-55ce-f426b7e5bcb4 [Follower]" leader-address= leader-id=
2025-07-14T05:39:56.472Z [WARN]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:56.472Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: entering candidate state: node="Node at ca855e6f-01ae-2440-f8f0-5b053788ce5e [Candidate]" term=2
2025-07-14T05:39:56.472Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: election won: term=2 tally=2
2025-07-14T05:39:56.472Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: entering leader state: leader="Node at ca855e6f-01ae-2440-f8f0-5b053788ce5e [Leader]"
2025-07-14T05:39:56.472Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: added peer, starting replication: peer=server-149faa5c-a687-0de1-6ddf-91f44cc16fbb
2025-07-14T05:39:56.472Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: added peer, starting replication: peer=server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4
2025-07-14T05:39:56.473Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: pipelining replication: peer="{Voter server-149faa5c-a687-0de1-6ddf-91f44cc16fbb 149faa5c-a687-0de1-6ddf-91f44cc16fbb}"
2025-07-14T05:39:56.473Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: pipelining replication: peer="{Voter server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4 ee978c77-b1a0-b03c-55ce-f426b7e5bcb4}"
2025-07-14T05:39:57.478Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:56.472989173 +0000 UTC m=+95.887449541 (1 nodes), 769.760731ms from start of poll, 770.065278ms from cluster start. Timeout at 2025-07-14 05:39:57.478355582 +0000 UTC m=+96.892815950, 1.005366409s after stability
2025-07-14T05:39:57.478Z [INFO]  creating new snapshot: path=/tmp/raft101553235/snapshots/2-10013-1752471597478.tmp
2025-07-14T05:39:57.479Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: copied to local snapshot: bytes=71
2025-07-14T05:39:57.479Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: snapshot restore progress: id=2-10013-1752471597478 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:57.479Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: removing all old logs from log store
2025-07-14T05:39:57.479Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: compacting logs: from=1 to=2
2025-07-14T05:39:57.479Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: restored user snapshot: index=10013
2025-07-14T05:39:57.479Z [ERROR] server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: failed to get log: index=2 error="log not found"
2025-07-14T05:39:57.479Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: aborting pipeline replication: peer="{Voter server-149faa5c-a687-0de1-6ddf-91f44cc16fbb 149faa5c-a687-0de1-6ddf-91f44cc16fbb}"
2025-07-14T05:39:57.479Z [ERROR] server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: failed to get log: index=2 error="log not found"
2025-07-14T05:39:57.479Z [ERROR] server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: failed to get log: index=2 error="log not found"
2025-07-14T05:39:57.479Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: aborting pipeline replication: peer="{Voter server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4 ee978c77-b1a0-b03c-55ce-f426b7e5bcb4}"
2025-07-14T05:39:57.479Z [INFO]  creating new snapshot: path=/tmp/raft3217229087/snapshots/2-10013-1752471597479.tmp
2025-07-14T05:39:57.479Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:57.480Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: copied to local snapshot: bytes=71
2025-07-14T05:39:57.480Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: snapshot restore progress: id=2-10013-1752471597479 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:57.480Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: removing all old logs from log store
2025-07-14T05:39:57.480Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: compacting logs: from=1 to=2
2025-07-14T05:39:57.480Z [INFO]  server-149faa5c-a687-0de1-6ddf-91f44cc16fbb: Installed remote snapshot
2025-07-14T05:39:57.480Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: pipelining replication: peer="{Voter server-149faa5c-a687-0de1-6ddf-91f44cc16fbb 149faa5c-a687-0de1-6ddf-91f44cc16fbb}"
2025-07-14T05:39:57.480Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:57.486Z [ERROR] server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: failed to get log: index=2 error="log not found"
2025-07-14T05:39:57.486Z [INFO]  creating new snapshot: path=/tmp/raft1691226468/snapshots/2-10013-1752471597486.tmp
2025-07-14T05:39:57.486Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:57.487Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: copied to local snapshot: bytes=71
2025-07-14T05:39:57.487Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: snapshot restore progress: id=2-10013-1752471597486 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-07-14T05:39:57.487Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: removing all old logs from log store
2025-07-14T05:39:57.487Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: compacting logs: from=1 to=2
2025-07-14T05:39:57.487Z [INFO]  server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4: Installed remote snapshot
2025-07-14T05:39:57.487Z [INFO]  server-ca855e6f-01ae-2440-f8f0-5b053788ce5e: pipelining replication: peer="{Voter server-ee978c77-b1a0-b03c-55ce-f426b7e5bcb4 ee978c77-b1a0-b03c-55ce-f426b7e5bcb4}"
2025-07-14T05:39:58.485Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:57.480503464 +0000 UTC m=+96.894963832 (2 nodes), 821ns from start of poll, 1.777579569s from cluster start. Timeout at 2025-07-14 05:39:58.485953399 +0000 UTC m=+97.900413787, 1.005449955s after stability
2025-07-14T05:39:58.497Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: aborting pipeline replication: peer="{Voter server-d24f56e8-ff92-edae-d1b0-5277300e2622 d24f56e8-ff92-edae-d1b0-5277300e2622}"
2025-07-14T05:39:58.497Z [INFO]  server-4f1eb574-0212-83e0-7783-5eef6bd3cf5c: aborting pipeline replication: peer="{Voter server-3c06f112-6bac-74a5-68bb-68edbfc22c4e 3c06f112-6bac-74a5-68bb-68edbfc22c4e}"
--- PASS: TestRaft_UserRestore (84.70s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.59s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.18s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.39s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (3.04s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.96s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.31s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.50s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.65s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.92s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.47s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.41s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.40s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.28s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.17s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.54s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.20s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.38s)
=== RUN   TestRaft_SendSnapshotFollower
2025-07-14T05:39:58.498Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c70c58a-c9d7-a153-b79d-356d038def20 Address:6c70c58a-c9d7-a153-b79d-356d038def20} {Suffrage:Voter ID:server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c Address:3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c} {Suffrage:Voter ID:server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a Address:f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a}]"
2025-07-14T05:39:58.498Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: entering follower state: follower="Node at 6c70c58a-c9d7-a153-b79d-356d038def20 [Follower]" leader-address= leader-id=
2025-07-14T05:39:58.498Z [INFO]  server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c70c58a-c9d7-a153-b79d-356d038def20 Address:6c70c58a-c9d7-a153-b79d-356d038def20} {Suffrage:Voter ID:server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c Address:3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c} {Suffrage:Voter ID:server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a Address:f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a}]"
2025-07-14T05:39:58.499Z [INFO]  server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: entering follower state: follower="Node at 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c [Follower]" leader-address= leader-id=
2025-07-14T05:39:58.499Z [INFO]  server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c70c58a-c9d7-a153-b79d-356d038def20 Address:6c70c58a-c9d7-a153-b79d-356d038def20} {Suffrage:Voter ID:server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c Address:3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c} {Suffrage:Voter ID:server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a Address:f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a}]"
2025-07-14T05:39:58.499Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:58.499Z [INFO]  server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a: entering follower state: follower="Node at f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a [Follower]" leader-address= leader-id=
2025-07-14T05:39:58.573Z [WARN]  server-6c70c58a-c9d7-a153-b79d-356d038def20: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:58.573Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: entering candidate state: node="Node at 6c70c58a-c9d7-a153-b79d-356d038def20 [Candidate]" term=2
2025-07-14T05:39:58.573Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: election won: term=2 tally=2
2025-07-14T05:39:58.573Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: entering leader state: leader="Node at 6c70c58a-c9d7-a153-b79d-356d038def20 [Leader]"
2025-07-14T05:39:58.573Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: added peer, starting replication: peer=server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c
2025-07-14T05:39:58.573Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: added peer, starting replication: peer=server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a
2025-07-14T05:39:58.574Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: pipelining replication: peer="{Voter server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a}"
2025-07-14T05:39:58.574Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: pipelining replication: peer="{Voter server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c}"
2025-07-14T05:39:58.679Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:58.573959799 +0000 UTC m=+97.988420177 (2 nodes), 74.900371ms from start of poll, 75.161803ms from cluster start. Timeout at 2025-07-14 05:39:58.679399658 +0000 UTC m=+98.093860036, 105.439859ms after stability
2025-07-14T05:39:58.679Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:58.785Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:58.679452357 +0000 UTC m=+98.093912735 (1 nodes), 610ns from start of poll, 180.654361ms from cluster start. Timeout at 2025-07-14 05:39:58.785410965 +0000 UTC m=+98.199871363, 105.958628ms after stability
2025-07-14T05:39:58.785Z [ERROR] server-6c70c58a-c9d7-a153-b79d-356d038def20: failed to pipeline appendEntries: peer="{Voter server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c}" error="append pipeline closed"
2025-07-14T05:39:58.785Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: aborting pipeline replication: peer="{Voter server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c}"
2025-07-14T05:39:58.785Z [ERROR] server-6c70c58a-c9d7-a153-b79d-356d038def20: failed to appendEntries to: peer="{Voter server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c}" error="failed to connect to peer: 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c"
2025-07-14T05:39:58.787Z [ERROR] server-6c70c58a-c9d7-a153-b79d-356d038def20: failed to heartbeat to: peer=3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c backoff time=10ms error="failed to connect to peer: 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-07-14T05:39:58.788Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: starting snapshot up to: index=102
2025-07-14T05:39:58.788Z [INFO]  creating new snapshot: path=/tmp/raft1184886692/snapshots/2-102-1752471598788.tmp
2025-07-14T05:39:58.789Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: compacting logs: from=1 to=92
2025-07-14T05:39:58.789Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: snapshot complete up to: index=102
2025-07-14T05:39:58.789Z [ERROR] server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: failed to take snapshot: error="nothing new to snapshot"
2025-07-14T05:39:58.789Z [INFO]  server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a: starting snapshot up to: index=95
2025-07-14T05:39:58.789Z [INFO]  creating new snapshot: path=/tmp/raft4021177270/snapshots/2-95-1752471598789.tmp
2025-07-14T05:39:58.789Z [INFO]  server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a: compacting logs: from=1 to=92
2025-07-14T05:39:58.789Z [INFO]  server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a: snapshot complete up to: index=95
2025-07-14T05:39:58.796Z [ERROR] server-6c70c58a-c9d7-a153-b79d-356d038def20: failed to get log: index=2 error="log not found"
2025-07-14T05:39:58.796Z [INFO]  creating new snapshot: path=/tmp/raft3612999791/snapshots/2-102-1752471598796.tmp
2025-07-14T05:39:58.796Z [INFO]  server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-07-14T05:39:58.797Z [INFO]  server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: copied to local snapshot: bytes=693
2025-07-14T05:39:58.797Z [INFO]  server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: snapshot restore progress: id=2-102-1752471598796 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-14T05:39:58.797Z [INFO]  server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: Installed remote snapshot
2025-07-14T05:39:58.797Z [WARN]  server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c: skipping application of old log: index=102
2025-07-14T05:39:58.797Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: pipelining replication: peer="{Voter server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c}"
2025-07-14T05:39:58.801Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: aborting pipeline replication: peer="{Voter server-f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a f7bb6fcb-adfb-78a9-c60a-a8495f7cdb7a}"
2025-07-14T05:39:58.801Z [INFO]  server-6c70c58a-c9d7-a153-b79d-356d038def20: aborting pipeline replication: peer="{Voter server-3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c 3a975d03-4a62-c3cb-d3cf-0ae9fccf4b7c}"
--- PASS: TestRaft_SendSnapshotFollower (0.30s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-07-14T05:39:58.802Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b630c26d-8011-3954-9c0e-358f1e5d5b7c Address:b630c26d-8011-3954-9c0e-358f1e5d5b7c} {Suffrage:Voter ID:server-570fdced-3500-a31a-daf7-3b331d5126c7 Address:570fdced-3500-a31a-daf7-3b331d5126c7} {Suffrage:Voter ID:server-e3ea270b-7968-898b-ca8f-e81d63f79ddf Address:e3ea270b-7968-898b-ca8f-e81d63f79ddf}]"
2025-07-14T05:39:58.802Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: entering follower state: follower="Node at b630c26d-8011-3954-9c0e-358f1e5d5b7c [Follower]" leader-address= leader-id=
2025-07-14T05:39:58.802Z [INFO]  server-570fdced-3500-a31a-daf7-3b331d5126c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b630c26d-8011-3954-9c0e-358f1e5d5b7c Address:b630c26d-8011-3954-9c0e-358f1e5d5b7c} {Suffrage:Voter ID:server-570fdced-3500-a31a-daf7-3b331d5126c7 Address:570fdced-3500-a31a-daf7-3b331d5126c7} {Suffrage:Voter ID:server-e3ea270b-7968-898b-ca8f-e81d63f79ddf Address:e3ea270b-7968-898b-ca8f-e81d63f79ddf}]"
2025-07-14T05:39:58.802Z [INFO]  server-570fdced-3500-a31a-daf7-3b331d5126c7: entering follower state: follower="Node at 570fdced-3500-a31a-daf7-3b331d5126c7 [Follower]" leader-address= leader-id=
2025-07-14T05:39:58.802Z [INFO]  server-e3ea270b-7968-898b-ca8f-e81d63f79ddf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b630c26d-8011-3954-9c0e-358f1e5d5b7c Address:b630c26d-8011-3954-9c0e-358f1e5d5b7c} {Suffrage:Voter ID:server-570fdced-3500-a31a-daf7-3b331d5126c7 Address:570fdced-3500-a31a-daf7-3b331d5126c7} {Suffrage:Voter ID:server-e3ea270b-7968-898b-ca8f-e81d63f79ddf Address:e3ea270b-7968-898b-ca8f-e81d63f79ddf}]"
2025-07-14T05:39:58.802Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:58.802Z [INFO]  server-e3ea270b-7968-898b-ca8f-e81d63f79ddf: entering follower state: follower="Node at e3ea270b-7968-898b-ca8f-e81d63f79ddf [Follower]" leader-address= leader-id=
2025-07-14T05:39:58.856Z [WARN]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:58.856Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: entering candidate state: node="Node at b630c26d-8011-3954-9c0e-358f1e5d5b7c [Candidate]" term=2
2025-07-14T05:39:58.856Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: election won: term=2 tally=2
2025-07-14T05:39:58.856Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: entering leader state: leader="Node at b630c26d-8011-3954-9c0e-358f1e5d5b7c [Leader]"
2025-07-14T05:39:58.856Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: added peer, starting replication: peer=server-570fdced-3500-a31a-daf7-3b331d5126c7
2025-07-14T05:39:58.856Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: added peer, starting replication: peer=server-e3ea270b-7968-898b-ca8f-e81d63f79ddf
2025-07-14T05:39:58.856Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: pipelining replication: peer="{Voter server-e3ea270b-7968-898b-ca8f-e81d63f79ddf e3ea270b-7968-898b-ca8f-e81d63f79ddf}"
2025-07-14T05:39:58.856Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: pipelining replication: peer="{Voter server-570fdced-3500-a31a-daf7-3b331d5126c7 570fdced-3500-a31a-daf7-3b331d5126c7}"
2025-07-14T05:39:58.961Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:58.856172718 +0000 UTC m=+98.270633086 (2 nodes), 53.541813ms from start of poll, 53.811798ms from cluster start. Timeout at 2025-07-14 05:39:58.961609132 +0000 UTC m=+98.376069511, 105.436425ms after stability
2025-07-14T05:39:58.961Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:59.067Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:58.961686569 +0000 UTC m=+98.376147037 (1 nodes), 902ns from start of poll, 159.325749ms from cluster start. Timeout at 2025-07-14 05:39:59.067330154 +0000 UTC m=+98.481790542, 105.643505ms after stability
2025-07-14T05:39:59.067Z [ERROR] server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: failed to pipeline appendEntries: peer="{Voter server-570fdced-3500-a31a-daf7-3b331d5126c7 570fdced-3500-a31a-daf7-3b331d5126c7}" error="append pipeline closed"
2025-07-14T05:39:59.067Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: aborting pipeline replication: peer="{Voter server-570fdced-3500-a31a-daf7-3b331d5126c7 570fdced-3500-a31a-daf7-3b331d5126c7}"
2025-07-14T05:39:59.067Z [ERROR] server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: failed to appendEntries to: peer="{Voter server-570fdced-3500-a31a-daf7-3b331d5126c7 570fdced-3500-a31a-daf7-3b331d5126c7}" error="failed to connect to peer: 570fdced-3500-a31a-daf7-3b331d5126c7"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-07-14T05:39:59.069Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: starting snapshot up to: index=102
2025-07-14T05:39:59.069Z [INFO]  creating new snapshot: path=/tmp/raft1255923658/snapshots/2-102-1752471599069.tmp
2025-07-14T05:39:59.070Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: compacting logs: from=1 to=92
2025-07-14T05:39:59.070Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: snapshot complete up to: index=102
2025-07-14T05:39:59.070Z [ERROR] server-570fdced-3500-a31a-daf7-3b331d5126c7: failed to take snapshot: error="nothing new to snapshot"
2025-07-14T05:39:59.070Z [INFO]  server-e3ea270b-7968-898b-ca8f-e81d63f79ddf: starting snapshot up to: index=93
2025-07-14T05:39:59.070Z [INFO]  creating new snapshot: path=/tmp/raft3217944449/snapshots/2-93-1752471599070.tmp
2025-07-14T05:39:59.071Z [INFO]  server-e3ea270b-7968-898b-ca8f-e81d63f79ddf: compacting logs: from=1 to=92
2025-07-14T05:39:59.071Z [INFO]  server-e3ea270b-7968-898b-ca8f-e81d63f79ddf: snapshot complete up to: index=93
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-07-14T05:39:59.078Z [ERROR] server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: failed to get log: index=2 error="log not found"
2025-07-14T05:39:59.078Z [INFO]  creating new snapshot: path=/tmp/raft3087431210/snapshots/2-102-1752471599078.tmp
2025-07-14T05:39:59.078Z [INFO]  server-570fdced-3500-a31a-daf7-3b331d5126c7: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-07-14T05:39:59.078Z [INFO]  server-570fdced-3500-a31a-daf7-3b331d5126c7: copied to local snapshot: bytes=693
2025-07-14T05:39:59.079Z [INFO]  server-570fdced-3500-a31a-daf7-3b331d5126c7: snapshot restore progress: id=2-102-1752471599078 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-07-14T05:39:59.079Z [INFO]  server-570fdced-3500-a31a-daf7-3b331d5126c7: Installed remote snapshot
2025-07-14T05:39:59.080Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: pipelining replication: peer="{Voter server-570fdced-3500-a31a-daf7-3b331d5126c7 570fdced-3500-a31a-daf7-3b331d5126c7}"
2025-07-14T05:39:59.084Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: aborting pipeline replication: peer="{Voter server-e3ea270b-7968-898b-ca8f-e81d63f79ddf e3ea270b-7968-898b-ca8f-e81d63f79ddf}"
2025-07-14T05:39:59.084Z [INFO]  server-b630c26d-8011-3954-9c0e-358f1e5d5b7c: aborting pipeline replication: peer="{Voter server-570fdced-3500-a31a-daf7-3b331d5126c7 570fdced-3500-a31a-daf7-3b331d5126c7}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.28s)
=== RUN   TestRaft_ReJoinFollower
2025-07-14T05:39:59.085Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de07878d-91ef-1538-df80-1a031b003854 Address:de07878d-91ef-1538-df80-1a031b003854} {Suffrage:Voter ID:server-4a94590c-9b39-2e47-d23b-b5128ea8ddea Address:4a94590c-9b39-2e47-d23b-b5128ea8ddea} {Suffrage:Voter ID:server-3bab5043-3baa-856f-476b-23476839997e Address:3bab5043-3baa-856f-476b-23476839997e}]"
2025-07-14T05:39:59.085Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: entering follower state: follower="Node at de07878d-91ef-1538-df80-1a031b003854 [Follower]" leader-address= leader-id=
2025-07-14T05:39:59.085Z [INFO]  server-4a94590c-9b39-2e47-d23b-b5128ea8ddea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de07878d-91ef-1538-df80-1a031b003854 Address:de07878d-91ef-1538-df80-1a031b003854} {Suffrage:Voter ID:server-4a94590c-9b39-2e47-d23b-b5128ea8ddea Address:4a94590c-9b39-2e47-d23b-b5128ea8ddea} {Suffrage:Voter ID:server-3bab5043-3baa-856f-476b-23476839997e Address:3bab5043-3baa-856f-476b-23476839997e}]"
2025-07-14T05:39:59.085Z [INFO]  server-4a94590c-9b39-2e47-d23b-b5128ea8ddea: entering follower state: follower="Node at 4a94590c-9b39-2e47-d23b-b5128ea8ddea [Follower]" leader-address= leader-id=
2025-07-14T05:39:59.085Z [INFO]  server-3bab5043-3baa-856f-476b-23476839997e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de07878d-91ef-1538-df80-1a031b003854 Address:de07878d-91ef-1538-df80-1a031b003854} {Suffrage:Voter ID:server-4a94590c-9b39-2e47-d23b-b5128ea8ddea Address:4a94590c-9b39-2e47-d23b-b5128ea8ddea} {Suffrage:Voter ID:server-3bab5043-3baa-856f-476b-23476839997e Address:3bab5043-3baa-856f-476b-23476839997e}]"
2025-07-14T05:39:59.085Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:59.085Z [INFO]  server-3bab5043-3baa-856f-476b-23476839997e: entering follower state: follower="Node at 3bab5043-3baa-856f-476b-23476839997e [Follower]" leader-address= leader-id=
2025-07-14T05:39:59.172Z [WARN]  server-de07878d-91ef-1538-df80-1a031b003854: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:59.172Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: entering candidate state: node="Node at de07878d-91ef-1538-df80-1a031b003854 [Candidate]" term=2
2025-07-14T05:39:59.172Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: election won: term=2 tally=2
2025-07-14T05:39:59.172Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: entering leader state: leader="Node at de07878d-91ef-1538-df80-1a031b003854 [Leader]"
2025-07-14T05:39:59.172Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: added peer, starting replication: peer=server-4a94590c-9b39-2e47-d23b-b5128ea8ddea
2025-07-14T05:39:59.172Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: added peer, starting replication: peer=server-3bab5043-3baa-856f-476b-23476839997e
2025-07-14T05:39:59.173Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: pipelining replication: peer="{Voter server-3bab5043-3baa-856f-476b-23476839997e 3bab5043-3baa-856f-476b-23476839997e}"
2025-07-14T05:39:59.173Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: pipelining replication: peer="{Voter server-4a94590c-9b39-2e47-d23b-b5128ea8ddea 4a94590c-9b39-2e47-d23b-b5128ea8ddea}"
2025-07-14T05:39:59.278Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:59.172998096 +0000 UTC m=+98.587458464 (1 nodes), 87.012048ms from start of poll, 87.294672ms from cluster start. Timeout at 2025-07-14 05:39:59.278823294 +0000 UTC m=+98.693283692, 105.825228ms after stability
2025-07-14T05:39:59.284Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:59.390Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:59.284693219 +0000 UTC m=+98.699153587 (2 nodes), 822ns from start of poll, 198.989795ms from cluster start. Timeout at 2025-07-14 05:39:59.39060749 +0000 UTC m=+98.805067878, 105.914291ms after stability
2025-07-14T05:39:59.390Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: updating configuration: command=RemoveServer server-id=server-4a94590c-9b39-2e47-d23b-b5128ea8ddea server-addr= servers="[{Suffrage:Voter ID:server-de07878d-91ef-1538-df80-1a031b003854 Address:de07878d-91ef-1538-df80-1a031b003854} {Suffrage:Voter ID:server-3bab5043-3baa-856f-476b-23476839997e Address:3bab5043-3baa-856f-476b-23476839997e}]"
2025-07-14T05:39:59.390Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: removed peer, stopping replication: peer=server-4a94590c-9b39-2e47-d23b-b5128ea8ddea last-index=3
2025-07-14T05:39:59.390Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: aborting pipeline replication: peer="{Voter server-4a94590c-9b39-2e47-d23b-b5128ea8ddea 4a94590c-9b39-2e47-d23b-b5128ea8ddea}"
2025-07-14T05:39:59.441Z [WARN]  server-4a94590c-9b39-2e47-d23b-b5128ea8ddea: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-07-14T05:39:59.496Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: updating configuration: command=AddVoter server-id=server-4a94590c-9b39-2e47-d23b-b5128ea8ddea server-addr=4a94590c-9b39-2e47-d23b-b5128ea8ddea servers="[{Suffrage:Voter ID:server-de07878d-91ef-1538-df80-1a031b003854 Address:de07878d-91ef-1538-df80-1a031b003854} {Suffrage:Voter ID:server-3bab5043-3baa-856f-476b-23476839997e Address:3bab5043-3baa-856f-476b-23476839997e} {Suffrage:Voter ID:server-4a94590c-9b39-2e47-d23b-b5128ea8ddea Address:4a94590c-9b39-2e47-d23b-b5128ea8ddea}]"
2025-07-14T05:39:59.496Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: added peer, starting replication: peer=server-4a94590c-9b39-2e47-d23b-b5128ea8ddea
2025-07-14T05:39:59.496Z [WARN]  server-4a94590c-9b39-2e47-d23b-b5128ea8ddea: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-07-14T05:39:59.496Z [WARN]  server-de07878d-91ef-1538-df80-1a031b003854: appendEntries rejected, sending older logs: peer="{Voter server-4a94590c-9b39-2e47-d23b-b5128ea8ddea 4a94590c-9b39-2e47-d23b-b5128ea8ddea}" next=4
2025-07-14T05:39:59.496Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:59.496Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: pipelining replication: peer="{Voter server-4a94590c-9b39-2e47-d23b-b5128ea8ddea 4a94590c-9b39-2e47-d23b-b5128ea8ddea}"
2025-07-14T05:39:59.601Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:59.496233699 +0000 UTC m=+98.910694077 (1 nodes), 751ns from start of poll, 410.530285ms from cluster start. Timeout at 2025-07-14 05:39:59.601449813 +0000 UTC m=+99.015910201, 105.216124ms after stability
2025-07-14T05:39:59.601Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: aborting pipeline replication: peer="{Voter server-3bab5043-3baa-856f-476b-23476839997e 3bab5043-3baa-856f-476b-23476839997e}"
2025-07-14T05:39:59.601Z [INFO]  server-de07878d-91ef-1538-df80-1a031b003854: aborting pipeline replication: peer="{Voter server-4a94590c-9b39-2e47-d23b-b5128ea8ddea 4a94590c-9b39-2e47-d23b-b5128ea8ddea}"
--- PASS: TestRaft_ReJoinFollower (0.52s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-07-14T05:39:59.602Z [INFO]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-295d4990-43cc-a9b5-2d8e-6a1809a2feae Address:295d4990-43cc-a9b5-2d8e-6a1809a2feae} {Suffrage:Voter ID:server-fb6dd369-f164-4063-8142-5ccd8ea78a28 Address:fb6dd369-f164-4063-8142-5ccd8ea78a28}]"
2025-07-14T05:39:59.602Z [INFO]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: entering follower state: follower="Node at 295d4990-43cc-a9b5-2d8e-6a1809a2feae [Follower]" leader-address= leader-id=
2025-07-14T05:39:59.602Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-295d4990-43cc-a9b5-2d8e-6a1809a2feae Address:295d4990-43cc-a9b5-2d8e-6a1809a2feae} {Suffrage:Voter ID:server-fb6dd369-f164-4063-8142-5ccd8ea78a28 Address:fb6dd369-f164-4063-8142-5ccd8ea78a28}]"
2025-07-14T05:39:59.602Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:59.602Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: entering follower state: follower="Node at fb6dd369-f164-4063-8142-5ccd8ea78a28 [Follower]" leader-address= leader-id=
2025-07-14T05:39:59.660Z [WARN]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:39:59.660Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: entering candidate state: node="Node at fb6dd369-f164-4063-8142-5ccd8ea78a28 [Candidate]" term=2
2025-07-14T05:39:59.660Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: election won: term=2 tally=2
2025-07-14T05:39:59.660Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: entering leader state: leader="Node at fb6dd369-f164-4063-8142-5ccd8ea78a28 [Leader]"
2025-07-14T05:39:59.660Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: added peer, starting replication: peer=server-295d4990-43cc-a9b5-2d8e-6a1809a2feae
2025-07-14T05:39:59.660Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: pipelining replication: peer="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}"
2025-07-14T05:39:59.766Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:39:59.660168101 +0000 UTC m=+99.074628469 (1 nodes), 57.576615ms from start of poll, 57.762103ms from cluster start. Timeout at 2025-07-14 05:39:59.766053158 +0000 UTC m=+99.180513536, 105.885067ms after stability
2025-07-14T05:39:59.771Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:39:59.877Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:39:59.771893699 +0000 UTC m=+99.186354077 (1 nodes), 621ns from start of poll, 169.487711ms from cluster start. Timeout at 2025-07-14 05:39:59.877502813 +0000 UTC m=+99.291963211, 105.609134ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 295d4990-43cc-a9b5-2d8e-6a1809a2feae [Follower]
2025-07-14T05:39:59.882Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to pipeline appendEntries: peer="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}" error="append pipeline closed"
2025-07-14T05:39:59.882Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: aborting pipeline replication: peer="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}"
2025-07-14T05:39:59.883Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to heartbeat to: peer=295d4990-43cc-a9b5-2d8e-6a1809a2feae backoff time=10ms error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae"
2025-07-14T05:39:59.890Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to appendEntries to: peer="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}" error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae"
2025-07-14T05:39:59.899Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to heartbeat to: peer=295d4990-43cc-a9b5-2d8e-6a1809a2feae backoff time=10ms error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae"
2025-07-14T05:39:59.907Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to appendEntries to: peer="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}" error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae"
2025-07-14T05:39:59.916Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to heartbeat to: peer=295d4990-43cc-a9b5-2d8e-6a1809a2feae backoff time=10ms error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae"
2025-07-14T05:39:59.924Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to appendEntries to: peer="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}" error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae"
2025-07-14T05:39:59.924Z [WARN]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to contact: server-id=server-295d4990-43cc-a9b5-2d8e-6a1809a2feae time=50.757907ms
2025-07-14T05:39:59.924Z [WARN]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to contact quorum of nodes, stepping down
2025-07-14T05:39:59.924Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: entering follower state: follower="Node at fb6dd369-f164-4063-8142-5ccd8ea78a28 [Follower]" leader-address= leader-id=
2025-07-14T05:39:59.924Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:39:59.966Z [WARN]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: heartbeat timeout reached, starting election: last-leader-addr=fb6dd369-f164-4063-8142-5ccd8ea78a28 last-leader-id=server-fb6dd369-f164-4063-8142-5ccd8ea78a28
2025-07-14T05:39:59.967Z [INFO]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: entering candidate state: node="Node at 295d4990-43cc-a9b5-2d8e-6a1809a2feae [Candidate]" term=3
2025-07-14T05:39:59.967Z [ERROR] server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: failed to make requestVote RPC: target="{Voter server-fb6dd369-f164-4063-8142-5ccd8ea78a28 fb6dd369-f164-4063-8142-5ccd8ea78a28}" error="failed to connect to peer: fb6dd369-f164-4063-8142-5ccd8ea78a28" term=3
2025-07-14T05:40:00.010Z [WARN]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:00.010Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: entering candidate state: node="Node at fb6dd369-f164-4063-8142-5ccd8ea78a28 [Candidate]" term=3
2025-07-14T05:40:00.010Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to make requestVote RPC: target="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}" error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae" term=3
2025-07-14T05:40:00.023Z [WARN]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: Election timeout reached, restarting election
2025-07-14T05:40:00.023Z [INFO]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: entering candidate state: node="Node at 295d4990-43cc-a9b5-2d8e-6a1809a2feae [Candidate]" term=4
2025-07-14T05:40:00.023Z [ERROR] server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: failed to make requestVote RPC: target="{Voter server-fb6dd369-f164-4063-8142-5ccd8ea78a28 fb6dd369-f164-4063-8142-5ccd8ea78a28}" error="failed to connect to peer: fb6dd369-f164-4063-8142-5ccd8ea78a28" term=4
2025-07-14T05:40:00.074Z [WARN]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: Election timeout reached, restarting election
2025-07-14T05:40:00.074Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: entering candidate state: node="Node at fb6dd369-f164-4063-8142-5ccd8ea78a28 [Candidate]" term=4
2025-07-14T05:40:00.074Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to make requestVote RPC: target="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}" error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae" term=4
2025-07-14T05:40:00.109Z [WARN]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: Election timeout reached, restarting election
2025-07-14T05:40:00.109Z [INFO]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: entering candidate state: node="Node at 295d4990-43cc-a9b5-2d8e-6a1809a2feae [Candidate]" term=5
2025-07-14T05:40:00.109Z [ERROR] server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: failed to make requestVote RPC: target="{Voter server-fb6dd369-f164-4063-8142-5ccd8ea78a28 fb6dd369-f164-4063-8142-5ccd8ea78a28}" error="failed to connect to peer: fb6dd369-f164-4063-8142-5ccd8ea78a28" term=5
2025-07-14T05:40:00.116Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:00.010561561 +0000 UTC m=+99.425022009 (0 nodes), 85.610516ms from start of poll, 408.155643ms from cluster start. Timeout at 2025-07-14 05:40:00.116075372 +0000 UTC m=+99.530535750, 105.513741ms after stability
2025-07-14T05:40:00.137Z [WARN]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: Election timeout reached, restarting election
2025-07-14T05:40:00.137Z [INFO]  server-fb6dd369-f164-4063-8142-5ccd8ea78a28: entering candidate state: node="Node at fb6dd369-f164-4063-8142-5ccd8ea78a28 [Candidate]" term=5
2025-07-14T05:40:00.137Z [ERROR] server-fb6dd369-f164-4063-8142-5ccd8ea78a28: failed to make requestVote RPC: target="{Voter server-295d4990-43cc-a9b5-2d8e-6a1809a2feae 295d4990-43cc-a9b5-2d8e-6a1809a2feae}" error="failed to connect to peer: 295d4990-43cc-a9b5-2d8e-6a1809a2feae" term=5
2025-07-14T05:40:00.201Z [WARN]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: Election timeout reached, restarting election
2025-07-14T05:40:00.201Z [INFO]  server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: entering candidate state: node="Node at 295d4990-43cc-a9b5-2d8e-6a1809a2feae [Candidate]" term=6
2025-07-14T05:40:00.201Z [ERROR] server-295d4990-43cc-a9b5-2d8e-6a1809a2feae: failed to make requestVote RPC: target="{Voter server-fb6dd369-f164-4063-8142-5ccd8ea78a28 fb6dd369-f164-4063-8142-5ccd8ea78a28}" error="failed to connect to peer: fb6dd369-f164-4063-8142-5ccd8ea78a28" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.62s)
=== RUN   TestRaft_Barrier
2025-07-14T05:40:00.222Z [INFO]  server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7 Address:bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7} {Suffrage:Voter ID:server-6cae69b6-4e45-e13d-a6cf-4ca344121379 Address:6cae69b6-4e45-e13d-a6cf-4ca344121379} {Suffrage:Voter ID:server-add5ad60-5455-fb6d-e651-4ffc778f9e68 Address:add5ad60-5455-fb6d-e651-4ffc778f9e68}]"
2025-07-14T05:40:00.222Z [INFO]  server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7: entering follower state: follower="Node at bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7 [Follower]" leader-address= leader-id=
2025-07-14T05:40:00.222Z [INFO]  server-6cae69b6-4e45-e13d-a6cf-4ca344121379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7 Address:bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7} {Suffrage:Voter ID:server-6cae69b6-4e45-e13d-a6cf-4ca344121379 Address:6cae69b6-4e45-e13d-a6cf-4ca344121379} {Suffrage:Voter ID:server-add5ad60-5455-fb6d-e651-4ffc778f9e68 Address:add5ad60-5455-fb6d-e651-4ffc778f9e68}]"
2025-07-14T05:40:00.222Z [INFO]  server-6cae69b6-4e45-e13d-a6cf-4ca344121379: entering follower state: follower="Node at 6cae69b6-4e45-e13d-a6cf-4ca344121379 [Follower]" leader-address= leader-id=
2025-07-14T05:40:00.222Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7 Address:bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7} {Suffrage:Voter ID:server-6cae69b6-4e45-e13d-a6cf-4ca344121379 Address:6cae69b6-4e45-e13d-a6cf-4ca344121379} {Suffrage:Voter ID:server-add5ad60-5455-fb6d-e651-4ffc778f9e68 Address:add5ad60-5455-fb6d-e651-4ffc778f9e68}]"
2025-07-14T05:40:00.222Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:00.222Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: entering follower state: follower="Node at add5ad60-5455-fb6d-e651-4ffc778f9e68 [Follower]" leader-address= leader-id=
2025-07-14T05:40:00.279Z [WARN]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:00.279Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: entering candidate state: node="Node at add5ad60-5455-fb6d-e651-4ffc778f9e68 [Candidate]" term=2
2025-07-14T05:40:00.279Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: election won: term=2 tally=2
2025-07-14T05:40:00.279Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: entering leader state: leader="Node at add5ad60-5455-fb6d-e651-4ffc778f9e68 [Leader]"
2025-07-14T05:40:00.279Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: added peer, starting replication: peer=server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7
2025-07-14T05:40:00.279Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: added peer, starting replication: peer=server-6cae69b6-4e45-e13d-a6cf-4ca344121379
2025-07-14T05:40:00.279Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: pipelining replication: peer="{Voter server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7 bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7}"
2025-07-14T05:40:00.279Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: pipelining replication: peer="{Voter server-6cae69b6-4e45-e13d-a6cf-4ca344121379 6cae69b6-4e45-e13d-a6cf-4ca344121379}"
2025-07-14T05:40:00.385Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:00.279812267 +0000 UTC m=+99.694272665 (1 nodes), 57.127231ms from start of poll, 57.438598ms from cluster start. Timeout at 2025-07-14 05:40:00.38565398 +0000 UTC m=+99.800114358, 105.841693ms after stability
2025-07-14T05:40:00.401Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: aborting pipeline replication: peer="{Voter server-6cae69b6-4e45-e13d-a6cf-4ca344121379 6cae69b6-4e45-e13d-a6cf-4ca344121379}"
2025-07-14T05:40:00.401Z [INFO]  server-add5ad60-5455-fb6d-e651-4ffc778f9e68: aborting pipeline replication: peer="{Voter server-bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7 bc4ed9e5-fbc8-bc05-6a30-6bfcc07592c7}"
--- PASS: TestRaft_Barrier (0.18s)
=== RUN   TestRaft_VerifyLeader
2025-07-14T05:40:00.402Z [INFO]  server-0776125e-e773-179c-5110-cff6cb723068: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0776125e-e773-179c-5110-cff6cb723068 Address:0776125e-e773-179c-5110-cff6cb723068} {Suffrage:Voter ID:server-bb2b81a1-5c79-b237-6c16-2e10ea13938d Address:bb2b81a1-5c79-b237-6c16-2e10ea13938d} {Suffrage:Voter ID:server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe Address:20bc3e51-7abd-f0fc-dba2-49f08711d9fe}]"
2025-07-14T05:40:00.402Z [INFO]  server-0776125e-e773-179c-5110-cff6cb723068: entering follower state: follower="Node at 0776125e-e773-179c-5110-cff6cb723068 [Follower]" leader-address= leader-id=
2025-07-14T05:40:00.402Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0776125e-e773-179c-5110-cff6cb723068 Address:0776125e-e773-179c-5110-cff6cb723068} {Suffrage:Voter ID:server-bb2b81a1-5c79-b237-6c16-2e10ea13938d Address:bb2b81a1-5c79-b237-6c16-2e10ea13938d} {Suffrage:Voter ID:server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe Address:20bc3e51-7abd-f0fc-dba2-49f08711d9fe}]"
2025-07-14T05:40:00.402Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: entering follower state: follower="Node at bb2b81a1-5c79-b237-6c16-2e10ea13938d [Follower]" leader-address= leader-id=
2025-07-14T05:40:00.402Z [INFO]  server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0776125e-e773-179c-5110-cff6cb723068 Address:0776125e-e773-179c-5110-cff6cb723068} {Suffrage:Voter ID:server-bb2b81a1-5c79-b237-6c16-2e10ea13938d Address:bb2b81a1-5c79-b237-6c16-2e10ea13938d} {Suffrage:Voter ID:server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe Address:20bc3e51-7abd-f0fc-dba2-49f08711d9fe}]"
2025-07-14T05:40:00.402Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:00.402Z [INFO]  server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe: entering follower state: follower="Node at 20bc3e51-7abd-f0fc-dba2-49f08711d9fe [Follower]" leader-address= leader-id=
2025-07-14T05:40:00.466Z [WARN]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:00.466Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: entering candidate state: node="Node at bb2b81a1-5c79-b237-6c16-2e10ea13938d [Candidate]" term=2
2025-07-14T05:40:00.466Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: election won: term=2 tally=2
2025-07-14T05:40:00.466Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: entering leader state: leader="Node at bb2b81a1-5c79-b237-6c16-2e10ea13938d [Leader]"
2025-07-14T05:40:00.466Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: added peer, starting replication: peer=server-0776125e-e773-179c-5110-cff6cb723068
2025-07-14T05:40:00.467Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: added peer, starting replication: peer=server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe
2025-07-14T05:40:00.467Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: pipelining replication: peer="{Voter server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe 20bc3e51-7abd-f0fc-dba2-49f08711d9fe}"
2025-07-14T05:40:00.467Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: pipelining replication: peer="{Voter server-0776125e-e773-179c-5110-cff6cb723068 0776125e-e773-179c-5110-cff6cb723068}"
2025-07-14T05:40:00.572Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:00.467022602 +0000 UTC m=+99.881482981 (1 nodes), 64.353621ms from start of poll, 64.61291ms from cluster start. Timeout at 2025-07-14 05:40:00.57210112 +0000 UTC m=+99.986561498, 105.078517ms after stability
2025-07-14T05:40:00.572Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: aborting pipeline replication: peer="{Voter server-20bc3e51-7abd-f0fc-dba2-49f08711d9fe 20bc3e51-7abd-f0fc-dba2-49f08711d9fe}"
2025-07-14T05:40:00.572Z [INFO]  server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: aborting pipeline replication: peer="{Voter server-0776125e-e773-179c-5110-cff6cb723068 0776125e-e773-179c-5110-cff6cb723068}"
2025-07-14T05:40:01.072Z [ERROR] server-bb2b81a1-5c79-b237-6c16-2e10ea13938d: failed to heartbeat to: peer=0776125e-e773-179c-5110-cff6cb723068 backoff time=10ms error="command timed out"
--- PASS: TestRaft_VerifyLeader (0.67s)
=== RUN   TestRaft_VerifyLeader_Single
2025-07-14T05:40:01.073Z [INFO]  server-6b434eea-889b-cbac-a269-a39cc9534013: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b434eea-889b-cbac-a269-a39cc9534013 Address:6b434eea-889b-cbac-a269-a39cc9534013}]"
2025-07-14T05:40:01.073Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:01.073Z [INFO]  server-6b434eea-889b-cbac-a269-a39cc9534013: entering follower state: follower="Node at 6b434eea-889b-cbac-a269-a39cc9534013 [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.132Z [WARN]  server-6b434eea-889b-cbac-a269-a39cc9534013: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:01.132Z [INFO]  server-6b434eea-889b-cbac-a269-a39cc9534013: entering candidate state: node="Node at 6b434eea-889b-cbac-a269-a39cc9534013 [Candidate]" term=2
2025-07-14T05:40:01.132Z [INFO]  server-6b434eea-889b-cbac-a269-a39cc9534013: election won: term=2 tally=1
2025-07-14T05:40:01.132Z [INFO]  server-6b434eea-889b-cbac-a269-a39cc9534013: entering leader state: leader="Node at 6b434eea-889b-cbac-a269-a39cc9534013 [Leader]"
2025-07-14T05:40:01.238Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:01.132536523 +0000 UTC m=+100.546996921 (1 nodes), 59.160991ms from start of poll, 59.270836ms from cluster start. Timeout at 2025-07-14 05:40:01.238089913 +0000 UTC m=+100.652550291, 105.55337ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.17s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-07-14T05:40:01.239Z [INFO]  server-350eb139-7aae-3488-d305-81df07adad55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-350eb139-7aae-3488-d305-81df07adad55 Address:350eb139-7aae-3488-d305-81df07adad55} {Suffrage:Voter ID:server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf Address:d34124f8-2f0c-a372-c7b4-84c7903c0fcf}]"
2025-07-14T05:40:01.239Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-350eb139-7aae-3488-d305-81df07adad55 Address:350eb139-7aae-3488-d305-81df07adad55} {Suffrage:Voter ID:server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf Address:d34124f8-2f0c-a372-c7b4-84c7903c0fcf}]"
2025-07-14T05:40:01.239Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:01.239Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: entering follower state: follower="Node at d34124f8-2f0c-a372-c7b4-84c7903c0fcf [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.239Z [INFO]  server-350eb139-7aae-3488-d305-81df07adad55: entering follower state: follower="Node at 350eb139-7aae-3488-d305-81df07adad55 [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.292Z [WARN]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:01.292Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: entering candidate state: node="Node at d34124f8-2f0c-a372-c7b4-84c7903c0fcf [Candidate]" term=2
2025-07-14T05:40:01.292Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: election won: term=2 tally=2
2025-07-14T05:40:01.292Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: entering leader state: leader="Node at d34124f8-2f0c-a372-c7b4-84c7903c0fcf [Leader]"
2025-07-14T05:40:01.292Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: added peer, starting replication: peer=server-350eb139-7aae-3488-d305-81df07adad55
2025-07-14T05:40:01.292Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: pipelining replication: peer="{Voter server-350eb139-7aae-3488-d305-81df07adad55 350eb139-7aae-3488-d305-81df07adad55}"
2025-07-14T05:40:01.398Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:01.292545537 +0000 UTC m=+100.707005905 (1 nodes), 53.362534ms from start of poll, 53.582224ms from cluster start. Timeout at 2025-07-14 05:40:01.398443674 +0000 UTC m=+100.812904062, 105.898157ms after stability
2025-07-14T05:40:01.398Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:01.504Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:01.398507319 +0000 UTC m=+100.812967697 (1 nodes), 630ns from start of poll, 159.544016ms from cluster start. Timeout at 2025-07-14 05:40:01.504445867 +0000 UTC m=+100.918906265, 105.938568ms after stability
2025-07-14T05:40:01.506Z [ERROR] server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: peer has newer term, stopping replication: peer="{Voter server-350eb139-7aae-3488-d305-81df07adad55 350eb139-7aae-3488-d305-81df07adad55}"
2025-07-14T05:40:01.506Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: aborting pipeline replication: peer="{Voter server-350eb139-7aae-3488-d305-81df07adad55 350eb139-7aae-3488-d305-81df07adad55}"
2025-07-14T05:40:01.506Z [INFO]  server-d34124f8-2f0c-a372-c7b4-84c7903c0fcf: entering follower state: follower="Node at d34124f8-2f0c-a372-c7b4-84c7903c0fcf [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.27s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-07-14T05:40:01.507Z [INFO]  server-6d804f69-361a-5d59-aa71-bd4300349959: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d804f69-361a-5d59-aa71-bd4300349959 Address:6d804f69-361a-5d59-aa71-bd4300349959} {Suffrage:Voter ID:server-5ff2cb70-fb2a-54a8-50df-883887c76cbc Address:5ff2cb70-fb2a-54a8-50df-883887c76cbc} {Suffrage:Voter ID:server-88c583a5-9b02-be2c-210d-1d83959b578c Address:88c583a5-9b02-be2c-210d-1d83959b578c}]"
2025-07-14T05:40:01.507Z [INFO]  server-6d804f69-361a-5d59-aa71-bd4300349959: entering follower state: follower="Node at 6d804f69-361a-5d59-aa71-bd4300349959 [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.507Z [INFO]  server-5ff2cb70-fb2a-54a8-50df-883887c76cbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d804f69-361a-5d59-aa71-bd4300349959 Address:6d804f69-361a-5d59-aa71-bd4300349959} {Suffrage:Voter ID:server-5ff2cb70-fb2a-54a8-50df-883887c76cbc Address:5ff2cb70-fb2a-54a8-50df-883887c76cbc} {Suffrage:Voter ID:server-88c583a5-9b02-be2c-210d-1d83959b578c Address:88c583a5-9b02-be2c-210d-1d83959b578c}]"
2025-07-14T05:40:01.508Z [INFO]  server-5ff2cb70-fb2a-54a8-50df-883887c76cbc: entering follower state: follower="Node at 5ff2cb70-fb2a-54a8-50df-883887c76cbc [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.508Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d804f69-361a-5d59-aa71-bd4300349959 Address:6d804f69-361a-5d59-aa71-bd4300349959} {Suffrage:Voter ID:server-5ff2cb70-fb2a-54a8-50df-883887c76cbc Address:5ff2cb70-fb2a-54a8-50df-883887c76cbc} {Suffrage:Voter ID:server-88c583a5-9b02-be2c-210d-1d83959b578c Address:88c583a5-9b02-be2c-210d-1d83959b578c}]"
2025-07-14T05:40:01.508Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:01.508Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: entering follower state: follower="Node at 88c583a5-9b02-be2c-210d-1d83959b578c [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.566Z [WARN]  server-88c583a5-9b02-be2c-210d-1d83959b578c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:01.566Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: entering candidate state: node="Node at 88c583a5-9b02-be2c-210d-1d83959b578c [Candidate]" term=2
2025-07-14T05:40:01.566Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: election won: term=2 tally=2
2025-07-14T05:40:01.566Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: entering leader state: leader="Node at 88c583a5-9b02-be2c-210d-1d83959b578c [Leader]"
2025-07-14T05:40:01.566Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: added peer, starting replication: peer=server-6d804f69-361a-5d59-aa71-bd4300349959
2025-07-14T05:40:01.566Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: added peer, starting replication: peer=server-5ff2cb70-fb2a-54a8-50df-883887c76cbc
2025-07-14T05:40:01.566Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: pipelining replication: peer="{Voter server-5ff2cb70-fb2a-54a8-50df-883887c76cbc 5ff2cb70-fb2a-54a8-50df-883887c76cbc}"
2025-07-14T05:40:01.566Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: pipelining replication: peer="{Voter server-6d804f69-361a-5d59-aa71-bd4300349959 6d804f69-361a-5d59-aa71-bd4300349959}"
2025-07-14T05:40:01.671Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:01.566221411 +0000 UTC m=+100.980681809 (1 nodes), 58.006059ms from start of poll, 58.442194ms from cluster start. Timeout at 2025-07-14 05:40:01.671226738 +0000 UTC m=+101.085687107, 105.005298ms after stability
2025-07-14T05:40:01.676Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:01.782Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:01.677015812 +0000 UTC m=+101.091476180 (2 nodes), 711ns from start of poll, 169.236565ms from cluster start. Timeout at 2025-07-14 05:40:01.78228101 +0000 UTC m=+101.196741408, 105.265228ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 6d804f69-361a-5d59-aa71-bd4300349959 [Follower]
2025-07-14T05:40:01.782Z [ERROR] server-88c583a5-9b02-be2c-210d-1d83959b578c: failed to heartbeat to: peer=6d804f69-361a-5d59-aa71-bd4300349959 backoff time=10ms error="failed to connect to peer: 6d804f69-361a-5d59-aa71-bd4300349959"
2025-07-14T05:40:01.782Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: aborting pipeline replication: peer="{Voter server-5ff2cb70-fb2a-54a8-50df-883887c76cbc 5ff2cb70-fb2a-54a8-50df-883887c76cbc}"
2025-07-14T05:40:01.782Z [INFO]  server-88c583a5-9b02-be2c-210d-1d83959b578c: aborting pipeline replication: peer="{Voter server-6d804f69-361a-5d59-aa71-bd4300349959 6d804f69-361a-5d59-aa71-bd4300349959}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s)
=== RUN   TestRaft_NotifyCh
2025-07-14T05:40:01.783Z [INFO]  server-3185831c-81a3-9965-01bd-4badf417a148: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3185831c-81a3-9965-01bd-4badf417a148 Address:3185831c-81a3-9965-01bd-4badf417a148}]"
2025-07-14T05:40:01.783Z [INFO]  server-3185831c-81a3-9965-01bd-4badf417a148: entering follower state: follower="Node at 3185831c-81a3-9965-01bd-4badf417a148 [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.880Z [WARN]  server-3185831c-81a3-9965-01bd-4badf417a148: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:01.880Z [INFO]  server-3185831c-81a3-9965-01bd-4badf417a148: entering candidate state: node="Node at 3185831c-81a3-9965-01bd-4badf417a148 [Candidate]" term=2
2025-07-14T05:40:01.880Z [INFO]  server-3185831c-81a3-9965-01bd-4badf417a148: election won: term=2 tally=1
2025-07-14T05:40:01.880Z [INFO]  server-3185831c-81a3-9965-01bd-4badf417a148: entering leader state: leader="Node at 3185831c-81a3-9965-01bd-4badf417a148 [Leader]"
--- PASS: TestRaft_NotifyCh (0.10s)
=== RUN   TestRaft_AppendEntry
2025-07-14T05:40:01.881Z [INFO]  server-f85fa9fe-15cb-a61c-d057-870c25d658c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f85fa9fe-15cb-a61c-d057-870c25d658c5 Address:f85fa9fe-15cb-a61c-d057-870c25d658c5} {Suffrage:Voter ID:server-683c0a6e-e998-91ef-56d4-07e354ab8a90 Address:683c0a6e-e998-91ef-56d4-07e354ab8a90} {Suffrage:Voter ID:server-8958084c-21b0-6e10-f192-1d2608e7bcc0 Address:8958084c-21b0-6e10-f192-1d2608e7bcc0}]"
2025-07-14T05:40:01.881Z [INFO]  server-f85fa9fe-15cb-a61c-d057-870c25d658c5: entering follower state: follower="Node at f85fa9fe-15cb-a61c-d057-870c25d658c5 [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.881Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f85fa9fe-15cb-a61c-d057-870c25d658c5 Address:f85fa9fe-15cb-a61c-d057-870c25d658c5} {Suffrage:Voter ID:server-683c0a6e-e998-91ef-56d4-07e354ab8a90 Address:683c0a6e-e998-91ef-56d4-07e354ab8a90} {Suffrage:Voter ID:server-8958084c-21b0-6e10-f192-1d2608e7bcc0 Address:8958084c-21b0-6e10-f192-1d2608e7bcc0}]"
2025-07-14T05:40:01.881Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: entering follower state: follower="Node at 683c0a6e-e998-91ef-56d4-07e354ab8a90 [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.881Z [INFO]  server-8958084c-21b0-6e10-f192-1d2608e7bcc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f85fa9fe-15cb-a61c-d057-870c25d658c5 Address:f85fa9fe-15cb-a61c-d057-870c25d658c5} {Suffrage:Voter ID:server-683c0a6e-e998-91ef-56d4-07e354ab8a90 Address:683c0a6e-e998-91ef-56d4-07e354ab8a90} {Suffrage:Voter ID:server-8958084c-21b0-6e10-f192-1d2608e7bcc0 Address:8958084c-21b0-6e10-f192-1d2608e7bcc0}]"
2025-07-14T05:40:01.881Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:01.881Z [INFO]  server-8958084c-21b0-6e10-f192-1d2608e7bcc0: entering follower state: follower="Node at 8958084c-21b0-6e10-f192-1d2608e7bcc0 [Follower]" leader-address= leader-id=
2025-07-14T05:40:01.962Z [WARN]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:01.962Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: entering candidate state: node="Node at 683c0a6e-e998-91ef-56d4-07e354ab8a90 [Candidate]" term=2
2025-07-14T05:40:01.962Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: election won: term=2 tally=2
2025-07-14T05:40:01.962Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: entering leader state: leader="Node at 683c0a6e-e998-91ef-56d4-07e354ab8a90 [Leader]"
2025-07-14T05:40:01.962Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: added peer, starting replication: peer=server-f85fa9fe-15cb-a61c-d057-870c25d658c5
2025-07-14T05:40:01.962Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: added peer, starting replication: peer=server-8958084c-21b0-6e10-f192-1d2608e7bcc0
2025-07-14T05:40:01.962Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: pipelining replication: peer="{Voter server-f85fa9fe-15cb-a61c-d057-870c25d658c5 f85fa9fe-15cb-a61c-d057-870c25d658c5}"
2025-07-14T05:40:01.962Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: pipelining replication: peer="{Voter server-8958084c-21b0-6e10-f192-1d2608e7bcc0 8958084c-21b0-6e10-f192-1d2608e7bcc0}"
2025-07-14T05:40:02.067Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:01.962563774 +0000 UTC m=+101.377024142 (2 nodes), 80.607331ms from start of poll, 81.001763ms from cluster start. Timeout at 2025-07-14 05:40:02.067566238 +0000 UTC m=+101.482026606, 105.002464ms after stability
2025-07-14T05:40:02.067Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:02.172Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:02.06764787 +0000 UTC m=+101.482108238 (1 nodes), 1.432µs from start of poll, 186.085859ms from cluster start. Timeout at 2025-07-14 05:40:02.17270133 +0000 UTC m=+101.587161698, 105.05346ms after stability
2025-07-14T05:40:02.172Z [WARN]  server-f85fa9fe-15cb-a61c-d057-870c25d658c5: clearing log suffix: from=1 to=2
2025-07-14T05:40:02.172Z [WARN]  server-f85fa9fe-15cb-a61c-d057-870c25d658c5: skipping application of old log: index=1
2025-07-14T05:40:02.172Z [WARN]  server-f85fa9fe-15cb-a61c-d057-870c25d658c5: skipping application of old log: index=1
2025-07-14T05:40:02.172Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: aborting pipeline replication: peer="{Voter server-8958084c-21b0-6e10-f192-1d2608e7bcc0 8958084c-21b0-6e10-f192-1d2608e7bcc0}"
2025-07-14T05:40:02.172Z [INFO]  server-683c0a6e-e998-91ef-56d4-07e354ab8a90: aborting pipeline replication: peer="{Voter server-f85fa9fe-15cb-a61c-d057-870c25d658c5 f85fa9fe-15cb-a61c-d057-870c25d658c5}"
--- PASS: TestRaft_AppendEntry (0.29s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-07-14T05:40:02.174Z [INFO]  server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03 Address:ef9ca0ec-f516-bf5a-494a-aa792c2bee03} {Suffrage:Voter ID:server-fa76f990-463b-7df2-3d92-f84a3d613e32 Address:fa76f990-463b-7df2-3d92-f84a3d613e32} {Suffrage:Voter ID:server-590ce51d-a026-171a-684f-1221332382be Address:590ce51d-a026-171a-684f-1221332382be}]"
2025-07-14T05:40:02.174Z [INFO]  server-fa76f990-463b-7df2-3d92-f84a3d613e32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03 Address:ef9ca0ec-f516-bf5a-494a-aa792c2bee03} {Suffrage:Voter ID:server-fa76f990-463b-7df2-3d92-f84a3d613e32 Address:fa76f990-463b-7df2-3d92-f84a3d613e32} {Suffrage:Voter ID:server-590ce51d-a026-171a-684f-1221332382be Address:590ce51d-a026-171a-684f-1221332382be}]"
2025-07-14T05:40:02.174Z [INFO]  server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03: entering follower state: follower="Node at ef9ca0ec-f516-bf5a-494a-aa792c2bee03 [Follower]" leader-address= leader-id=
2025-07-14T05:40:02.174Z [INFO]  server-fa76f990-463b-7df2-3d92-f84a3d613e32: entering follower state: follower="Node at fa76f990-463b-7df2-3d92-f84a3d613e32 [Follower]" leader-address= leader-id=
2025-07-14T05:40:02.174Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03 Address:ef9ca0ec-f516-bf5a-494a-aa792c2bee03} {Suffrage:Voter ID:server-fa76f990-463b-7df2-3d92-f84a3d613e32 Address:fa76f990-463b-7df2-3d92-f84a3d613e32} {Suffrage:Voter ID:server-590ce51d-a026-171a-684f-1221332382be Address:590ce51d-a026-171a-684f-1221332382be}]"
2025-07-14T05:40:02.174Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:02.174Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: entering follower state: follower="Node at 590ce51d-a026-171a-684f-1221332382be [Follower]" leader-address= leader-id=
2025-07-14T05:40:02.236Z [WARN]  server-590ce51d-a026-171a-684f-1221332382be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:02.236Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: entering candidate state: node="Node at 590ce51d-a026-171a-684f-1221332382be [Candidate]" term=2
2025-07-14T05:40:02.236Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: election won: term=2 tally=2
2025-07-14T05:40:02.236Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: entering leader state: leader="Node at 590ce51d-a026-171a-684f-1221332382be [Leader]"
2025-07-14T05:40:02.236Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: added peer, starting replication: peer=server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03
2025-07-14T05:40:02.236Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: added peer, starting replication: peer=server-fa76f990-463b-7df2-3d92-f84a3d613e32
2025-07-14T05:40:02.236Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: pipelining replication: peer="{Voter server-fa76f990-463b-7df2-3d92-f84a3d613e32 fa76f990-463b-7df2-3d92-f84a3d613e32}"
2025-07-14T05:40:02.236Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: pipelining replication: peer="{Voter server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03 ef9ca0ec-f516-bf5a-494a-aa792c2bee03}"
2025-07-14T05:40:02.342Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:02.236837464 +0000 UTC m=+101.651297843 (2 nodes), 62.164006ms from start of poll, 62.488543ms from cluster start. Timeout at 2025-07-14 05:40:02.342506087 +0000 UTC m=+101.756966455, 105.668612ms after stability
2025-07-14T05:40:02.342Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:02.447Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:02.342573028 +0000 UTC m=+101.757033406 (1 nodes), 841ns from start of poll, 168.224106ms from cluster start. Timeout at 2025-07-14 05:40:02.447579518 +0000 UTC m=+101.862039906, 105.0065ms after stability
2025-07-14T05:40:02.447Z [WARN]  server-fa76f990-463b-7df2-3d92-f84a3d613e32: rejecting vote request since we have a leader: from=ef9ca0ec-f516-bf5a-494a-aa792c2bee03 leader=590ce51d-a026-171a-684f-1221332382be leader-id=server-590ce51d-a026-171a-684f-1221332382be
2025-07-14T05:40:02.447Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: aborting pipeline replication: peer="{Voter server-fa76f990-463b-7df2-3d92-f84a3d613e32 fa76f990-463b-7df2-3d92-f84a3d613e32}"
2025-07-14T05:40:02.447Z [INFO]  server-590ce51d-a026-171a-684f-1221332382be: aborting pipeline replication: peer="{Voter server-ef9ca0ec-f516-bf5a-494a-aa792c2bee03 ef9ca0ec-f516-bf5a-494a-aa792c2bee03}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-07-14T05:40:02.448Z [INFO]  server-1bd70273-b760-92b9-e772-b2dffdd10228: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1bd70273-b760-92b9-e772-b2dffdd10228 Address:1bd70273-b760-92b9-e772-b2dffdd10228} {Suffrage:Voter ID:server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad Address:d5345381-08f5-fad0-fa75-b9aa3cfda9ad} {Suffrage:Voter ID:server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0 Address:e0b00f13-7af9-f334-eecd-1ff9e693e9f0}]"
2025-07-14T05:40:02.448Z [INFO]  server-1bd70273-b760-92b9-e772-b2dffdd10228: entering follower state: follower="Node at 1bd70273-b760-92b9-e772-b2dffdd10228 [Follower]" leader-address= leader-id=
2025-07-14T05:40:02.448Z [INFO]  server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1bd70273-b760-92b9-e772-b2dffdd10228 Address:1bd70273-b760-92b9-e772-b2dffdd10228} {Suffrage:Voter ID:server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad Address:d5345381-08f5-fad0-fa75-b9aa3cfda9ad} {Suffrage:Voter ID:server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0 Address:e0b00f13-7af9-f334-eecd-1ff9e693e9f0}]"
2025-07-14T05:40:02.448Z [INFO]  server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad: entering follower state: follower="Node at d5345381-08f5-fad0-fa75-b9aa3cfda9ad [Follower]" leader-address= leader-id=
2025-07-14T05:40:02.449Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1bd70273-b760-92b9-e772-b2dffdd10228 Address:1bd70273-b760-92b9-e772-b2dffdd10228} {Suffrage:Voter ID:server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad Address:d5345381-08f5-fad0-fa75-b9aa3cfda9ad} {Suffrage:Voter ID:server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0 Address:e0b00f13-7af9-f334-eecd-1ff9e693e9f0}]"
2025-07-14T05:40:02.449Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:02.449Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: entering follower state: follower="Node at e0b00f13-7af9-f334-eecd-1ff9e693e9f0 [Follower]" leader-address= leader-id=
2025-07-14T05:40:02.521Z [WARN]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:02.521Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: entering candidate state: node="Node at e0b00f13-7af9-f334-eecd-1ff9e693e9f0 [Candidate]" term=2
2025-07-14T05:40:02.521Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: election won: term=2 tally=2
2025-07-14T05:40:02.521Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: entering leader state: leader="Node at e0b00f13-7af9-f334-eecd-1ff9e693e9f0 [Leader]"
2025-07-14T05:40:02.521Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: added peer, starting replication: peer=server-1bd70273-b760-92b9-e772-b2dffdd10228
2025-07-14T05:40:02.521Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: added peer, starting replication: peer=server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad
2025-07-14T05:40:02.521Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: pipelining replication: peer="{Voter server-1bd70273-b760-92b9-e772-b2dffdd10228 1bd70273-b760-92b9-e772-b2dffdd10228}"
2025-07-14T05:40:02.521Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: pipelining replication: peer="{Voter server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad d5345381-08f5-fad0-fa75-b9aa3cfda9ad}"
2025-07-14T05:40:02.627Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:02.521366718 +0000 UTC m=+101.935827096 (2 nodes), 72.297926ms from start of poll, 72.575753ms from cluster start. Timeout at 2025-07-14 05:40:02.627162362 +0000 UTC m=+102.041622760, 105.795664ms after stability
2025-07-14T05:40:02.627Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:02.732Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:02.62725467 +0000 UTC m=+102.041715048 (1 nodes), 901ns from start of poll, 178.463705ms from cluster start. Timeout at 2025-07-14 05:40:02.732799948 +0000 UTC m=+102.147260316, 105.545268ms after stability
2025-07-14T05:40:02.732Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: aborting pipeline replication: peer="{Voter server-1bd70273-b760-92b9-e772-b2dffdd10228 1bd70273-b760-92b9-e772-b2dffdd10228}"
2025-07-14T05:40:02.732Z [INFO]  server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: aborting pipeline replication: peer="{Voter server-d5345381-08f5-fad0-fa75-b9aa3cfda9ad d5345381-08f5-fad0-fa75-b9aa3cfda9ad}"
2025-07-14T05:40:03.232Z [ERROR] server-e0b00f13-7af9-f334-eecd-1ff9e693e9f0: failed to heartbeat to: peer=1bd70273-b760-92b9-e772-b2dffdd10228 backoff time=10ms error="command timed out"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.79s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-07-14T05:40:03.234Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e428928-f64b-6d67-64d5-740d41a7d00b Address:5e428928-f64b-6d67-64d5-740d41a7d00b} {Suffrage:Voter ID:server-e9405a26-dafb-0261-b27f-a1b3e9af984a Address:e9405a26-dafb-0261-b27f-a1b3e9af984a} {Suffrage:Voter ID:server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b Address:b717e3b8-fb48-e5bb-10c9-93ad0ac6933b}]"
2025-07-14T05:40:03.234Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: entering follower state: follower="Node at 5e428928-f64b-6d67-64d5-740d41a7d00b [Follower]" leader-address= leader-id=
2025-07-14T05:40:03.234Z [INFO]  server-e9405a26-dafb-0261-b27f-a1b3e9af984a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e428928-f64b-6d67-64d5-740d41a7d00b Address:5e428928-f64b-6d67-64d5-740d41a7d00b} {Suffrage:Voter ID:server-e9405a26-dafb-0261-b27f-a1b3e9af984a Address:e9405a26-dafb-0261-b27f-a1b3e9af984a} {Suffrage:Voter ID:server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b Address:b717e3b8-fb48-e5bb-10c9-93ad0ac6933b}]"
2025-07-14T05:40:03.234Z [INFO]  server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e428928-f64b-6d67-64d5-740d41a7d00b Address:5e428928-f64b-6d67-64d5-740d41a7d00b} {Suffrage:Voter ID:server-e9405a26-dafb-0261-b27f-a1b3e9af984a Address:e9405a26-dafb-0261-b27f-a1b3e9af984a} {Suffrage:Voter ID:server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b Address:b717e3b8-fb48-e5bb-10c9-93ad0ac6933b}]"
2025-07-14T05:40:03.234Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:03.234Z [INFO]  server-e9405a26-dafb-0261-b27f-a1b3e9af984a: entering follower state: follower="Node at e9405a26-dafb-0261-b27f-a1b3e9af984a [Follower]" leader-address= leader-id=
2025-07-14T05:40:03.234Z [INFO]  server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b: entering follower state: follower="Node at b717e3b8-fb48-e5bb-10c9-93ad0ac6933b [Follower]" leader-address= leader-id=
2025-07-14T05:40:03.287Z [WARN]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:03.287Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: entering candidate state: node="Node at 5e428928-f64b-6d67-64d5-740d41a7d00b [Candidate]" term=2
2025-07-14T05:40:03.287Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: election won: term=2 tally=2
2025-07-14T05:40:03.287Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: entering leader state: leader="Node at 5e428928-f64b-6d67-64d5-740d41a7d00b [Leader]"
2025-07-14T05:40:03.287Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: added peer, starting replication: peer=server-e9405a26-dafb-0261-b27f-a1b3e9af984a
2025-07-14T05:40:03.287Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: added peer, starting replication: peer=server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b
2025-07-14T05:40:03.287Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: pipelining replication: peer="{Voter server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b b717e3b8-fb48-e5bb-10c9-93ad0ac6933b}"
2025-07-14T05:40:03.287Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: pipelining replication: peer="{Voter server-e9405a26-dafb-0261-b27f-a1b3e9af984a e9405a26-dafb-0261-b27f-a1b3e9af984a}"
2025-07-14T05:40:03.393Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:03.287473248 +0000 UTC m=+102.701933626 (1 nodes), 53.156475ms from start of poll, 53.430446ms from cluster start. Timeout at 2025-07-14 05:40:03.393109653 +0000 UTC m=+102.807570051, 105.636425ms after stability
2025-07-14T05:40:03.393Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:03.498Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:03.393199487 +0000 UTC m=+102.807659865 (1 nodes), 600ns from start of poll, 159.156685ms from cluster start. Timeout at 2025-07-14 05:40:03.49895468 +0000 UTC m=+102.913415078, 105.755213ms after stability
2025-07-14T05:40:03.499Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:03.605Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:03.499104455 +0000 UTC m=+102.913564823 (1 nodes), 1.072µs from start of poll, 265.061643ms from cluster start. Timeout at 2025-07-14 05:40:03.605386498 +0000 UTC m=+103.019846886, 106.282063ms after stability
2025-07-14T05:40:03.606Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:03.712Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:03.606080058 +0000 UTC m=+103.020540436 (1 nodes), 1.081µs from start of poll, 372.037256ms from cluster start. Timeout at 2025-07-14 05:40:03.712163473 +0000 UTC m=+103.126623841, 106.083405ms after stability
2025-07-14T05:40:03.712Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:03.818Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:03.71231441 +0000 UTC m=+103.126774808 (1 nodes), 922ns from start of poll, 478.271628ms from cluster start. Timeout at 2025-07-14 05:40:03.818044395 +0000 UTC m=+103.232504783, 105.729975ms after stability
2025-07-14T05:40:03.818Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:03.924Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:03.818193238 +0000 UTC m=+103.232653636 (1 nodes), 781ns from start of poll, 584.150456ms from cluster start. Timeout at 2025-07-14 05:40:03.924188401 +0000 UTC m=+103.338648779, 105.995143ms after stability
2025-07-14T05:40:04.029Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:04.135Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:04.029418336 +0000 UTC m=+103.443878704 (1 nodes), 641ns from start of poll, 795.375524ms from cluster start. Timeout at 2025-07-14 05:40:04.135123363 +0000 UTC m=+103.549583751, 105.705047ms after stability
2025-07-14T05:40:04.135Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: aborting pipeline replication: peer="{Voter server-b717e3b8-fb48-e5bb-10c9-93ad0ac6933b b717e3b8-fb48-e5bb-10c9-93ad0ac6933b}"
2025-07-14T05:40:04.135Z [INFO]  server-5e428928-f64b-6d67-64d5-740d41a7d00b: aborting pipeline replication: peer="{Voter server-e9405a26-dafb-0261-b27f-a1b3e9af984a e9405a26-dafb-0261-b27f-a1b3e9af984a}"
--- PASS: TestRaft_LeaderID_Propagated (0.90s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-07-14T05:40:04.136Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f5ece51-27bf-67af-9b50-f25a1b5583f3 Address:1f5ece51-27bf-67af-9b50-f25a1b5583f3} {Suffrage:Voter ID:server-b4e0d883-e800-ee1c-bb53-686d8a161f04 Address:b4e0d883-e800-ee1c-bb53-686d8a161f04} {Suffrage:Voter ID:server-c76999f0-1777-2060-fa35-d2636e95746c Address:c76999f0-1777-2060-fa35-d2636e95746c}]"
2025-07-14T05:40:04.136Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: entering follower state: follower="Node at 1f5ece51-27bf-67af-9b50-f25a1b5583f3 [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.136Z [INFO]  server-b4e0d883-e800-ee1c-bb53-686d8a161f04: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f5ece51-27bf-67af-9b50-f25a1b5583f3 Address:1f5ece51-27bf-67af-9b50-f25a1b5583f3} {Suffrage:Voter ID:server-b4e0d883-e800-ee1c-bb53-686d8a161f04 Address:b4e0d883-e800-ee1c-bb53-686d8a161f04} {Suffrage:Voter ID:server-c76999f0-1777-2060-fa35-d2636e95746c Address:c76999f0-1777-2060-fa35-d2636e95746c}]"
2025-07-14T05:40:04.136Z [INFO]  server-b4e0d883-e800-ee1c-bb53-686d8a161f04: entering follower state: follower="Node at b4e0d883-e800-ee1c-bb53-686d8a161f04 [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.136Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f5ece51-27bf-67af-9b50-f25a1b5583f3 Address:1f5ece51-27bf-67af-9b50-f25a1b5583f3} {Suffrage:Voter ID:server-b4e0d883-e800-ee1c-bb53-686d8a161f04 Address:b4e0d883-e800-ee1c-bb53-686d8a161f04} {Suffrage:Voter ID:server-c76999f0-1777-2060-fa35-d2636e95746c Address:c76999f0-1777-2060-fa35-d2636e95746c}]"
2025-07-14T05:40:04.136Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:04.136Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: entering follower state: follower="Node at c76999f0-1777-2060-fa35-d2636e95746c [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.208Z [WARN]  server-c76999f0-1777-2060-fa35-d2636e95746c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:04.208Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: entering candidate state: node="Node at c76999f0-1777-2060-fa35-d2636e95746c [Candidate]" term=2
2025-07-14T05:40:04.208Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: election won: term=2 tally=2
2025-07-14T05:40:04.208Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: entering leader state: leader="Node at c76999f0-1777-2060-fa35-d2636e95746c [Leader]"
2025-07-14T05:40:04.208Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: added peer, starting replication: peer=server-1f5ece51-27bf-67af-9b50-f25a1b5583f3
2025-07-14T05:40:04.208Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: added peer, starting replication: peer=server-b4e0d883-e800-ee1c-bb53-686d8a161f04
2025-07-14T05:40:04.208Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: pipelining replication: peer="{Voter server-b4e0d883-e800-ee1c-bb53-686d8a161f04 b4e0d883-e800-ee1c-bb53-686d8a161f04}"
2025-07-14T05:40:04.208Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: pipelining replication: peer="{Voter server-1f5ece51-27bf-67af-9b50-f25a1b5583f3 1f5ece51-27bf-67af-9b50-f25a1b5583f3}"
2025-07-14T05:40:04.314Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:04.208609452 +0000 UTC m=+103.623069830 (1 nodes), 71.956184ms from start of poll, 72.228182ms from cluster start. Timeout at 2025-07-14 05:40:04.314577454 +0000 UTC m=+103.729037842, 105.968012ms after stability
2025-07-14T05:40:04.314Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:04.420Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:04.31468786 +0000 UTC m=+103.729148238 (1 nodes), 952ns from start of poll, 178.30659ms from cluster start. Timeout at 2025-07-14 05:40:04.420090794 +0000 UTC m=+103.834551162, 105.402924ms after stability
2025-07-14T05:40:04.420Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: entering candidate state: node="Node at 1f5ece51-27bf-67af-9b50-f25a1b5583f3 [Candidate]" term=3
2025-07-14T05:40:04.420Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:04.420Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: entering follower state: follower="Node at c76999f0-1777-2060-fa35-d2636e95746c [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.420Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: aborting pipeline replication: peer="{Voter server-1f5ece51-27bf-67af-9b50-f25a1b5583f3 1f5ece51-27bf-67af-9b50-f25a1b5583f3}"
2025-07-14T05:40:04.420Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: election won: term=3 tally=2
2025-07-14T05:40:04.420Z [INFO]  server-c76999f0-1777-2060-fa35-d2636e95746c: aborting pipeline replication: peer="{Voter server-b4e0d883-e800-ee1c-bb53-686d8a161f04 b4e0d883-e800-ee1c-bb53-686d8a161f04}"
2025-07-14T05:40:04.420Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: entering leader state: leader="Node at 1f5ece51-27bf-67af-9b50-f25a1b5583f3 [Leader]"
2025-07-14T05:40:04.420Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: added peer, starting replication: peer=server-b4e0d883-e800-ee1c-bb53-686d8a161f04
2025-07-14T05:40:04.420Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: added peer, starting replication: peer=server-c76999f0-1777-2060-fa35-d2636e95746c
2025-07-14T05:40:04.420Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: pipelining replication: peer="{Voter server-c76999f0-1777-2060-fa35-d2636e95746c c76999f0-1777-2060-fa35-d2636e95746c}"
2025-07-14T05:40:04.420Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: pipelining replication: peer="{Voter server-b4e0d883-e800-ee1c-bb53-686d8a161f04 b4e0d883-e800-ee1c-bb53-686d8a161f04}"
2025-07-14T05:40:04.526Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:04.420262852 +0000 UTC m=+103.834723250 (1 nodes), 44.226µs from start of poll, 283.881602ms from cluster start. Timeout at 2025-07-14 05:40:04.52618161 +0000 UTC m=+103.940641998, 105.918748ms after stability
2025-07-14T05:40:04.526Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: aborting pipeline replication: peer="{Voter server-c76999f0-1777-2060-fa35-d2636e95746c c76999f0-1777-2060-fa35-d2636e95746c}"
2025-07-14T05:40:04.526Z [INFO]  server-1f5ece51-27bf-67af-9b50-f25a1b5583f3: aborting pipeline replication: peer="{Voter server-b4e0d883-e800-ee1c-bb53-686d8a161f04 b4e0d883-e800-ee1c-bb53-686d8a161f04}"
--- PASS: TestRaft_LeadershipTransfer (0.39s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-07-14T05:40:04.527Z [INFO]  server-59279605-8b8e-8da0-98be-aad458e6d8b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59279605-8b8e-8da0-98be-aad458e6d8b4 Address:59279605-8b8e-8da0-98be-aad458e6d8b4}]"
2025-07-14T05:40:04.527Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:04.527Z [INFO]  server-59279605-8b8e-8da0-98be-aad458e6d8b4: entering follower state: follower="Node at 59279605-8b8e-8da0-98be-aad458e6d8b4 [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.602Z [WARN]  server-59279605-8b8e-8da0-98be-aad458e6d8b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:04.602Z [INFO]  server-59279605-8b8e-8da0-98be-aad458e6d8b4: entering candidate state: node="Node at 59279605-8b8e-8da0-98be-aad458e6d8b4 [Candidate]" term=2
2025-07-14T05:40:04.602Z [INFO]  server-59279605-8b8e-8da0-98be-aad458e6d8b4: election won: term=2 tally=1
2025-07-14T05:40:04.602Z [INFO]  server-59279605-8b8e-8da0-98be-aad458e6d8b4: entering leader state: leader="Node at 59279605-8b8e-8da0-98be-aad458e6d8b4 [Leader]"
2025-07-14T05:40:04.708Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:04.60224824 +0000 UTC m=+104.016708638 (1 nodes), 75.020961ms from start of poll, 75.156244ms from cluster start. Timeout at 2025-07-14 05:40:04.708359177 +0000 UTC m=+104.122819545, 106.110907ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-07-14T05:40:04.709Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14a6e24e-c05d-907c-7d18-3b72df1dc415 Address:14a6e24e-c05d-907c-7d18-3b72df1dc415} {Suffrage:Voter ID:server-257c3c23-347b-4b6f-3708-6a52bb510cc1 Address:257c3c23-347b-4b6f-3708-6a52bb510cc1} {Suffrage:Voter ID:server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e Address:a676dea3-a9dc-3c2f-b30c-b950b9b8130e} {Suffrage:Voter ID:server-53c315c7-ee1a-2d53-519b-648e7df1054c Address:53c315c7-ee1a-2d53-519b-648e7df1054c} {Suffrage:Voter ID:server-190acafb-c167-f066-4a06-aa09a337150b Address:190acafb-c167-f066-4a06-aa09a337150b} {Suffrage:Voter ID:server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 Address:8986b0f5-ee71-9ea9-9230-a2318ad3f411} {Suffrage:Voter ID:server-f92cf14b-416e-47a4-349e-6a0335e004ab Address:f92cf14b-416e-47a4-349e-6a0335e004ab}]"
2025-07-14T05:40:04.709Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: entering follower state: follower="Node at 14a6e24e-c05d-907c-7d18-3b72df1dc415 [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.709Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14a6e24e-c05d-907c-7d18-3b72df1dc415 Address:14a6e24e-c05d-907c-7d18-3b72df1dc415} {Suffrage:Voter ID:server-257c3c23-347b-4b6f-3708-6a52bb510cc1 Address:257c3c23-347b-4b6f-3708-6a52bb510cc1} {Suffrage:Voter ID:server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e Address:a676dea3-a9dc-3c2f-b30c-b950b9b8130e} {Suffrage:Voter ID:server-53c315c7-ee1a-2d53-519b-648e7df1054c Address:53c315c7-ee1a-2d53-519b-648e7df1054c} {Suffrage:Voter ID:server-190acafb-c167-f066-4a06-aa09a337150b Address:190acafb-c167-f066-4a06-aa09a337150b} {Suffrage:Voter ID:server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 Address:8986b0f5-ee71-9ea9-9230-a2318ad3f411} {Suffrage:Voter ID:server-f92cf14b-416e-47a4-349e-6a0335e004ab Address:f92cf14b-416e-47a4-349e-6a0335e004ab}]"
2025-07-14T05:40:04.709Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: entering follower state: follower="Node at 257c3c23-347b-4b6f-3708-6a52bb510cc1 [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.709Z [INFO]  server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14a6e24e-c05d-907c-7d18-3b72df1dc415 Address:14a6e24e-c05d-907c-7d18-3b72df1dc415} {Suffrage:Voter ID:server-257c3c23-347b-4b6f-3708-6a52bb510cc1 Address:257c3c23-347b-4b6f-3708-6a52bb510cc1} {Suffrage:Voter ID:server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e Address:a676dea3-a9dc-3c2f-b30c-b950b9b8130e} {Suffrage:Voter ID:server-53c315c7-ee1a-2d53-519b-648e7df1054c Address:53c315c7-ee1a-2d53-519b-648e7df1054c} {Suffrage:Voter ID:server-190acafb-c167-f066-4a06-aa09a337150b Address:190acafb-c167-f066-4a06-aa09a337150b} {Suffrage:Voter ID:server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 Address:8986b0f5-ee71-9ea9-9230-a2318ad3f411} {Suffrage:Voter ID:server-f92cf14b-416e-47a4-349e-6a0335e004ab Address:f92cf14b-416e-47a4-349e-6a0335e004ab}]"
2025-07-14T05:40:04.710Z [INFO]  server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e: entering follower state: follower="Node at a676dea3-a9dc-3c2f-b30c-b950b9b8130e [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.710Z [INFO]  server-53c315c7-ee1a-2d53-519b-648e7df1054c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14a6e24e-c05d-907c-7d18-3b72df1dc415 Address:14a6e24e-c05d-907c-7d18-3b72df1dc415} {Suffrage:Voter ID:server-257c3c23-347b-4b6f-3708-6a52bb510cc1 Address:257c3c23-347b-4b6f-3708-6a52bb510cc1} {Suffrage:Voter ID:server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e Address:a676dea3-a9dc-3c2f-b30c-b950b9b8130e} {Suffrage:Voter ID:server-53c315c7-ee1a-2d53-519b-648e7df1054c Address:53c315c7-ee1a-2d53-519b-648e7df1054c} {Suffrage:Voter ID:server-190acafb-c167-f066-4a06-aa09a337150b Address:190acafb-c167-f066-4a06-aa09a337150b} {Suffrage:Voter ID:server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 Address:8986b0f5-ee71-9ea9-9230-a2318ad3f411} {Suffrage:Voter ID:server-f92cf14b-416e-47a4-349e-6a0335e004ab Address:f92cf14b-416e-47a4-349e-6a0335e004ab}]"
2025-07-14T05:40:04.710Z [INFO]  server-53c315c7-ee1a-2d53-519b-648e7df1054c: entering follower state: follower="Node at 53c315c7-ee1a-2d53-519b-648e7df1054c [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.710Z [INFO]  server-190acafb-c167-f066-4a06-aa09a337150b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14a6e24e-c05d-907c-7d18-3b72df1dc415 Address:14a6e24e-c05d-907c-7d18-3b72df1dc415} {Suffrage:Voter ID:server-257c3c23-347b-4b6f-3708-6a52bb510cc1 Address:257c3c23-347b-4b6f-3708-6a52bb510cc1} {Suffrage:Voter ID:server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e Address:a676dea3-a9dc-3c2f-b30c-b950b9b8130e} {Suffrage:Voter ID:server-53c315c7-ee1a-2d53-519b-648e7df1054c Address:53c315c7-ee1a-2d53-519b-648e7df1054c} {Suffrage:Voter ID:server-190acafb-c167-f066-4a06-aa09a337150b Address:190acafb-c167-f066-4a06-aa09a337150b} {Suffrage:Voter ID:server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 Address:8986b0f5-ee71-9ea9-9230-a2318ad3f411} {Suffrage:Voter ID:server-f92cf14b-416e-47a4-349e-6a0335e004ab Address:f92cf14b-416e-47a4-349e-6a0335e004ab}]"
2025-07-14T05:40:04.710Z [INFO]  server-190acafb-c167-f066-4a06-aa09a337150b: entering follower state: follower="Node at 190acafb-c167-f066-4a06-aa09a337150b [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.710Z [INFO]  server-8986b0f5-ee71-9ea9-9230-a2318ad3f411: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14a6e24e-c05d-907c-7d18-3b72df1dc415 Address:14a6e24e-c05d-907c-7d18-3b72df1dc415} {Suffrage:Voter ID:server-257c3c23-347b-4b6f-3708-6a52bb510cc1 Address:257c3c23-347b-4b6f-3708-6a52bb510cc1} {Suffrage:Voter ID:server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e Address:a676dea3-a9dc-3c2f-b30c-b950b9b8130e} {Suffrage:Voter ID:server-53c315c7-ee1a-2d53-519b-648e7df1054c Address:53c315c7-ee1a-2d53-519b-648e7df1054c} {Suffrage:Voter ID:server-190acafb-c167-f066-4a06-aa09a337150b Address:190acafb-c167-f066-4a06-aa09a337150b} {Suffrage:Voter ID:server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 Address:8986b0f5-ee71-9ea9-9230-a2318ad3f411} {Suffrage:Voter ID:server-f92cf14b-416e-47a4-349e-6a0335e004ab Address:f92cf14b-416e-47a4-349e-6a0335e004ab}]"
2025-07-14T05:40:04.710Z [INFO]  server-8986b0f5-ee71-9ea9-9230-a2318ad3f411: entering follower state: follower="Node at 8986b0f5-ee71-9ea9-9230-a2318ad3f411 [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.710Z [INFO]  server-f92cf14b-416e-47a4-349e-6a0335e004ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14a6e24e-c05d-907c-7d18-3b72df1dc415 Address:14a6e24e-c05d-907c-7d18-3b72df1dc415} {Suffrage:Voter ID:server-257c3c23-347b-4b6f-3708-6a52bb510cc1 Address:257c3c23-347b-4b6f-3708-6a52bb510cc1} {Suffrage:Voter ID:server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e Address:a676dea3-a9dc-3c2f-b30c-b950b9b8130e} {Suffrage:Voter ID:server-53c315c7-ee1a-2d53-519b-648e7df1054c Address:53c315c7-ee1a-2d53-519b-648e7df1054c} {Suffrage:Voter ID:server-190acafb-c167-f066-4a06-aa09a337150b Address:190acafb-c167-f066-4a06-aa09a337150b} {Suffrage:Voter ID:server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 Address:8986b0f5-ee71-9ea9-9230-a2318ad3f411} {Suffrage:Voter ID:server-f92cf14b-416e-47a4-349e-6a0335e004ab Address:f92cf14b-416e-47a4-349e-6a0335e004ab}]"
2025-07-14T05:40:04.710Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:04.710Z [INFO]  server-f92cf14b-416e-47a4-349e-6a0335e004ab: entering follower state: follower="Node at f92cf14b-416e-47a4-349e-6a0335e004ab [Follower]" leader-address= leader-id=
2025-07-14T05:40:04.762Z [WARN]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: entering candidate state: node="Node at 14a6e24e-c05d-907c-7d18-3b72df1dc415 [Candidate]" term=2
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: election won: term=2 tally=4
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: entering leader state: leader="Node at 14a6e24e-c05d-907c-7d18-3b72df1dc415 [Leader]"
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: added peer, starting replication: peer=server-257c3c23-347b-4b6f-3708-6a52bb510cc1
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: added peer, starting replication: peer=server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: added peer, starting replication: peer=server-53c315c7-ee1a-2d53-519b-648e7df1054c
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: added peer, starting replication: peer=server-190acafb-c167-f066-4a06-aa09a337150b
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: added peer, starting replication: peer=server-8986b0f5-ee71-9ea9-9230-a2318ad3f411
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: added peer, starting replication: peer=server-f92cf14b-416e-47a4-349e-6a0335e004ab
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: pipelining replication: peer="{Voter server-190acafb-c167-f066-4a06-aa09a337150b 190acafb-c167-f066-4a06-aa09a337150b}"
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: pipelining replication: peer="{Voter server-f92cf14b-416e-47a4-349e-6a0335e004ab f92cf14b-416e-47a4-349e-6a0335e004ab}"
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: pipelining replication: peer="{Voter server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e a676dea3-a9dc-3c2f-b30c-b950b9b8130e}"
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: pipelining replication: peer="{Voter server-53c315c7-ee1a-2d53-519b-648e7df1054c 53c315c7-ee1a-2d53-519b-648e7df1054c}"
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: pipelining replication: peer="{Voter server-257c3c23-347b-4b6f-3708-6a52bb510cc1 257c3c23-347b-4b6f-3708-6a52bb510cc1}"
2025-07-14T05:40:04.762Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: pipelining replication: peer="{Voter server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 8986b0f5-ee71-9ea9-9230-a2318ad3f411}"
2025-07-14T05:40:04.868Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:04.76230829 +0000 UTC m=+104.176768688 (1 nodes), 51.94022ms from start of poll, 52.58003ms from cluster start. Timeout at 2025-07-14 05:40:04.86834125 +0000 UTC m=+104.282801628, 106.03294ms after stability
2025-07-14T05:40:04.868Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:04.973Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:04.8683946 +0000 UTC m=+104.282854968 (6 nodes), 1.342µs from start of poll, 158.66631ms from cluster start. Timeout at 2025-07-14 05:40:04.973683453 +0000 UTC m=+104.388143841, 105.288873ms after stability
2025-07-14T05:40:04.973Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:05.078Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:04.973745977 +0000 UTC m=+104.388206355 (1 nodes), 831ns from start of poll, 264.017697ms from cluster start. Timeout at 2025-07-14 05:40:05.078935621 +0000 UTC m=+104.493395989, 105.189634ms after stability
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: entering candidate state: node="Node at 257c3c23-347b-4b6f-3708-6a52bb510cc1 [Candidate]" term=3
2025-07-14T05:40:05.079Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: election won: term=3 tally=4
2025-07-14T05:40:05.079Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: entering follower state: follower="Node at 14a6e24e-c05d-907c-7d18-3b72df1dc415 [Follower]" leader-address= leader-id=
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: entering leader state: leader="Node at 257c3c23-347b-4b6f-3708-6a52bb510cc1 [Leader]"
2025-07-14T05:40:05.079Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: aborting pipeline replication: peer="{Voter server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 8986b0f5-ee71-9ea9-9230-a2318ad3f411}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: added peer, starting replication: peer=server-14a6e24e-c05d-907c-7d18-3b72df1dc415
2025-07-14T05:40:05.079Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: aborting pipeline replication: peer="{Voter server-53c315c7-ee1a-2d53-519b-648e7df1054c 53c315c7-ee1a-2d53-519b-648e7df1054c}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: added peer, starting replication: peer=server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: added peer, starting replication: peer=server-53c315c7-ee1a-2d53-519b-648e7df1054c
2025-07-14T05:40:05.079Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: aborting pipeline replication: peer="{Voter server-190acafb-c167-f066-4a06-aa09a337150b 190acafb-c167-f066-4a06-aa09a337150b}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: added peer, starting replication: peer=server-190acafb-c167-f066-4a06-aa09a337150b
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: added peer, starting replication: peer=server-8986b0f5-ee71-9ea9-9230-a2318ad3f411
2025-07-14T05:40:05.079Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: aborting pipeline replication: peer="{Voter server-257c3c23-347b-4b6f-3708-6a52bb510cc1 257c3c23-347b-4b6f-3708-6a52bb510cc1}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: added peer, starting replication: peer=server-f92cf14b-416e-47a4-349e-6a0335e004ab
2025-07-14T05:40:05.079Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: aborting pipeline replication: peer="{Voter server-f92cf14b-416e-47a4-349e-6a0335e004ab f92cf14b-416e-47a4-349e-6a0335e004ab}"
2025-07-14T05:40:05.079Z [INFO]  server-14a6e24e-c05d-907c-7d18-3b72df1dc415: aborting pipeline replication: peer="{Voter server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e a676dea3-a9dc-3c2f-b30c-b950b9b8130e}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: pipelining replication: peer="{Voter server-14a6e24e-c05d-907c-7d18-3b72df1dc415 14a6e24e-c05d-907c-7d18-3b72df1dc415}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: pipelining replication: peer="{Voter server-53c315c7-ee1a-2d53-519b-648e7df1054c 53c315c7-ee1a-2d53-519b-648e7df1054c}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: pipelining replication: peer="{Voter server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 8986b0f5-ee71-9ea9-9230-a2318ad3f411}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: pipelining replication: peer="{Voter server-190acafb-c167-f066-4a06-aa09a337150b 190acafb-c167-f066-4a06-aa09a337150b}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: pipelining replication: peer="{Voter server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e a676dea3-a9dc-3c2f-b30c-b950b9b8130e}"
2025-07-14T05:40:05.079Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: pipelining replication: peer="{Voter server-f92cf14b-416e-47a4-349e-6a0335e004ab f92cf14b-416e-47a4-349e-6a0335e004ab}"
2025-07-14T05:40:05.184Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:05.079227139 +0000 UTC m=+104.493687537 (1 nodes), 117.707µs from start of poll, 369.498879ms from cluster start. Timeout at 2025-07-14 05:40:05.184713458 +0000 UTC m=+104.599173846, 105.486309ms after stability
2025-07-14T05:40:05.184Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: aborting pipeline replication: peer="{Voter server-14a6e24e-c05d-907c-7d18-3b72df1dc415 14a6e24e-c05d-907c-7d18-3b72df1dc415}"
2025-07-14T05:40:05.184Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: aborting pipeline replication: peer="{Voter server-f92cf14b-416e-47a4-349e-6a0335e004ab f92cf14b-416e-47a4-349e-6a0335e004ab}"
2025-07-14T05:40:05.184Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: aborting pipeline replication: peer="{Voter server-53c315c7-ee1a-2d53-519b-648e7df1054c 53c315c7-ee1a-2d53-519b-648e7df1054c}"
2025-07-14T05:40:05.184Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: aborting pipeline replication: peer="{Voter server-a676dea3-a9dc-3c2f-b30c-b950b9b8130e a676dea3-a9dc-3c2f-b30c-b950b9b8130e}"
2025-07-14T05:40:05.184Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: aborting pipeline replication: peer="{Voter server-190acafb-c167-f066-4a06-aa09a337150b 190acafb-c167-f066-4a06-aa09a337150b}"
2025-07-14T05:40:05.184Z [INFO]  server-257c3c23-347b-4b6f-3708-6a52bb510cc1: aborting pipeline replication: peer="{Voter server-8986b0f5-ee71-9ea9-9230-a2318ad3f411 8986b0f5-ee71-9ea9-9230-a2318ad3f411}"
2025-07-14T05:40:05.685Z [ERROR] server-257c3c23-347b-4b6f-3708-6a52bb510cc1: failed to heartbeat to: peer=8986b0f5-ee71-9ea9-9230-a2318ad3f411 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.98s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-07-14T05:40:05.687Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31 Address:e7fd76ab-8e77-0a1e-1f76-523dca33bf31} {Suffrage:Voter ID:server-91bb7029-8a51-3303-358c-7112fd3c3c40 Address:91bb7029-8a51-3303-358c-7112fd3c3c40} {Suffrage:Voter ID:server-a7a67950-4971-04f5-9215-c2be107ec262 Address:a7a67950-4971-04f5-9215-c2be107ec262}]"
2025-07-14T05:40:05.687Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: entering follower state: follower="Node at e7fd76ab-8e77-0a1e-1f76-523dca33bf31 [Follower]" leader-address= leader-id=
2025-07-14T05:40:05.687Z [INFO]  server-91bb7029-8a51-3303-358c-7112fd3c3c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31 Address:e7fd76ab-8e77-0a1e-1f76-523dca33bf31} {Suffrage:Voter ID:server-91bb7029-8a51-3303-358c-7112fd3c3c40 Address:91bb7029-8a51-3303-358c-7112fd3c3c40} {Suffrage:Voter ID:server-a7a67950-4971-04f5-9215-c2be107ec262 Address:a7a67950-4971-04f5-9215-c2be107ec262}]"
2025-07-14T05:40:05.687Z [INFO]  server-91bb7029-8a51-3303-358c-7112fd3c3c40: entering follower state: follower="Node at 91bb7029-8a51-3303-358c-7112fd3c3c40 [Follower]" leader-address= leader-id=
2025-07-14T05:40:05.687Z [INFO]  server-a7a67950-4971-04f5-9215-c2be107ec262: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31 Address:e7fd76ab-8e77-0a1e-1f76-523dca33bf31} {Suffrage:Voter ID:server-91bb7029-8a51-3303-358c-7112fd3c3c40 Address:91bb7029-8a51-3303-358c-7112fd3c3c40} {Suffrage:Voter ID:server-a7a67950-4971-04f5-9215-c2be107ec262 Address:a7a67950-4971-04f5-9215-c2be107ec262}]"
2025-07-14T05:40:05.687Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:05.687Z [INFO]  server-a7a67950-4971-04f5-9215-c2be107ec262: entering follower state: follower="Node at a7a67950-4971-04f5-9215-c2be107ec262 [Follower]" leader-address= leader-id=
2025-07-14T05:40:05.763Z [WARN]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:05.763Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: entering candidate state: node="Node at e7fd76ab-8e77-0a1e-1f76-523dca33bf31 [Candidate]" term=2
2025-07-14T05:40:05.763Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: election won: term=2 tally=2
2025-07-14T05:40:05.763Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: entering leader state: leader="Node at e7fd76ab-8e77-0a1e-1f76-523dca33bf31 [Leader]"
2025-07-14T05:40:05.763Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: added peer, starting replication: peer=server-91bb7029-8a51-3303-358c-7112fd3c3c40
2025-07-14T05:40:05.763Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: added peer, starting replication: peer=server-a7a67950-4971-04f5-9215-c2be107ec262
2025-07-14T05:40:05.763Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: pipelining replication: peer="{Voter server-91bb7029-8a51-3303-358c-7112fd3c3c40 91bb7029-8a51-3303-358c-7112fd3c3c40}"
2025-07-14T05:40:05.763Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: pipelining replication: peer="{Voter server-a7a67950-4971-04f5-9215-c2be107ec262 a7a67950-4971-04f5-9215-c2be107ec262}"
2025-07-14T05:40:05.869Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:05.763320008 +0000 UTC m=+105.177780376 (1 nodes), 75.632459ms from start of poll, 75.90657ms from cluster start. Timeout at 2025-07-14 05:40:05.869051916 +0000 UTC m=+105.283512304, 105.731928ms after stability
2025-07-14T05:40:05.869Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: aborting pipeline replication: peer="{Voter server-a7a67950-4971-04f5-9215-c2be107ec262 a7a67950-4971-04f5-9215-c2be107ec262}"
2025-07-14T05:40:05.869Z [INFO]  server-e7fd76ab-8e77-0a1e-1f76-523dca33bf31: aborting pipeline replication: peer="{Voter server-91bb7029-8a51-3303-358c-7112fd3c3c40 91bb7029-8a51-3303-358c-7112fd3c3c40}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-07-14T05:40:05.870Z [INFO]  server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081 Address:dc002861-2f50-8e3b-d6bc-2ea7a32d8081} {Suffrage:Voter ID:server-019c0dfb-d2bf-affe-83a3-3243a2d53213 Address:019c0dfb-d2bf-affe-83a3-3243a2d53213} {Suffrage:Voter ID:server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c Address:7a4118c6-2aa0-d648-daa8-f49b3e789b8c}]"
2025-07-14T05:40:05.870Z [INFO]  server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081: entering follower state: follower="Node at dc002861-2f50-8e3b-d6bc-2ea7a32d8081 [Follower]" leader-address= leader-id=
2025-07-14T05:40:05.870Z [INFO]  server-019c0dfb-d2bf-affe-83a3-3243a2d53213: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081 Address:dc002861-2f50-8e3b-d6bc-2ea7a32d8081} {Suffrage:Voter ID:server-019c0dfb-d2bf-affe-83a3-3243a2d53213 Address:019c0dfb-d2bf-affe-83a3-3243a2d53213} {Suffrage:Voter ID:server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c Address:7a4118c6-2aa0-d648-daa8-f49b3e789b8c}]"
2025-07-14T05:40:05.870Z [INFO]  server-019c0dfb-d2bf-affe-83a3-3243a2d53213: entering follower state: follower="Node at 019c0dfb-d2bf-affe-83a3-3243a2d53213 [Follower]" leader-address= leader-id=
2025-07-14T05:40:05.870Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081 Address:dc002861-2f50-8e3b-d6bc-2ea7a32d8081} {Suffrage:Voter ID:server-019c0dfb-d2bf-affe-83a3-3243a2d53213 Address:019c0dfb-d2bf-affe-83a3-3243a2d53213} {Suffrage:Voter ID:server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c Address:7a4118c6-2aa0-d648-daa8-f49b3e789b8c}]"
2025-07-14T05:40:05.870Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:05.870Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: entering follower state: follower="Node at 7a4118c6-2aa0-d648-daa8-f49b3e789b8c [Follower]" leader-address= leader-id=
2025-07-14T05:40:05.924Z [WARN]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:05.924Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: entering candidate state: node="Node at 7a4118c6-2aa0-d648-daa8-f49b3e789b8c [Candidate]" term=2
2025-07-14T05:40:05.924Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: election won: term=2 tally=2
2025-07-14T05:40:05.924Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: entering leader state: leader="Node at 7a4118c6-2aa0-d648-daa8-f49b3e789b8c [Leader]"
2025-07-14T05:40:05.924Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: added peer, starting replication: peer=server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081
2025-07-14T05:40:05.924Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: added peer, starting replication: peer=server-019c0dfb-d2bf-affe-83a3-3243a2d53213
2025-07-14T05:40:05.924Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: pipelining replication: peer="{Voter server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081 dc002861-2f50-8e3b-d6bc-2ea7a32d8081}"
2025-07-14T05:40:05.924Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: pipelining replication: peer="{Voter server-019c0dfb-d2bf-affe-83a3-3243a2d53213 019c0dfb-d2bf-affe-83a3-3243a2d53213}"
2025-07-14T05:40:06.030Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:05.92446843 +0000 UTC m=+105.338928828 (2 nodes), 54.023785ms from start of poll, 54.282113ms from cluster start. Timeout at 2025-07-14 05:40:06.030363333 +0000 UTC m=+105.444823701, 105.894873ms after stability
2025-07-14T05:40:06.030Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:06.135Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:06.030445596 +0000 UTC m=+105.444905974 (1 nodes), 921ns from start of poll, 160.259259ms from cluster start. Timeout at 2025-07-14 05:40:06.135464675 +0000 UTC m=+105.549925083, 105.019109ms after stability
2025-07-14T05:40:06.135Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: aborting pipeline replication: peer="{Voter server-019c0dfb-d2bf-affe-83a3-3243a2d53213 019c0dfb-d2bf-affe-83a3-3243a2d53213}"
2025-07-14T05:40:06.135Z [INFO]  server-7a4118c6-2aa0-d648-daa8-f49b3e789b8c: aborting pipeline replication: peer="{Voter server-dc002861-2f50-8e3b-d6bc-2ea7a32d8081 dc002861-2f50-8e3b-d6bc-2ea7a32d8081}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-07-14T05:40:06.136Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 Address:9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1} {Suffrage:Voter ID:server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a Address:7e0e4ceb-f5c2-988b-2177-af78c89ca61a} {Suffrage:Voter ID:server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a Address:c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a}]"
2025-07-14T05:40:06.136Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: entering follower state: follower="Node at 9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.136Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 Address:9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1} {Suffrage:Voter ID:server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a Address:7e0e4ceb-f5c2-988b-2177-af78c89ca61a} {Suffrage:Voter ID:server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a Address:c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a}]"
2025-07-14T05:40:06.136Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: entering follower state: follower="Node at 7e0e4ceb-f5c2-988b-2177-af78c89ca61a [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.136Z [INFO]  server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 Address:9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1} {Suffrage:Voter ID:server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a Address:7e0e4ceb-f5c2-988b-2177-af78c89ca61a} {Suffrage:Voter ID:server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a Address:c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a}]"
2025-07-14T05:40:06.136Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:06.136Z [INFO]  server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a: entering follower state: follower="Node at c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.198Z [WARN]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:06.198Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: entering candidate state: node="Node at 7e0e4ceb-f5c2-988b-2177-af78c89ca61a [Candidate]" term=2
2025-07-14T05:40:06.198Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: election won: term=2 tally=2
2025-07-14T05:40:06.198Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: entering leader state: leader="Node at 7e0e4ceb-f5c2-988b-2177-af78c89ca61a [Leader]"
2025-07-14T05:40:06.198Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: added peer, starting replication: peer=server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1
2025-07-14T05:40:06.198Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: added peer, starting replication: peer=server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a
2025-07-14T05:40:06.198Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: pipelining replication: peer="{Voter server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1}"
2025-07-14T05:40:06.198Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: pipelining replication: peer="{Voter server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a}"
2025-07-14T05:40:06.304Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:06.198609071 +0000 UTC m=+105.613069449 (1 nodes), 61.737085ms from start of poll, 62.001111ms from cluster start. Timeout at 2025-07-14 05:40:06.304630634 +0000 UTC m=+105.719091032, 106.021583ms after stability
2025-07-14T05:40:06.304Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:06.409Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:06.30470145 +0000 UTC m=+105.719161838 (2 nodes), 1.302µs from start of poll, 168.0935ms from cluster start. Timeout at 2025-07-14 05:40:06.409845676 +0000 UTC m=+105.824306054, 105.144216ms after stability
2025-07-14T05:40:06.444Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: entering candidate state: node="Node at 9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 [Candidate]" term=3
2025-07-14T05:40:06.444Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:06.444Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: election won: term=3 tally=2
2025-07-14T05:40:06.444Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: entering leader state: leader="Node at 9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 [Leader]"
2025-07-14T05:40:06.444Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: added peer, starting replication: peer=server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a
2025-07-14T05:40:06.444Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: entering follower state: follower="Node at 7e0e4ceb-f5c2-988b-2177-af78c89ca61a [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.444Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: added peer, starting replication: peer=server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a
2025-07-14T05:40:06.444Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: pipelining replication: peer="{Voter server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a 7e0e4ceb-f5c2-988b-2177-af78c89ca61a}"
2025-07-14T05:40:06.444Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: pipelining replication: peer="{Voter server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a}"
2025-07-14T05:40:06.444Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: aborting pipeline replication: peer="{Voter server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a}"
2025-07-14T05:40:06.444Z [INFO]  server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a: aborting pipeline replication: peer="{Voter server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1 9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1}"
2025-07-14T05:40:06.550Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:06.444652166 +0000 UTC m=+105.859112544 (1 nodes), 98.918µs from start of poll, 308.044206ms from cluster start. Timeout at 2025-07-14 05:40:06.549973438 +0000 UTC m=+105.964433806, 105.321262ms after stability
2025-07-14T05:40:06.550Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: aborting pipeline replication: peer="{Voter server-c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a c6c7ea69-6e20-9b84-2a17-10e18f3e7b6a}"
2025-07-14T05:40:06.550Z [INFO]  server-9cf08094-8cb4-d808-ba7c-2ffc23ea3cf1: aborting pipeline replication: peer="{Voter server-7e0e4ceb-f5c2-988b-2177-af78c89ca61a 7e0e4ceb-f5c2-988b-2177-af78c89ca61a}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-07-14T05:40:06.562Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41c401f5-6306-93ae-5d29-ee84720b0a4c Address:41c401f5-6306-93ae-5d29-ee84720b0a4c} {Suffrage:Voter ID:server-c4734929-49d6-fd22-121c-3f702b058164 Address:c4734929-49d6-fd22-121c-3f702b058164} {Suffrage:Voter ID:server-4cf8cfce-eee8-5823-ffc2-98751def4889 Address:4cf8cfce-eee8-5823-ffc2-98751def4889}]"
2025-07-14T05:40:06.562Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: entering follower state: follower="Node at 41c401f5-6306-93ae-5d29-ee84720b0a4c [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.562Z [INFO]  server-c4734929-49d6-fd22-121c-3f702b058164: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41c401f5-6306-93ae-5d29-ee84720b0a4c Address:41c401f5-6306-93ae-5d29-ee84720b0a4c} {Suffrage:Voter ID:server-c4734929-49d6-fd22-121c-3f702b058164 Address:c4734929-49d6-fd22-121c-3f702b058164} {Suffrage:Voter ID:server-4cf8cfce-eee8-5823-ffc2-98751def4889 Address:4cf8cfce-eee8-5823-ffc2-98751def4889}]"
2025-07-14T05:40:06.562Z [INFO]  server-c4734929-49d6-fd22-121c-3f702b058164: entering follower state: follower="Node at c4734929-49d6-fd22-121c-3f702b058164 [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.562Z [INFO]  server-4cf8cfce-eee8-5823-ffc2-98751def4889: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41c401f5-6306-93ae-5d29-ee84720b0a4c Address:41c401f5-6306-93ae-5d29-ee84720b0a4c} {Suffrage:Voter ID:server-c4734929-49d6-fd22-121c-3f702b058164 Address:c4734929-49d6-fd22-121c-3f702b058164} {Suffrage:Voter ID:server-4cf8cfce-eee8-5823-ffc2-98751def4889 Address:4cf8cfce-eee8-5823-ffc2-98751def4889}]"
2025-07-14T05:40:06.562Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:06.562Z [INFO]  server-4cf8cfce-eee8-5823-ffc2-98751def4889: entering follower state: follower="Node at 4cf8cfce-eee8-5823-ffc2-98751def4889 [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.649Z [WARN]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:06.649Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: entering candidate state: node="Node at 41c401f5-6306-93ae-5d29-ee84720b0a4c [Candidate]" term=2
2025-07-14T05:40:06.649Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: election won: term=2 tally=2
2025-07-14T05:40:06.649Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: entering leader state: leader="Node at 41c401f5-6306-93ae-5d29-ee84720b0a4c [Leader]"
2025-07-14T05:40:06.649Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: added peer, starting replication: peer=server-c4734929-49d6-fd22-121c-3f702b058164
2025-07-14T05:40:06.649Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: added peer, starting replication: peer=server-4cf8cfce-eee8-5823-ffc2-98751def4889
2025-07-14T05:40:06.649Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: pipelining replication: peer="{Voter server-c4734929-49d6-fd22-121c-3f702b058164 c4734929-49d6-fd22-121c-3f702b058164}"
2025-07-14T05:40:06.649Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: pipelining replication: peer="{Voter server-4cf8cfce-eee8-5823-ffc2-98751def4889 4cf8cfce-eee8-5823-ffc2-98751def4889}"
2025-07-14T05:40:06.755Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:06.649562081 +0000 UTC m=+106.064022459 (1 nodes), 87.1532ms from start of poll, 87.485078ms from cluster start. Timeout at 2025-07-14 05:40:06.75563522 +0000 UTC m=+106.170095599, 106.07314ms after stability
2025-07-14T05:40:06.755Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: cannot transfer leadership to itself
2025-07-14T05:40:06.755Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: aborting pipeline replication: peer="{Voter server-c4734929-49d6-fd22-121c-3f702b058164 c4734929-49d6-fd22-121c-3f702b058164}"
2025-07-14T05:40:06.755Z [INFO]  server-41c401f5-6306-93ae-5d29-ee84720b0a4c: aborting pipeline replication: peer="{Voter server-4cf8cfce-eee8-5823-ffc2-98751def4889 4cf8cfce-eee8-5823-ffc2-98751def4889}"
--- PASS: TestRaft_LeadershipTransferToItself (0.20s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-07-14T05:40:06.756Z [INFO]  server-41153349-afc7-0e99-663a-8e34b97dba27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41153349-afc7-0e99-663a-8e34b97dba27 Address:41153349-afc7-0e99-663a-8e34b97dba27} {Suffrage:Voter ID:server-348a540a-e6aa-5599-d39b-e2382801c487 Address:348a540a-e6aa-5599-d39b-e2382801c487} {Suffrage:Voter ID:server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33 Address:46c4f6a1-9387-6fc6-abe1-74bdb1363a33}]"
2025-07-14T05:40:06.756Z [INFO]  server-41153349-afc7-0e99-663a-8e34b97dba27: entering follower state: follower="Node at 41153349-afc7-0e99-663a-8e34b97dba27 [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.757Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41153349-afc7-0e99-663a-8e34b97dba27 Address:41153349-afc7-0e99-663a-8e34b97dba27} {Suffrage:Voter ID:server-348a540a-e6aa-5599-d39b-e2382801c487 Address:348a540a-e6aa-5599-d39b-e2382801c487} {Suffrage:Voter ID:server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33 Address:46c4f6a1-9387-6fc6-abe1-74bdb1363a33}]"
2025-07-14T05:40:06.757Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: entering follower state: follower="Node at 348a540a-e6aa-5599-d39b-e2382801c487 [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.757Z [INFO]  server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41153349-afc7-0e99-663a-8e34b97dba27 Address:41153349-afc7-0e99-663a-8e34b97dba27} {Suffrage:Voter ID:server-348a540a-e6aa-5599-d39b-e2382801c487 Address:348a540a-e6aa-5599-d39b-e2382801c487} {Suffrage:Voter ID:server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33 Address:46c4f6a1-9387-6fc6-abe1-74bdb1363a33}]"
2025-07-14T05:40:06.757Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:06.757Z [INFO]  server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33: entering follower state: follower="Node at 46c4f6a1-9387-6fc6-abe1-74bdb1363a33 [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.829Z [WARN]  server-348a540a-e6aa-5599-d39b-e2382801c487: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:06.829Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: entering candidate state: node="Node at 348a540a-e6aa-5599-d39b-e2382801c487 [Candidate]" term=2
2025-07-14T05:40:06.829Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: election won: term=2 tally=2
2025-07-14T05:40:06.829Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: entering leader state: leader="Node at 348a540a-e6aa-5599-d39b-e2382801c487 [Leader]"
2025-07-14T05:40:06.829Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: added peer, starting replication: peer=server-41153349-afc7-0e99-663a-8e34b97dba27
2025-07-14T05:40:06.829Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: added peer, starting replication: peer=server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33
2025-07-14T05:40:06.829Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: pipelining replication: peer="{Voter server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33 46c4f6a1-9387-6fc6-abe1-74bdb1363a33}"
2025-07-14T05:40:06.829Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: pipelining replication: peer="{Voter server-41153349-afc7-0e99-663a-8e34b97dba27 41153349-afc7-0e99-663a-8e34b97dba27}"
2025-07-14T05:40:06.934Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:06.829172817 +0000 UTC m=+106.243633215 (1 nodes), 72.023415ms from start of poll, 72.3265ms from cluster start. Timeout at 2025-07-14 05:40:06.934702581 +0000 UTC m=+106.349162960, 105.529745ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-07-14T05:40:06.935Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: aborting pipeline replication: peer="{Voter server-46c4f6a1-9387-6fc6-abe1-74bdb1363a33 46c4f6a1-9387-6fc6-abe1-74bdb1363a33}"
2025-07-14T05:40:06.935Z [INFO]  server-348a540a-e6aa-5599-d39b-e2382801c487: aborting pipeline replication: peer="{Voter server-41153349-afc7-0e99-663a-8e34b97dba27 41153349-afc7-0e99-663a-8e34b97dba27}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.18s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-07-14T05:40:06.936Z [INFO]  server-98208dfd-10ab-fe00-85ee-ee7b07f88626: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98208dfd-10ab-fe00-85ee-ee7b07f88626 Address:98208dfd-10ab-fe00-85ee-ee7b07f88626} {Suffrage:Voter ID:server-fed7e170-552d-ae1f-9ded-7416c65e577e Address:fed7e170-552d-ae1f-9ded-7416c65e577e} {Suffrage:Voter ID:server-27a96b4c-0fa7-abd1-aa05-27c42b649a74 Address:27a96b4c-0fa7-abd1-aa05-27c42b649a74}]"
2025-07-14T05:40:06.936Z [INFO]  server-98208dfd-10ab-fe00-85ee-ee7b07f88626: entering follower state: follower="Node at 98208dfd-10ab-fe00-85ee-ee7b07f88626 [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.936Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98208dfd-10ab-fe00-85ee-ee7b07f88626 Address:98208dfd-10ab-fe00-85ee-ee7b07f88626} {Suffrage:Voter ID:server-fed7e170-552d-ae1f-9ded-7416c65e577e Address:fed7e170-552d-ae1f-9ded-7416c65e577e} {Suffrage:Voter ID:server-27a96b4c-0fa7-abd1-aa05-27c42b649a74 Address:27a96b4c-0fa7-abd1-aa05-27c42b649a74}]"
2025-07-14T05:40:06.936Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: entering follower state: follower="Node at fed7e170-552d-ae1f-9ded-7416c65e577e [Follower]" leader-address= leader-id=
2025-07-14T05:40:06.936Z [INFO]  server-27a96b4c-0fa7-abd1-aa05-27c42b649a74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98208dfd-10ab-fe00-85ee-ee7b07f88626 Address:98208dfd-10ab-fe00-85ee-ee7b07f88626} {Suffrage:Voter ID:server-fed7e170-552d-ae1f-9ded-7416c65e577e Address:fed7e170-552d-ae1f-9ded-7416c65e577e} {Suffrage:Voter ID:server-27a96b4c-0fa7-abd1-aa05-27c42b649a74 Address:27a96b4c-0fa7-abd1-aa05-27c42b649a74}]"
2025-07-14T05:40:06.936Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:06.936Z [INFO]  server-27a96b4c-0fa7-abd1-aa05-27c42b649a74: entering follower state: follower="Node at 27a96b4c-0fa7-abd1-aa05-27c42b649a74 [Follower]" leader-address= leader-id=
2025-07-14T05:40:07.007Z [WARN]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:07.007Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: entering candidate state: node="Node at fed7e170-552d-ae1f-9ded-7416c65e577e [Candidate]" term=2
2025-07-14T05:40:07.007Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: election won: term=2 tally=2
2025-07-14T05:40:07.007Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: entering leader state: leader="Node at fed7e170-552d-ae1f-9ded-7416c65e577e [Leader]"
2025-07-14T05:40:07.007Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: added peer, starting replication: peer=server-98208dfd-10ab-fe00-85ee-ee7b07f88626
2025-07-14T05:40:07.007Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: added peer, starting replication: peer=server-27a96b4c-0fa7-abd1-aa05-27c42b649a74
2025-07-14T05:40:07.007Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: pipelining replication: peer="{Voter server-98208dfd-10ab-fe00-85ee-ee7b07f88626 98208dfd-10ab-fe00-85ee-ee7b07f88626}"
2025-07-14T05:40:07.008Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: pipelining replication: peer="{Voter server-27a96b4c-0fa7-abd1-aa05-27c42b649a74 27a96b4c-0fa7-abd1-aa05-27c42b649a74}"
2025-07-14T05:40:07.112Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:07.007520819 +0000 UTC m=+106.421981187 (1 nodes), 71.166179ms from start of poll, 71.434262ms from cluster start. Timeout at 2025-07-14 05:40:07.112552006 +0000 UTC m=+106.527012404, 105.031217ms after stability
2025-07-14T05:40:07.112Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:07.218Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:07.112627479 +0000 UTC m=+106.527087857 (2 nodes), 952ns from start of poll, 176.540932ms from cluster start. Timeout at 2025-07-14 05:40:07.218151375 +0000 UTC m=+106.632611743, 105.523886ms after stability
2025-07-14T05:40:07.527Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: aborting pipeline replication: peer="{Voter server-27a96b4c-0fa7-abd1-aa05-27c42b649a74 27a96b4c-0fa7-abd1-aa05-27c42b649a74}"
2025-07-14T05:40:07.527Z [INFO]  server-fed7e170-552d-ae1f-9ded-7416c65e577e: aborting pipeline replication: peer="{Voter server-98208dfd-10ab-fe00-85ee-ee7b07f88626 98208dfd-10ab-fe00-85ee-ee7b07f88626}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.59s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-07-14T05:40:07.528Z [INFO]  server-34b24c28-fdd7-1ca4-e24f-6792c768a79a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34b24c28-fdd7-1ca4-e24f-6792c768a79a Address:34b24c28-fdd7-1ca4-e24f-6792c768a79a} {Suffrage:Voter ID:server-5bf02619-8151-5c59-36d7-3f7afd792c8d Address:5bf02619-8151-5c59-36d7-3f7afd792c8d}]"
2025-07-14T05:40:07.528Z [INFO]  server-34b24c28-fdd7-1ca4-e24f-6792c768a79a: entering follower state: follower="Node at 34b24c28-fdd7-1ca4-e24f-6792c768a79a [Follower]" leader-address= leader-id=
2025-07-14T05:40:07.528Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34b24c28-fdd7-1ca4-e24f-6792c768a79a Address:34b24c28-fdd7-1ca4-e24f-6792c768a79a} {Suffrage:Voter ID:server-5bf02619-8151-5c59-36d7-3f7afd792c8d Address:5bf02619-8151-5c59-36d7-3f7afd792c8d}]"
2025-07-14T05:40:07.528Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:07.528Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: entering follower state: follower="Node at 5bf02619-8151-5c59-36d7-3f7afd792c8d [Follower]" leader-address= leader-id=
2025-07-14T05:40:07.620Z [WARN]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:07.620Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: entering candidate state: node="Node at 5bf02619-8151-5c59-36d7-3f7afd792c8d [Candidate]" term=2
2025-07-14T05:40:07.620Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: election won: term=2 tally=2
2025-07-14T05:40:07.620Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: entering leader state: leader="Node at 5bf02619-8151-5c59-36d7-3f7afd792c8d [Leader]"
2025-07-14T05:40:07.620Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: added peer, starting replication: peer=server-34b24c28-fdd7-1ca4-e24f-6792c768a79a
2025-07-14T05:40:07.620Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: pipelining replication: peer="{Voter server-34b24c28-fdd7-1ca4-e24f-6792c768a79a 34b24c28-fdd7-1ca4-e24f-6792c768a79a}"
2025-07-14T05:40:07.725Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:07.62040153 +0000 UTC m=+107.034861908 (1 nodes), 91.54635ms from start of poll, 91.74021ms from cluster start. Timeout at 2025-07-14 05:40:07.725691044 +0000 UTC m=+107.140151422, 105.289514ms after stability
2025-07-14T05:40:07.725Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:07.831Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:07.72576168 +0000 UTC m=+107.140222058 (1 nodes), 921ns from start of poll, 197.10036ms from cluster start. Timeout at 2025-07-14 05:40:07.831484364 +0000 UTC m=+107.245944742, 105.722684ms after stability
2025-07-14T05:40:07.831Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: updating configuration: command=DemoteVoter server-id=server-34b24c28-fdd7-1ca4-e24f-6792c768a79a server-addr= servers="[{Suffrage:Nonvoter ID:server-34b24c28-fdd7-1ca4-e24f-6792c768a79a Address:34b24c28-fdd7-1ca4-e24f-6792c768a79a} {Suffrage:Voter ID:server-5bf02619-8151-5c59-36d7-3f7afd792c8d Address:5bf02619-8151-5c59-36d7-3f7afd792c8d}]"
2025-07-14T05:40:07.831Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:07.937Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:07.831614579 +0000 UTC m=+107.246074947 (1 nodes), 821ns from start of poll, 302.953249ms from cluster start. Timeout at 2025-07-14 05:40:07.937223232 +0000 UTC m=+107.351683630, 105.608683ms after stability
2025-07-14T05:40:07.937Z [INFO]  server-5bf02619-8151-5c59-36d7-3f7afd792c8d: aborting pipeline replication: peer="{Voter server-34b24c28-fdd7-1ca4-e24f-6792c768a79a 34b24c28-fdd7-1ca4-e24f-6792c768a79a}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.41s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-07-14T05:40:07.938Z [INFO]  server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91 Address:7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91} {Suffrage:Voter ID:server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3 Address:4b69fdff-ecad-8ef1-8feb-f4514dc58dd3}]"
2025-07-14T05:40:07.938Z [INFO]  server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91: entering follower state: follower="Node at 7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91 [Follower]" leader-address= leader-id=
2025-07-14T05:40:07.938Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91 Address:7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91} {Suffrage:Voter ID:server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3 Address:4b69fdff-ecad-8ef1-8feb-f4514dc58dd3}]"
2025-07-14T05:40:07.938Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:07.938Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: entering follower state: follower="Node at 4b69fdff-ecad-8ef1-8feb-f4514dc58dd3 [Follower]" leader-address= leader-id=
2025-07-14T05:40:07.994Z [WARN]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:07.994Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: entering candidate state: node="Node at 4b69fdff-ecad-8ef1-8feb-f4514dc58dd3 [Candidate]" term=2
2025-07-14T05:40:07.994Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: election won: term=2 tally=2
2025-07-14T05:40:07.994Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: entering leader state: leader="Node at 4b69fdff-ecad-8ef1-8feb-f4514dc58dd3 [Leader]"
2025-07-14T05:40:07.994Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: added peer, starting replication: peer=server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91
2025-07-14T05:40:07.994Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: pipelining replication: peer="{Voter server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91 7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91}"
2025-07-14T05:40:08.099Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:07.994577544 +0000 UTC m=+107.409037922 (1 nodes), 56.2866ms from start of poll, 56.476535ms from cluster start. Timeout at 2025-07-14 05:40:08.099762341 +0000 UTC m=+107.514222719, 105.184797ms after stability
2025-07-14T05:40:08.099Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:08.205Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:08.099831184 +0000 UTC m=+107.514291552 (1 nodes), 821ns from start of poll, 161.730165ms from cluster start. Timeout at 2025-07-14 05:40:08.205188951 +0000 UTC m=+107.619649319, 105.357767ms after stability
2025-07-14T05:40:08.216Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91 Address:7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91} {Suffrage:Voter ID:server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3 Address:4b69fdff-ecad-8ef1-8feb-f4514dc58dd3}]"
2025-07-14T05:40:08.216Z [INFO]  server-4b69fdff-ecad-8ef1-8feb-f4514dc58dd3: aborting pipeline replication: peer="{Voter server-7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91 7e5e45d0-5955-d2b7-beb0-cf2be0fc6d91}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-07-14T05:40:08.217Z [INFO]  server-3beb7f53-b8e0-df78-54eb-be451f051335: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3beb7f53-b8e0-df78-54eb-be451f051335 Address:3beb7f53-b8e0-df78-54eb-be451f051335}]"
2025-07-14T05:40:08.217Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:08.217Z [INFO]  server-3beb7f53-b8e0-df78-54eb-be451f051335: entering follower state: follower="Node at 3beb7f53-b8e0-df78-54eb-be451f051335 [Follower]" leader-address= leader-id=
2025-07-14T05:40:08.269Z [WARN]  server-3beb7f53-b8e0-df78-54eb-be451f051335: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:08.269Z [INFO]  server-3beb7f53-b8e0-df78-54eb-be451f051335: entering candidate state: node="Node at 3beb7f53-b8e0-df78-54eb-be451f051335 [Candidate]" term=2
2025-07-14T05:40:08.269Z [INFO]  server-3beb7f53-b8e0-df78-54eb-be451f051335: election won: term=2 tally=1
2025-07-14T05:40:08.269Z [INFO]  server-3beb7f53-b8e0-df78-54eb-be451f051335: entering leader state: leader="Node at 3beb7f53-b8e0-df78-54eb-be451f051335 [Leader]"
2025-07-14T05:40:08.375Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:08.269874109 +0000 UTC m=+107.684334487 (1 nodes), 52.609314ms from start of poll, 52.705268ms from cluster start. Timeout at 2025-07-14 05:40:08.375657113 +0000 UTC m=+107.790117501, 105.783014ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.16s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-07-14T05:40:08.376Z [INFO]  server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257 Address:fa86e593-03f0-1d8c-7f02-3a5ed30a3257} {Suffrage:Voter ID:server-00fe79ae-96eb-cdef-867f-45049fc3fcb4 Address:00fe79ae-96eb-cdef-867f-45049fc3fcb4}]"
2025-07-14T05:40:08.376Z [INFO]  server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257: entering follower state: follower="Node at fa86e593-03f0-1d8c-7f02-3a5ed30a3257 [Follower]" leader-address= leader-id=
2025-07-14T05:40:08.376Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257 Address:fa86e593-03f0-1d8c-7f02-3a5ed30a3257} {Suffrage:Voter ID:server-00fe79ae-96eb-cdef-867f-45049fc3fcb4 Address:00fe79ae-96eb-cdef-867f-45049fc3fcb4}]"
2025-07-14T05:40:08.376Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:08.376Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: entering follower state: follower="Node at 00fe79ae-96eb-cdef-867f-45049fc3fcb4 [Follower]" leader-address= leader-id=
2025-07-14T05:40:08.436Z [WARN]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:08.436Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: entering candidate state: node="Node at 00fe79ae-96eb-cdef-867f-45049fc3fcb4 [Candidate]" term=2
2025-07-14T05:40:08.436Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: election won: term=2 tally=2
2025-07-14T05:40:08.436Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: entering leader state: leader="Node at 00fe79ae-96eb-cdef-867f-45049fc3fcb4 [Leader]"
2025-07-14T05:40:08.436Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: added peer, starting replication: peer=server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257
2025-07-14T05:40:08.436Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: pipelining replication: peer="{Voter server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257 fa86e593-03f0-1d8c-7f02-3a5ed30a3257}"
2025-07-14T05:40:08.542Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:08.436855521 +0000 UTC m=+107.851315899 (1 nodes), 60.277446ms from start of poll, 60.473821ms from cluster start. Timeout at 2025-07-14 05:40:08.5423218 +0000 UTC m=+107.956782198, 105.466299ms after stability
2025-07-14T05:40:08.542Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:08.648Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:08.542409612 +0000 UTC m=+107.956869980 (1 nodes), 982ns from start of poll, 166.027902ms from cluster start. Timeout at 2025-07-14 05:40:08.648069131 +0000 UTC m=+108.062529519, 105.659539ms after stability
2025-07-14T05:40:08.648Z [ERROR] server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-07-14T05:40:08.648Z [WARN]  server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-07-14T05:40:08.648Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: aborting pipeline replication: peer="{Voter server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257 fa86e593-03f0-1d8c-7f02-3a5ed30a3257}"
2025-07-14T05:40:08.656Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: pipelining replication: peer="{Voter server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257 fa86e593-03f0-1d8c-7f02-3a5ed30a3257}"
2025-07-14T05:40:08.656Z [INFO]  server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257 Address:fa86e593-03f0-1d8c-7f02-3a5ed30a3257} {Suffrage:Voter ID:server-00fe79ae-96eb-cdef-867f-45049fc3fcb4 Address:00fe79ae-96eb-cdef-867f-45049fc3fcb4}]"
2025-07-14T05:40:08.656Z [INFO]  server-00fe79ae-96eb-cdef-867f-45049fc3fcb4: aborting pipeline replication: peer="{Voter server-fa86e593-03f0-1d8c-7f02-3a5ed30a3257 fa86e593-03f0-1d8c-7f02-3a5ed30a3257}"
--- PASS: TestRaft_CacheLogWithStoreError (0.28s)
=== RUN   TestRaft_ReloadConfig
2025-07-14T05:40:08.657Z [INFO]  server-dfbea12e-a4c3-e4e9-c457-191654a651e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfbea12e-a4c3-e4e9-c457-191654a651e4 Address:dfbea12e-a4c3-e4e9-c457-191654a651e4}]"
2025-07-14T05:40:08.657Z [INFO]  server-dfbea12e-a4c3-e4e9-c457-191654a651e4: entering follower state: follower="Node at dfbea12e-a4c3-e4e9-c457-191654a651e4 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-07-14T05:40:08.658Z [INFO]  server-1898f76a-1eb3-b5c8-f20b-6c24ade7f63f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1898f76a-1eb3-b5c8-f20b-6c24ade7f63f Address:1898f76a-1eb3-b5c8-f20b-6c24ade7f63f}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-07-14T05:40:08.658Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-07-14T05:40:08.659Z [INFO]  server-9e5f9fdf-9d85-f738-7105-146a71f8c730: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e5f9fdf-9d85-f738-7105-146a71f8c730 Address:9e5f9fdf-9d85-f738-7105-146a71f8c730} {Suffrage:Voter ID:server-2c09627d-5e4c-8575-31a9-07254dcaab99 Address:2c09627d-5e4c-8575-31a9-07254dcaab99} {Suffrage:Voter ID:server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec Address:dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}]"
2025-07-14T05:40:08.659Z [INFO]  server-9e5f9fdf-9d85-f738-7105-146a71f8c730: entering follower state: follower="Node at 9e5f9fdf-9d85-f738-7105-146a71f8c730 [Follower]" leader-address= leader-id=
2025-07-14T05:40:08.659Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e5f9fdf-9d85-f738-7105-146a71f8c730 Address:9e5f9fdf-9d85-f738-7105-146a71f8c730} {Suffrage:Voter ID:server-2c09627d-5e4c-8575-31a9-07254dcaab99 Address:2c09627d-5e4c-8575-31a9-07254dcaab99} {Suffrage:Voter ID:server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec Address:dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}]"
2025-07-14T05:40:08.659Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: entering follower state: follower="Node at 2c09627d-5e4c-8575-31a9-07254dcaab99 [Follower]" leader-address= leader-id=
2025-07-14T05:40:08.659Z [INFO]  server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e5f9fdf-9d85-f738-7105-146a71f8c730 Address:9e5f9fdf-9d85-f738-7105-146a71f8c730} {Suffrage:Voter ID:server-2c09627d-5e4c-8575-31a9-07254dcaab99 Address:2c09627d-5e4c-8575-31a9-07254dcaab99} {Suffrage:Voter ID:server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec Address:dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}]"
2025-07-14T05:40:08.659Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:08.659Z [INFO]  server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec: entering follower state: follower="Node at dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec [Follower]" leader-address= leader-id=
2025-07-14T05:40:08.715Z [WARN]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:08.715Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: entering candidate state: node="Node at 2c09627d-5e4c-8575-31a9-07254dcaab99 [Candidate]" term=2
2025-07-14T05:40:08.715Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: election won: term=2 tally=2
2025-07-14T05:40:08.715Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: entering leader state: leader="Node at 2c09627d-5e4c-8575-31a9-07254dcaab99 [Leader]"
2025-07-14T05:40:08.715Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: added peer, starting replication: peer=server-9e5f9fdf-9d85-f738-7105-146a71f8c730
2025-07-14T05:40:08.715Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: added peer, starting replication: peer=server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec
2025-07-14T05:40:08.715Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: pipelining replication: peer="{Voter server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}"
2025-07-14T05:40:08.715Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: pipelining replication: peer="{Voter server-9e5f9fdf-9d85-f738-7105-146a71f8c730 9e5f9fdf-9d85-f738-7105-146a71f8c730}"
2025-07-14T05:40:08.820Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:08.715296122 +0000 UTC m=+108.129756500 (1 nodes), 55.963546ms from start of poll, 56.245519ms from cluster start. Timeout at 2025-07-14 05:40:08.820304174 +0000 UTC m=+108.234764572, 105.008072ms after stability
2025-07-14T05:40:08.826Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:08.931Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:08.826086267 +0000 UTC m=+108.240546645 (2 nodes), 1.212µs from start of poll, 167.035664ms from cluster start. Timeout at 2025-07-14 05:40:08.931124534 +0000 UTC m=+108.345584912, 105.038267ms after stability
2025-07-14T05:40:08.931Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: updating configuration: command=RemoveServer server-id=server-9e5f9fdf-9d85-f738-7105-146a71f8c730 server-addr= servers="[{Suffrage:Voter ID:server-2c09627d-5e4c-8575-31a9-07254dcaab99 Address:2c09627d-5e4c-8575-31a9-07254dcaab99} {Suffrage:Voter ID:server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec Address:dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}]"
2025-07-14T05:40:08.931Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: removed peer, stopping replication: peer=server-9e5f9fdf-9d85-f738-7105-146a71f8c730 last-index=3
2025-07-14T05:40:08.931Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: aborting pipeline replication: peer="{Voter server-9e5f9fdf-9d85-f738-7105-146a71f8c730 9e5f9fdf-9d85-f738-7105-146a71f8c730}"
2025-07-14T05:40:09.014Z [WARN]  server-9e5f9fdf-9d85-f738-7105-146a71f8c730: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-07-14T05:40:09.039Z [ERROR] server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to pipeline appendEntries: peer="{Voter server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}" error="append pipeline closed"
2025-07-14T05:40:09.039Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: aborting pipeline replication: peer="{Voter server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}"
2025-07-14T05:40:09.041Z [ERROR] server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to heartbeat to: peer=dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec backoff time=10ms error="failed to connect to peer: dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec"
2025-07-14T05:40:09.045Z [ERROR] server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to appendEntries to: peer="{Voter server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}" error="failed to connect to peer: dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec"
2025-07-14T05:40:09.060Z [ERROR] server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to heartbeat to: peer=dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec backoff time=10ms error="failed to connect to peer: dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec"
2025-07-14T05:40:09.064Z [ERROR] server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to appendEntries to: peer="{Voter server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}" error="failed to connect to peer: dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec"
2025-07-14T05:40:09.081Z [ERROR] server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to heartbeat to: peer=dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec backoff time=10ms error="failed to connect to peer: dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec"
2025-07-14T05:40:09.082Z [WARN]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to contact: server-id=server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec time=50.030726ms
2025-07-14T05:40:09.082Z [WARN]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to contact quorum of nodes, stepping down
2025-07-14T05:40:09.082Z [INFO]  server-2c09627d-5e4c-8575-31a9-07254dcaab99: entering follower state: follower="Node at 2c09627d-5e4c-8575-31a9-07254dcaab99 [Follower]" leader-address= leader-id=
2025-07-14T05:40:09.084Z [ERROR] server-2c09627d-5e4c-8575-31a9-07254dcaab99: failed to appendEntries to: peer="{Voter server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec}" error="failed to connect to peer: dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec"
2025-07-14T05:40:09.150Z [WARN]  server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec: heartbeat timeout reached, starting election: last-leader-addr=2c09627d-5e4c-8575-31a9-07254dcaab99 last-leader-id=server-2c09627d-5e4c-8575-31a9-07254dcaab99
2025-07-14T05:40:09.150Z [INFO]  server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec: entering candidate state: node="Node at dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec [Candidate]" term=3
2025-07-14T05:40:09.150Z [ERROR] server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec: failed to make requestVote RPC: target="{Voter server-2c09627d-5e4c-8575-31a9-07254dcaab99 2c09627d-5e4c-8575-31a9-07254dcaab99}" error="failed to connect to peer: 2c09627d-5e4c-8575-31a9-07254dcaab99" term=3
2025-07-14T05:40:09.152Z [WARN]  server-dabfeb1a-af8b-02a3-e0a4-af0ab1e0e7ec: rejecting vote request since node is not in configuration: from=9e5f9fdf-9d85-f738-7105-146a71f8c730
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.49s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-07-14T05:40:09.154Z [INFO]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e9914ae-79d8-016f-5eb5-531305c1297e Address:9e9914ae-79d8-016f-5eb5-531305c1297e} {Suffrage:Voter ID:server-a65c5880-136e-29ab-e043-5ec155994992 Address:a65c5880-136e-29ab-e043-5ec155994992} {Suffrage:Voter ID:server-92111b67-0d79-b3b3-7b9a-1af4ed97165e Address:92111b67-0d79-b3b3-7b9a-1af4ed97165e}]"
2025-07-14T05:40:09.154Z [INFO]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: entering follower state: follower="Node at 9e9914ae-79d8-016f-5eb5-531305c1297e [Follower]" leader-address= leader-id=
2025-07-14T05:40:09.154Z [INFO]  server-a65c5880-136e-29ab-e043-5ec155994992: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e9914ae-79d8-016f-5eb5-531305c1297e Address:9e9914ae-79d8-016f-5eb5-531305c1297e} {Suffrage:Voter ID:server-a65c5880-136e-29ab-e043-5ec155994992 Address:a65c5880-136e-29ab-e043-5ec155994992} {Suffrage:Voter ID:server-92111b67-0d79-b3b3-7b9a-1af4ed97165e Address:92111b67-0d79-b3b3-7b9a-1af4ed97165e}]"
2025-07-14T05:40:09.154Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e9914ae-79d8-016f-5eb5-531305c1297e Address:9e9914ae-79d8-016f-5eb5-531305c1297e} {Suffrage:Voter ID:server-a65c5880-136e-29ab-e043-5ec155994992 Address:a65c5880-136e-29ab-e043-5ec155994992} {Suffrage:Voter ID:server-92111b67-0d79-b3b3-7b9a-1af4ed97165e Address:92111b67-0d79-b3b3-7b9a-1af4ed97165e}]"
2025-07-14T05:40:09.154Z [INFO]  server-a65c5880-136e-29ab-e043-5ec155994992: entering follower state: follower="Node at a65c5880-136e-29ab-e043-5ec155994992 [Follower]" leader-address= leader-id=
2025-07-14T05:40:09.154Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:09.154Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: entering follower state: follower="Node at 92111b67-0d79-b3b3-7b9a-1af4ed97165e [Follower]" leader-address= leader-id=
2025-07-14T05:40:09.237Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:09.237Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: entering candidate state: node="Node at 92111b67-0d79-b3b3-7b9a-1af4ed97165e [Candidate]" term=2
2025-07-14T05:40:09.237Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: election won: term=2 tally=2
2025-07-14T05:40:09.237Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: entering leader state: leader="Node at 92111b67-0d79-b3b3-7b9a-1af4ed97165e [Leader]"
2025-07-14T05:40:09.237Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: added peer, starting replication: peer=server-9e9914ae-79d8-016f-5eb5-531305c1297e
2025-07-14T05:40:09.237Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: added peer, starting replication: peer=server-a65c5880-136e-29ab-e043-5ec155994992
2025-07-14T05:40:09.237Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: pipelining replication: peer="{Voter server-a65c5880-136e-29ab-e043-5ec155994992 a65c5880-136e-29ab-e043-5ec155994992}"
2025-07-14T05:40:09.237Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: pipelining replication: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:09.342Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:09.237194789 +0000 UTC m=+108.651655178 (1 nodes), 82.381292ms from start of poll, 82.711257ms from cluster start. Timeout at 2025-07-14 05:40:09.342373187 +0000 UTC m=+108.756833565, 105.178387ms after stability
2025-07-14T05:40:09.348Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:09.453Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:09.348351404 +0000 UTC m=+108.762811782 (2 nodes), 1.201µs from start of poll, 193.867861ms from cluster start. Timeout at 2025-07-14 05:40:09.453451935 +0000 UTC m=+108.867912303, 105.100521ms after stability
2025-07-14T05:40:09.454Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to pipeline appendEntries: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" error="append pipeline closed"
2025-07-14T05:40:09.454Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: aborting pipeline replication: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:09.458Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to heartbeat to: peer=9e9914ae-79d8-016f-5eb5-531305c1297e backoff time=10ms error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.465Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to appendEntries to: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.479Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to heartbeat to: peer=9e9914ae-79d8-016f-5eb5-531305c1297e backoff time=10ms error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.486Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to appendEntries to: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.497Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to heartbeat to: peer=9e9914ae-79d8-016f-5eb5-531305c1297e backoff time=10ms error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.497Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to contact: server-id=server-9e9914ae-79d8-016f-5eb5-531305c1297e time=50.223145ms
2025-07-14T05:40:09.505Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to appendEntries to: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.519Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to heartbeat to: peer=9e9914ae-79d8-016f-5eb5-531305c1297e backoff time=20ms error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.534Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to appendEntries to: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.540Z [WARN]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: heartbeat timeout reached, starting election: last-leader-addr=92111b67-0d79-b3b3-7b9a-1af4ed97165e last-leader-id=server-92111b67-0d79-b3b3-7b9a-1af4ed97165e
2025-07-14T05:40:09.540Z [INFO]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: entering candidate state: node="Node at 9e9914ae-79d8-016f-5eb5-531305c1297e [Candidate]" term=3
2025-07-14T05:40:09.540Z [ERROR] server-9e9914ae-79d8-016f-5eb5-531305c1297e: failed to make requestVote RPC: target="{Voter server-92111b67-0d79-b3b3-7b9a-1af4ed97165e 92111b67-0d79-b3b3-7b9a-1af4ed97165e}" error="failed to connect to peer: 92111b67-0d79-b3b3-7b9a-1af4ed97165e" term=3
2025-07-14T05:40:09.540Z [ERROR] server-9e9914ae-79d8-016f-5eb5-531305c1297e: failed to make requestVote RPC: target="{Voter server-a65c5880-136e-29ab-e043-5ec155994992 a65c5880-136e-29ab-e043-5ec155994992}" error="failed to connect to peer: a65c5880-136e-29ab-e043-5ec155994992" term=3
2025-07-14T05:40:09.545Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to heartbeat to: peer=9e9914ae-79d8-016f-5eb5-531305c1297e backoff time=25ms error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.547Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to contact: server-id=server-9e9914ae-79d8-016f-5eb5-531305c1297e time=99.978187ms
2025-07-14T05:40:09.558Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: updating configuration: command=RemoveServer server-id=server-9e9914ae-79d8-016f-5eb5-531305c1297e server-addr= servers="[{Suffrage:Voter ID:server-a65c5880-136e-29ab-e043-5ec155994992 Address:a65c5880-136e-29ab-e043-5ec155994992} {Suffrage:Voter ID:server-92111b67-0d79-b3b3-7b9a-1af4ed97165e Address:92111b67-0d79-b3b3-7b9a-1af4ed97165e}]"
2025-07-14T05:40:09.558Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: removed peer, stopping replication: peer=server-9e9914ae-79d8-016f-5eb5-531305c1297e last-index=3
2025-07-14T05:40:09.558Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: updating configuration: command=AddNonvoter server-id=server-9e9914ae-79d8-016f-5eb5-531305c1297e server-addr=9e9914ae-79d8-016f-5eb5-531305c1297e servers="[{Suffrage:Voter ID:server-a65c5880-136e-29ab-e043-5ec155994992 Address:a65c5880-136e-29ab-e043-5ec155994992} {Suffrage:Voter ID:server-92111b67-0d79-b3b3-7b9a-1af4ed97165e Address:92111b67-0d79-b3b3-7b9a-1af4ed97165e} {Suffrage:Nonvoter ID:server-9e9914ae-79d8-016f-5eb5-531305c1297e Address:9e9914ae-79d8-016f-5eb5-531305c1297e}]"
2025-07-14T05:40:09.558Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: added peer, starting replication: peer=server-9e9914ae-79d8-016f-5eb5-531305c1297e
2025-07-14T05:40:09.558Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: failed to appendEntries to: peer="{Nonvoter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" error="failed to connect to peer: 9e9914ae-79d8-016f-5eb5-531305c1297e"
2025-07-14T05:40:09.579Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: peer has newer term, stopping replication: peer="{Nonvoter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:09.579Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: entering follower state: follower="Node at 92111b67-0d79-b3b3-7b9a-1af4ed97165e [Follower]" leader-address= leader-id=
2025-07-14T05:40:09.579Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: aborting pipeline replication: peer="{Voter server-a65c5880-136e-29ab-e043-5ec155994992 a65c5880-136e-29ab-e043-5ec155994992}"
2025-07-14T05:40:09.582Z [ERROR] server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: peer has newer term, stopping replication: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:09.599Z [WARN]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: Election timeout reached, restarting election
2025-07-14T05:40:09.599Z [INFO]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: entering candidate state: node="Node at 9e9914ae-79d8-016f-5eb5-531305c1297e [Candidate]" term=23
2025-07-14T05:40:09.599Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: rejecting vote request since node is not a voter: from=9e9914ae-79d8-016f-5eb5-531305c1297e
2025-07-14T05:40:09.599Z [WARN]  server-a65c5880-136e-29ab-e043-5ec155994992: rejecting vote request since we have a leader: from=9e9914ae-79d8-016f-5eb5-531305c1297e leader=92111b67-0d79-b3b3-7b9a-1af4ed97165e leader-id=server-92111b67-0d79-b3b3-7b9a-1af4ed97165e
2025-07-14T05:40:09.655Z [WARN]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: Election timeout reached, restarting election
2025-07-14T05:40:09.655Z [INFO]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: entering candidate state: node="Node at 9e9914ae-79d8-016f-5eb5-531305c1297e [Candidate]" term=24
2025-07-14T05:40:09.655Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: rejecting vote request since node is not a voter: from=9e9914ae-79d8-016f-5eb5-531305c1297e
2025-07-14T05:40:09.655Z [WARN]  server-a65c5880-136e-29ab-e043-5ec155994992: rejecting vote request since we have a leader: from=9e9914ae-79d8-016f-5eb5-531305c1297e leader=92111b67-0d79-b3b3-7b9a-1af4ed97165e leader-id=server-92111b67-0d79-b3b3-7b9a-1af4ed97165e
2025-07-14T05:40:09.664Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:09.667Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:09.667Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: entering candidate state: node="Node at 92111b67-0d79-b3b3-7b9a-1af4ed97165e [Candidate]" term=25
2025-07-14T05:40:09.667Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: election won: term=25 tally=2
2025-07-14T05:40:09.667Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: entering leader state: leader="Node at 92111b67-0d79-b3b3-7b9a-1af4ed97165e [Leader]"
2025-07-14T05:40:09.667Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: added peer, starting replication: peer=server-a65c5880-136e-29ab-e043-5ec155994992
2025-07-14T05:40:09.667Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: added peer, starting replication: peer=server-9e9914ae-79d8-016f-5eb5-531305c1297e
2025-07-14T05:40:09.667Z [WARN]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-07-14T05:40:09.667Z [INFO]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: entering follower state: follower="Node at 9e9914ae-79d8-016f-5eb5-531305c1297e [Follower]" leader-address=92111b67-0d79-b3b3-7b9a-1af4ed97165e leader-id=server-92111b67-0d79-b3b3-7b9a-1af4ed97165e
2025-07-14T05:40:09.667Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: appendEntries rejected, sending older logs: peer="{Nonvoter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" next=3
2025-07-14T05:40:09.667Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: pipelining replication: peer="{Voter server-a65c5880-136e-29ab-e043-5ec155994992 a65c5880-136e-29ab-e043-5ec155994992}"
2025-07-14T05:40:09.667Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: pipelining replication: peer="{Nonvoter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:09.773Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:09.667484875 +0000 UTC m=+109.081945253 (1 nodes), 2.585018ms from start of poll, 513.001332ms from cluster start. Timeout at 2025-07-14 05:40:09.773067368 +0000 UTC m=+109.187527746, 105.582493ms after stability
2025-07-14T05:40:09.793Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: updating configuration: command=RemoveServer server-id=server-9e9914ae-79d8-016f-5eb5-531305c1297e server-addr= servers="[{Suffrage:Voter ID:server-a65c5880-136e-29ab-e043-5ec155994992 Address:a65c5880-136e-29ab-e043-5ec155994992} {Suffrage:Voter ID:server-92111b67-0d79-b3b3-7b9a-1af4ed97165e Address:92111b67-0d79-b3b3-7b9a-1af4ed97165e}]"
2025-07-14T05:40:09.793Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: removed peer, stopping replication: peer=server-9e9914ae-79d8-016f-5eb5-531305c1297e last-index=106
2025-07-14T05:40:09.793Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: aborting pipeline replication: peer="{Nonvoter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:09.793Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: updating configuration: command=AddVoter server-id=server-9e9914ae-79d8-016f-5eb5-531305c1297e server-addr=9e9914ae-79d8-016f-5eb5-531305c1297e servers="[{Suffrage:Voter ID:server-a65c5880-136e-29ab-e043-5ec155994992 Address:a65c5880-136e-29ab-e043-5ec155994992} {Suffrage:Voter ID:server-92111b67-0d79-b3b3-7b9a-1af4ed97165e Address:92111b67-0d79-b3b3-7b9a-1af4ed97165e} {Suffrage:Voter ID:server-9e9914ae-79d8-016f-5eb5-531305c1297e Address:9e9914ae-79d8-016f-5eb5-531305c1297e}]"
2025-07-14T05:40:09.793Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: added peer, starting replication: peer=server-9e9914ae-79d8-016f-5eb5-531305c1297e
2025-07-14T05:40:09.793Z [WARN]  server-9e9914ae-79d8-016f-5eb5-531305c1297e: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2025-07-14T05:40:09.793Z [WARN]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: appendEntries rejected, sending older logs: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}" next=107
2025-07-14T05:40:09.794Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: pipelining replication: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:10.864Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:10.970Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:10.86472186 +0000 UTC m=+110.279182228 (1 nodes), 771ns from start of poll, 1.710238307s from cluster start. Timeout at 2025-07-14 05:40:10.97008868 +0000 UTC m=+110.384549068, 105.36684ms after stability
2025-07-14T05:40:10.970Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: aborting pipeline replication: peer="{Voter server-9e9914ae-79d8-016f-5eb5-531305c1297e 9e9914ae-79d8-016f-5eb5-531305c1297e}"
2025-07-14T05:40:10.970Z [INFO]  server-92111b67-0d79-b3b3-7b9a-1af4ed97165e: aborting pipeline replication: peer="{Voter server-a65c5880-136e-29ab-e043-5ec155994992 a65c5880-136e-29ab-e043-5ec155994992}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.82s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-07-14T05:40:10.971Z [INFO]  server-59612f4e-08a5-e966-e960-24c4b8022c68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59612f4e-08a5-e966-e960-24c4b8022c68 Address:59612f4e-08a5-e966-e960-24c4b8022c68} {Suffrage:Voter ID:server-88babea9-0023-91c8-fbfb-7bcee6212778 Address:88babea9-0023-91c8-fbfb-7bcee6212778} {Suffrage:Voter ID:server-2a6d0069-a690-1d20-668e-1f77294f2833 Address:2a6d0069-a690-1d20-668e-1f77294f2833}]"
2025-07-14T05:40:10.971Z [INFO]  server-59612f4e-08a5-e966-e960-24c4b8022c68: entering follower state: follower="Node at 59612f4e-08a5-e966-e960-24c4b8022c68 [Follower]" leader-address= leader-id=
2025-07-14T05:40:10.971Z [INFO]  server-88babea9-0023-91c8-fbfb-7bcee6212778: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59612f4e-08a5-e966-e960-24c4b8022c68 Address:59612f4e-08a5-e966-e960-24c4b8022c68} {Suffrage:Voter ID:server-88babea9-0023-91c8-fbfb-7bcee6212778 Address:88babea9-0023-91c8-fbfb-7bcee6212778} {Suffrage:Voter ID:server-2a6d0069-a690-1d20-668e-1f77294f2833 Address:2a6d0069-a690-1d20-668e-1f77294f2833}]"
2025-07-14T05:40:10.971Z [INFO]  server-88babea9-0023-91c8-fbfb-7bcee6212778: entering follower state: follower="Node at 88babea9-0023-91c8-fbfb-7bcee6212778 [Follower]" leader-address= leader-id=
2025-07-14T05:40:10.971Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59612f4e-08a5-e966-e960-24c4b8022c68 Address:59612f4e-08a5-e966-e960-24c4b8022c68} {Suffrage:Voter ID:server-88babea9-0023-91c8-fbfb-7bcee6212778 Address:88babea9-0023-91c8-fbfb-7bcee6212778} {Suffrage:Voter ID:server-2a6d0069-a690-1d20-668e-1f77294f2833 Address:2a6d0069-a690-1d20-668e-1f77294f2833}]"
2025-07-14T05:40:10.971Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:10.971Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: entering follower state: follower="Node at 2a6d0069-a690-1d20-668e-1f77294f2833 [Follower]" leader-address= leader-id=
2025-07-14T05:40:11.089Z [WARN]  server-2a6d0069-a690-1d20-668e-1f77294f2833: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:11.089Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: entering candidate state: node="Node at 2a6d0069-a690-1d20-668e-1f77294f2833 [Candidate]" term=2
2025-07-14T05:40:11.090Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: election won: term=2 tally=2
2025-07-14T05:40:11.090Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: entering leader state: leader="Node at 2a6d0069-a690-1d20-668e-1f77294f2833 [Leader]"
2025-07-14T05:40:11.090Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: added peer, starting replication: peer=server-59612f4e-08a5-e966-e960-24c4b8022c68
2025-07-14T05:40:11.090Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: added peer, starting replication: peer=server-88babea9-0023-91c8-fbfb-7bcee6212778
2025-07-14T05:40:11.090Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: pipelining replication: peer="{Voter server-59612f4e-08a5-e966-e960-24c4b8022c68 59612f4e-08a5-e966-e960-24c4b8022c68}"
2025-07-14T05:40:11.090Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: pipelining replication: peer="{Voter server-88babea9-0023-91c8-fbfb-7bcee6212778 88babea9-0023-91c8-fbfb-7bcee6212778}"
2025-07-14T05:40:11.296Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:11.09050233 +0000 UTC m=+110.504962719 (1 nodes), 118.691247ms from start of poll, 119.18607ms from cluster start. Timeout at 2025-07-14 05:40:11.295989852 +0000 UTC m=+110.710450250, 205.487531ms after stability
2025-07-14T05:40:11.296Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:11.501Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:11.296090393 +0000 UTC m=+110.710550771 (1 nodes), 1.172µs from start of poll, 324.774122ms from cluster start. Timeout at 2025-07-14 05:40:11.501860528 +0000 UTC m=+110.916320926, 205.770155ms after stability
2025-07-14T05:40:11.508Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:11.713Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:11.508069121 +0000 UTC m=+110.922529499 (2 nodes), 1.092µs from start of poll, 536.75285ms from cluster start. Timeout at 2025-07-14 05:40:11.713073918 +0000 UTC m=+111.127534286, 205.004787ms after stability
2025-07-14T05:40:11.713Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:11.918Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:11.713127408 +0000 UTC m=+111.127587786 (1 nodes), 591ns from start of poll, 741.811137ms from cluster start. Timeout at 2025-07-14 05:40:11.918479015 +0000 UTC m=+111.332939383, 205.351597ms after stability
    raft_test.go:2961: [INFO] restarting Node at 59612f4e-08a5-e966-e960-24c4b8022c68 [Follower]
2025-07-14T05:40:11.918Z [INFO]  server-59612f4e-08a5-e966-e960-24c4b8022c68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59612f4e-08a5-e966-e960-24c4b8022c68 Address:59612f4e-08a5-e966-e960-24c4b8022c68} {Suffrage:Voter ID:server-88babea9-0023-91c8-fbfb-7bcee6212778 Address:88babea9-0023-91c8-fbfb-7bcee6212778} {Suffrage:Voter ID:server-2a6d0069-a690-1d20-668e-1f77294f2833 Address:2a6d0069-a690-1d20-668e-1f77294f2833}]"
2025-07-14T05:40:11.918Z [INFO]  server-59612f4e-08a5-e966-e960-24c4b8022c68: entering follower state: follower="Node at 59612f4e-08a5-e966-e960-24c4b8022c68 [Follower]" leader-address= leader-id=
2025-07-14T05:40:12.169Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: aborting pipeline replication: peer="{Voter server-88babea9-0023-91c8-fbfb-7bcee6212778 88babea9-0023-91c8-fbfb-7bcee6212778}"
2025-07-14T05:40:12.558Z [ERROR] server-2a6d0069-a690-1d20-668e-1f77294f2833: failed to pipeline appendEntries: peer="{Voter server-59612f4e-08a5-e966-e960-24c4b8022c68 59612f4e-08a5-e966-e960-24c4b8022c68}" error="command enqueue timeout"
2025-07-14T05:40:12.558Z [INFO]  server-2a6d0069-a690-1d20-668e-1f77294f2833: aborting pipeline replication: peer="{Voter server-59612f4e-08a5-e966-e960-24c4b8022c68 59612f4e-08a5-e966-e960-24c4b8022c68}"
2025-07-14T05:40:12.677Z [ERROR] server-2a6d0069-a690-1d20-668e-1f77294f2833: failed to heartbeat to: peer=59612f4e-08a5-e966-e960-24c4b8022c68 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.71s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-07-14T05:40:12.678Z [INFO]  server-1140641e-c34b-7821-86c9-31fd298777de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1140641e-c34b-7821-86c9-31fd298777de Address:1140641e-c34b-7821-86c9-31fd298777de} {Suffrage:Voter ID:server-0dccb3b2-fabb-326d-043c-1c45b7b96039 Address:0dccb3b2-fabb-326d-043c-1c45b7b96039} {Suffrage:Voter ID:server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 Address:c36cf6c2-c1eb-99a5-52aa-3c63f3292af6}]"
2025-07-14T05:40:12.678Z [INFO]  server-1140641e-c34b-7821-86c9-31fd298777de: entering follower state: follower="Node at 1140641e-c34b-7821-86c9-31fd298777de [Follower]" leader-address= leader-id=
2025-07-14T05:40:12.678Z [INFO]  server-0dccb3b2-fabb-326d-043c-1c45b7b96039: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1140641e-c34b-7821-86c9-31fd298777de Address:1140641e-c34b-7821-86c9-31fd298777de} {Suffrage:Voter ID:server-0dccb3b2-fabb-326d-043c-1c45b7b96039 Address:0dccb3b2-fabb-326d-043c-1c45b7b96039} {Suffrage:Voter ID:server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 Address:c36cf6c2-c1eb-99a5-52aa-3c63f3292af6}]"
2025-07-14T05:40:12.678Z [INFO]  server-0dccb3b2-fabb-326d-043c-1c45b7b96039: entering follower state: follower="Node at 0dccb3b2-fabb-326d-043c-1c45b7b96039 [Follower]" leader-address= leader-id=
2025-07-14T05:40:12.678Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1140641e-c34b-7821-86c9-31fd298777de Address:1140641e-c34b-7821-86c9-31fd298777de} {Suffrage:Voter ID:server-0dccb3b2-fabb-326d-043c-1c45b7b96039 Address:0dccb3b2-fabb-326d-043c-1c45b7b96039} {Suffrage:Voter ID:server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 Address:c36cf6c2-c1eb-99a5-52aa-3c63f3292af6}]"
2025-07-14T05:40:12.678Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:12.678Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: entering follower state: follower="Node at c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 [Follower]" leader-address= leader-id=
2025-07-14T05:40:12.755Z [WARN]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-07-14T05:40:12.755Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: entering candidate state: node="Node at c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 [Candidate]" term=2
2025-07-14T05:40:12.755Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: election won: term=2 tally=2
2025-07-14T05:40:12.755Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: entering leader state: leader="Node at c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 [Leader]"
2025-07-14T05:40:12.755Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: added peer, starting replication: peer=server-1140641e-c34b-7821-86c9-31fd298777de
2025-07-14T05:40:12.755Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: added peer, starting replication: peer=server-0dccb3b2-fabb-326d-043c-1c45b7b96039
2025-07-14T05:40:12.756Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: pipelining replication: peer="{Voter server-1140641e-c34b-7821-86c9-31fd298777de 1140641e-c34b-7821-86c9-31fd298777de}"
2025-07-14T05:40:12.756Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: pipelining replication: peer="{Voter server-0dccb3b2-fabb-326d-043c-1c45b7b96039 0dccb3b2-fabb-326d-043c-1c45b7b96039}"
2025-07-14T05:40:12.861Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:12.755969922 +0000 UTC m=+112.170430300 (1 nodes), 77.565408ms from start of poll, 77.82654ms from cluster start. Timeout at 2025-07-14 05:40:12.861334498 +0000 UTC m=+112.275794867, 105.364567ms after stability
2025-07-14T05:40:12.861Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-07-14T05:40:12.967Z [INFO]  cluster: stable state for Leader reached at 2025-07-14 05:40:12.861406116 +0000 UTC m=+112.275866494 (1 nodes), 901ns from start of poll, 183.262734ms from cluster start. Timeout at 2025-07-14 05:40:12.96704868 +0000 UTC m=+112.381509048, 105.642554ms after stability
2025-07-14T05:40:12.972Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-07-14T05:40:13.078Z [INFO]  cluster: stable state for Follower reached at 2025-07-14 05:40:12.972861699 +0000 UTC m=+112.387322067 (2 nodes), 711ns from start of poll, 294.718307ms from cluster start. Timeout at 2025-07-14 05:40:13.078173367 +0000 UTC m=+112.492633745, 105.311678ms after stability
2025-07-14T05:40:13.079Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to pipeline appendEntries: peer="{Voter server-0dccb3b2-fabb-326d-043c-1c45b7b96039 0dccb3b2-fabb-326d-043c-1c45b7b96039}" error="append pipeline closed"
2025-07-14T05:40:13.079Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: aborting pipeline replication: peer="{Voter server-0dccb3b2-fabb-326d-043c-1c45b7b96039 0dccb3b2-fabb-326d-043c-1c45b7b96039}"
2025-07-14T05:40:13.081Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=1140641e-c34b-7821-86c9-31fd298777de backoff time=10ms error="failed to connect to peer: 1140641e-c34b-7821-86c9-31fd298777de"
2025-07-14T05:40:13.084Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to pipeline appendEntries: peer="{Voter server-1140641e-c34b-7821-86c9-31fd298777de 1140641e-c34b-7821-86c9-31fd298777de}" error="append pipeline closed"
2025-07-14T05:40:13.084Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: aborting pipeline replication: peer="{Voter server-1140641e-c34b-7821-86c9-31fd298777de 1140641e-c34b-7821-86c9-31fd298777de}"
2025-07-14T05:40:13.085Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=0dccb3b2-fabb-326d-043c-1c45b7b96039 backoff time=10ms error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.086Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to appendEntries to: peer="{Voter server-0dccb3b2-fabb-326d-043c-1c45b7b96039 0dccb3b2-fabb-326d-043c-1c45b7b96039}" error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.090Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to appendEntries to: peer="{Voter server-1140641e-c34b-7821-86c9-31fd298777de 1140641e-c34b-7821-86c9-31fd298777de}" error="failed to connect to peer: 1140641e-c34b-7821-86c9-31fd298777de"
2025-07-14T05:40:13.098Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=1140641e-c34b-7821-86c9-31fd298777de backoff time=10ms error="failed to connect to peer: 1140641e-c34b-7821-86c9-31fd298777de"
2025-07-14T05:40:13.102Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=0dccb3b2-fabb-326d-043c-1c45b7b96039 backoff time=10ms error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.102Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to appendEntries to: peer="{Voter server-0dccb3b2-fabb-326d-043c-1c45b7b96039 0dccb3b2-fabb-326d-043c-1c45b7b96039}" error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.111Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to appendEntries to: peer="{Voter server-1140641e-c34b-7821-86c9-31fd298777de 1140641e-c34b-7821-86c9-31fd298777de}" error="failed to connect to peer: 1140641e-c34b-7821-86c9-31fd298777de"
2025-07-14T05:40:13.114Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=1140641e-c34b-7821-86c9-31fd298777de backoff time=10ms error="failed to connect to peer: 1140641e-c34b-7821-86c9-31fd298777de"
2025-07-14T05:40:13.120Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to appendEntries to: peer="{Voter server-0dccb3b2-fabb-326d-043c-1c45b7b96039 0dccb3b2-fabb-326d-043c-1c45b7b96039}" error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.121Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=0dccb3b2-fabb-326d-043c-1c45b7b96039 backoff time=10ms error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.125Z [WARN]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to contact: server-id=server-1140641e-c34b-7821-86c9-31fd298777de time=50.67992ms
2025-07-14T05:40:13.131Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=1140641e-c34b-7821-86c9-31fd298777de backoff time=20ms error="failed to connect to peer: 1140641e-c34b-7821-86c9-31fd298777de"
2025-07-14T05:40:13.132Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to appendEntries to: peer="{Voter server-1140641e-c34b-7821-86c9-31fd298777de 1140641e-c34b-7821-86c9-31fd298777de}" error="failed to connect to peer: 1140641e-c34b-7821-86c9-31fd298777de"
2025-07-14T05:40:13.135Z [WARN]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to contact: server-id=server-1140641e-c34b-7821-86c9-31fd298777de time=60.849965ms
2025-07-14T05:40:13.135Z [WARN]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to contact: server-id=server-0dccb3b2-fabb-326d-043c-1c45b7b96039 time=59.661072ms
2025-07-14T05:40:13.135Z [WARN]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to contact quorum of nodes, stepping down
2025-07-14T05:40:13.135Z [INFO]  server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: entering follower state: follower="Node at c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 [Follower]" leader-address= leader-id=
2025-07-14T05:40:13.142Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to heartbeat to: peer=0dccb3b2-fabb-326d-043c-1c45b7b96039 backoff time=20ms error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.147Z [ERROR] server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6: failed to appendEntries to: peer="{Voter server-0dccb3b2-fabb-326d-043c-1c45b7b96039 0dccb3b2-fabb-326d-043c-1c45b7b96039}" error="failed to connect to peer: 0dccb3b2-fabb-326d-043c-1c45b7b96039"
2025-07-14T05:40:13.176Z [WARN]  server-1140641e-c34b-7821-86c9-31fd298777de: heartbeat timeout reached, starting election: last-leader-addr=c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 last-leader-id=server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6
2025-07-14T05:40:13.176Z [INFO]  server-1140641e-c34b-7821-86c9-31fd298777de: entering candidate state: node="Node at 1140641e-c34b-7821-86c9-31fd298777de [Candidate]" term=3
2025-07-14T05:40:13.176Z [ERROR] server-1140641e-c34b-7821-86c9-31fd298777de: failed to make requestVote RPC: target="{Voter server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 c36cf6c2-c1eb-99a5-52aa-3c63f3292af6}" error="failed to connect to peer: c36cf6c2-c1eb-99a5-52aa-3c63f3292af6" term=3
2025-07-14T05:40:13.176Z [WARN]  server-0dccb3b2-fabb-326d-043c-1c45b7b96039: rejecting vote request since we have a leader: from=1140641e-c34b-7821-86c9-31fd298777de leader=c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 leader-id=server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6
2025-07-14T05:40:13.183Z [WARN]  server-0dccb3b2-fabb-326d-043c-1c45b7b96039: heartbeat timeout reached, starting election: last-leader-addr=c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 last-leader-id=server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6
2025-07-14T05:40:13.183Z [INFO]  server-0dccb3b2-fabb-326d-043c-1c45b7b96039: entering candidate state: node="Node at 0dccb3b2-fabb-326d-043c-1c45b7b96039 [Candidate]" term=3
2025-07-14T05:40:13.183Z [INFO]  server-1140641e-c34b-7821-86c9-31fd298777de: entering follower state: follower="Node at 1140641e-c34b-7821-86c9-31fd298777de [Follower]" leader-address= leader-id=
2025-07-14T05:40:13.183Z [ERROR] server-0dccb3b2-fabb-326d-043c-1c45b7b96039: failed to make requestVote RPC: target="{Voter server-c36cf6c2-c1eb-99a5-52aa-3c63f3292af6 c36cf6c2-c1eb-99a5-52aa-3c63f3292af6}" error="failed to connect to peer: c36cf6c2-c1eb-99a5-52aa-3c63f3292af6" term=3
2025-07-14T05:40:13.683Z [ERROR] server-0dccb3b2-fabb-326d-043c-1c45b7b96039: failed to make requestVote RPC: target="{Voter server-1140641e-c34b-7821-86c9-31fd298777de 1140641e-c34b-7821-86c9-31fd298777de}" error="command timed out" term=3
--- PASS: TestRaft_VoteWithNoIDNoAddr (1.01s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-07-14T05:40:13.684Z [INFO]  127.0.0.1:34469 :: starting node: addr=127.0.0.1:34469
2025-07-14T05:40:13.684Z [INFO]  127.0.0.1:34469 :: initial configuration: index=0 servers=[]
2025-07-14T05:40:13.684Z [INFO]  127.0.0.1:34469 :: entering follower state: follower="Node at 127.0.0.1:34469 [Follower]" leader-address= leader-id=
2025-07-14T05:40:13.743Z [WARN]  127.0.0.1:34469 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-07-14T05:40:13.836Z [INFO]  127.0.0.1:41735 :: starting node: addr=127.0.0.1:41735
2025-07-14T05:40:13.836Z [INFO]  127.0.0.1:41735 :: initial configuration: index=0 servers=[]
2025-07-14T05:40:13.836Z [INFO]  127.0.0.1:41735 :: entering follower state: follower="Node at 127.0.0.1:41735 [Follower]" leader-address= leader-id=
2025-07-14T05:40:13.920Z [WARN]  127.0.0.1:41735 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-07-14T05:40:13.986Z [INFO]  127.0.0.1:34885 :: starting node: addr=127.0.0.1:34885
2025-07-14T05:40:13.986Z [INFO]  127.0.0.1:34885 :: initial configuration: index=0 servers=[]
2025-07-14T05:40:13.986Z [INFO]  127.0.0.1:34885 :: entering follower state: follower="Node at 127.0.0.1:34885 [Follower]" leader-address= leader-id=
2025-07-14T05:40:14.042Z [WARN]  127.0.0.1:34885 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-07-14T05:40:14.137Z [INFO]  127.0.0.1:43737 :: starting node: addr=127.0.0.1:43737
2025-07-14T05:40:14.137Z [INFO]  127.0.0.1:43737 :: initial configuration: index=0 servers=[]
2025-07-14T05:40:14.137Z [INFO]  127.0.0.1:43737 :: entering follower state: follower="Node at 127.0.0.1:43737 [Follower]" leader-address= leader-id=
2025-07-14T05:40:14.137Z [WARN]  127.0.0.1:43737 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	113.715s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/B01_cleanup starting
I: user script /srv/workspace/pbuilder/20195/tmp/hooks/B01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/20195 and its subdirectories
I: Current time: Mon Jul 14 19:40:18 +14 2025
I: pbuilder-time-stamp: 1752471618