Thu Oct 30 18:16:04 UTC 2025  I: starting to build golang-github-hashicorp-raft/forky/amd64 on jenkins on '2025-10-30 18:15'
Thu Oct 30 18:16:04 UTC 2025  I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/amd64_3/73301/console.log
Thu Oct 30 18:16:04 UTC 2025  I: Downloading source for forky/golang-github-hashicorp-raft=1.5.0-1
--2025-10-30 18:16:04--  http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc
Connecting to 46.16.76.132:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2480 (2.4K) [text/prs.lines.tag]
Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’

     0K ..                                                    100%  308M=0s

2025-10-30 18:16:04 (308 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480]

Thu Oct 30 18:16:04 UTC 2025  I: golang-github-hashicorp-raft_1.5.0-1.dsc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 3.0 (quilt)
Source: golang-github-hashicorp-raft
Binary: golang-github-hashicorp-raft-dev
Architecture: all
Version: 1.5.0-1
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Uploaders: Tim Potter <tpot@hpe.com>, Dmitry Smirnov <onlyjob@debian.org>,
Homepage: https://github.com/hashicorp/raft
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft
Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git
Testsuite: autopkgtest-pkg-go
Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
Package-List:
 golang-github-hashicorp-raft-dev deb golang optional arch=all
Checksums-Sha1:
 cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Checksums-Sha256:
 f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Files:
 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Go-Import-Path: github.com/hashicorp/raft

-----BEGIN PGP SIGNATURE-----

iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp
Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p
nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj
sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz
oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU
Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs
T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u
xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg
+DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF
gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb
LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys
VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z
14FQMVvbK31veGXJjA==
=okTI
-----END PGP SIGNATURE-----
Thu Oct 30 18:16:04 UTC 2025  I: Checking whether the package is not for us
Thu Oct 30 18:16:04 UTC 2025  I: Starting 1st build on remote node ionos1-amd64.debian.net.
Thu Oct 30 18:16:04 UTC 2025  I: Preparing to do remote build '1' on ionos1-amd64.debian.net.
Thu Oct 30 18:19:33 UTC 2025  I: Deleting $TMPDIR on ionos1-amd64.debian.net.
I: pbuilder: network access will be disabled during build
I: Current time: Thu Oct 30 06:16:05 -12 2025
I: pbuilder-time-stamp: 1761848165
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/forky-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/228153/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='amd64'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=40 '
  DISTRIBUTION='forky'
  HOME='/root'
  HOST_ARCH='amd64'
  IFS=' 	
  '
  INVOCATION_ID='e730915eb9b04aec9d96ad94fe1f8ca9'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='228153'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.29Yc7fJR/pbuilderrc_98bK --distribution forky --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/forky-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.29Yc7fJR/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='110'
  SUDO_HOME='/var/lib/jenkins'
  SUDO_UID='105'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://46.16.76.132:3128'
I: uname -a
  Linux ionos1-amd64 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Aug 10 12:30 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/228153/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: amd64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19869 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-v2-dev{a} golang-github-aws-smithy-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} golang-uber-goleak-dev{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2-16{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 108 newly installed, 0 to remove and 0 not upgraded.
Need to get 126 MB of archives. After unpacking 858 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian forky/main amd64 sensible-utils all 0.0.26 [27.0 kB]
Get: 2 http://deb.debian.org/debian forky/main amd64 libmagic-mgc amd64 1:5.46-5 [338 kB]
Get: 3 http://deb.debian.org/debian forky/main amd64 libmagic1t64 amd64 1:5.46-5 [109 kB]
Get: 4 http://deb.debian.org/debian forky/main amd64 file amd64 1:5.46-5 [43.6 kB]
Get: 5 http://deb.debian.org/debian forky/main amd64 gettext-base amd64 0.23.1-2+b1 [244 kB]
Get: 6 http://deb.debian.org/debian forky/main amd64 libuchardet0 amd64 0.0.8-2 [68.5 kB]
Get: 7 http://deb.debian.org/debian forky/main amd64 groff-base amd64 1.23.0-9 [1187 kB]
Get: 8 http://deb.debian.org/debian forky/main amd64 bsdextrautils amd64 2.41.2-4 [98.5 kB]
Get: 9 http://deb.debian.org/debian forky/main amd64 libpipeline1 amd64 1.5.8-1 [42.0 kB]
Get: 10 http://deb.debian.org/debian forky/main amd64 man-db amd64 2.13.1-1 [1469 kB]
Get: 11 http://deb.debian.org/debian forky/main amd64 m4 amd64 1.4.20-2 [325 kB]
Get: 12 http://deb.debian.org/debian forky/main amd64 autoconf all 2.72-3.1 [494 kB]
Get: 13 http://deb.debian.org/debian forky/main amd64 autotools-dev all 20240727.1 [60.2 kB]
Get: 14 http://deb.debian.org/debian forky/main amd64 automake all 1:1.18.1-2 [877 kB]
Get: 15 http://deb.debian.org/debian forky/main amd64 autopoint all 0.23.1-2 [770 kB]
Get: 16 http://deb.debian.org/debian forky/main amd64 libdebhelper-perl all 13.28 [92.4 kB]
Get: 17 http://deb.debian.org/debian forky/main amd64 libtool all 2.5.4-7 [540 kB]
Get: 18 http://deb.debian.org/debian forky/main amd64 dh-autoreconf all 21 [12.2 kB]
Get: 19 http://deb.debian.org/debian forky/main amd64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian forky/main amd64 libfile-stripnondeterminism-perl all 1.15.0-1 [19.9 kB]
Get: 21 http://deb.debian.org/debian forky/main amd64 dh-strip-nondeterminism all 1.15.0-1 [8812 B]
Get: 22 http://deb.debian.org/debian forky/main amd64 libelf1t64 amd64 0.193-3 [192 kB]
Get: 23 http://deb.debian.org/debian forky/main amd64 dwz amd64 0.16-2 [108 kB]
Get: 24 http://deb.debian.org/debian forky/main amd64 libunistring5 amd64 1.3-2 [477 kB]
Get: 25 http://deb.debian.org/debian forky/main amd64 libxml2-16 amd64 2.14.6+dfsg-0.1 [651 kB]
Get: 26 http://deb.debian.org/debian forky/main amd64 gettext amd64 0.23.1-2+b1 [1680 kB]
Get: 27 http://deb.debian.org/debian forky/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 28 http://deb.debian.org/debian forky/main amd64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 29 http://deb.debian.org/debian forky/main amd64 debhelper all 13.28 [941 kB]
Get: 30 http://deb.debian.org/debian forky/main amd64 dh-golang all 1.63 [24.4 kB]
Get: 31 http://deb.debian.org/debian forky/main amd64 golang-1.24-src all 1.24.9-1 [21.2 MB]
Get: 32 http://deb.debian.org/debian forky/main amd64 golang-1.24-go amd64 1.24.9-1 [28.7 MB]
Get: 33 http://deb.debian.org/debian forky/main amd64 golang-src all 2:1.24~2 [5136 B]
Get: 34 http://deb.debian.org/debian forky/main amd64 golang-go amd64 2:1.24~2 [44.3 kB]
Get: 35 http://deb.debian.org/debian forky/main amd64 golang-any amd64 2:1.24~2 [5216 B]
Get: 36 http://deb.debian.org/debian forky/main amd64 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B]
Get: 37 http://deb.debian.org/debian forky/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 38 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 39 http://deb.debian.org/debian forky/main amd64 golang-golang-x-sys-dev all 0.36.0-1 [640 kB]
Get: 40 http://deb.debian.org/debian forky/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 41 http://deb.debian.org/debian forky/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 42 http://deb.debian.org/debian forky/main amd64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 43 http://deb.debian.org/debian forky/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 44 http://deb.debian.org/debian forky/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 45 http://deb.debian.org/debian forky/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 46 http://deb.debian.org/debian forky/main amd64 golang-gopkg-yaml.v3-dev all 3.0.4-1 [76.0 kB]
Get: 47 http://deb.debian.org/debian forky/main amd64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB]
Get: 48 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 49 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 50 http://deb.debian.org/debian forky/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 51 http://deb.debian.org/debian forky/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 52 http://deb.debian.org/debian forky/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 53 http://deb.debian.org/debian forky/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 54 http://deb.debian.org/debian forky/main amd64 golang-github-google-go-cmp-dev all 0.7.0-1 [87.0 kB]
Get: 55 http://deb.debian.org/debian forky/main amd64 golang-google-protobuf-dev all 1.36.7-1 [736 kB]
Get: 56 http://deb.debian.org/debian forky/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-2 [137 kB]
Get: 57 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 58 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 59 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 60 http://deb.debian.org/debian forky/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 61 http://deb.debian.org/debian forky/main amd64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB]
Get: 62 http://deb.debian.org/debian forky/main amd64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 63 http://deb.debian.org/debian forky/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 64 http://deb.debian.org/debian forky/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 65 http://deb.debian.org/debian forky/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 66 http://deb.debian.org/debian forky/main amd64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB]
Get: 67 http://deb.debian.org/debian forky/main amd64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 68 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-client-model-dev all 0.6.2-1 [11.1 kB]
Get: 69 http://deb.debian.org/debian forky/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 70 http://deb.debian.org/debian forky/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 71 http://deb.debian.org/debian forky/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 72 http://deb.debian.org/debian forky/main amd64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 73 http://deb.debian.org/debian forky/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 74 http://deb.debian.org/debian forky/main amd64 golang-golang-x-term-dev all 0.35.0-1 [16.1 kB]
Get: 75 http://deb.debian.org/debian forky/main amd64 golang-golang-x-text-dev all 0.29.0-2 [3992 kB]
Get: 76 http://deb.debian.org/debian forky/main amd64 golang-golang-x-crypto-dev all 1:0.43.0-2 [1950 kB]
Get: 77 http://deb.debian.org/debian forky/main amd64 golang-golang-x-net-dev all 1:0.27.0-2 [904 kB]
Get: 78 http://deb.debian.org/debian forky/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB]
Get: 79 http://deb.debian.org/debian forky/main amd64 golang-github-aws-smithy-go-dev all 1.20.3-1 [125 kB]
Get: 80 http://deb.debian.org/debian forky/main amd64 golang-gopkg-yaml.v2-dev all 2.4.2-1 [60.8 kB]
Get: 81 http://deb.debian.org/debian forky/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 82 http://deb.debian.org/debian forky/main amd64 golang-github-aws-aws-sdk-go-v2-dev all 1.30.3-1 [15.8 MB]
Get: 83 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-sigv4-dev all 0.2.1-1 [9724 B]
Get: 84 http://deb.debian.org/debian forky/main amd64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB]
Get: 85 http://deb.debian.org/debian forky/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 86 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-common-dev all 0.65.0-1 [120 kB]
Get: 87 http://deb.debian.org/debian forky/main amd64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 88 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-procfs-dev all 0.17.0-1 [199 kB]
Get: 89 http://deb.debian.org/debian forky/main amd64 golang-uber-goleak-dev all 1.3.0-1 [18.1 kB]
Get: 90 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-client-golang-dev all 1.23.0-2 [220 kB]
Get: 91 http://deb.debian.org/debian forky/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-2 [35.2 kB]
Get: 92 http://deb.debian.org/debian forky/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 93 http://deb.debian.org/debian forky/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 94 http://deb.debian.org/debian forky/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 95 http://deb.debian.org/debian forky/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 96 http://deb.debian.org/debian forky/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-10 [19.8 kB]
Get: 97 http://deb.debian.org/debian forky/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-10 [57.8 kB]
Get: 98 http://deb.debian.org/debian forky/main amd64 libssl-dev amd64 3.5.4-1 [2980 kB]
Get: 99 http://deb.debian.org/debian forky/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-10 [235 kB]
Get: 100 http://deb.debian.org/debian forky/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 101 http://deb.debian.org/debian forky/main amd64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 102 http://deb.debian.org/debian forky/main amd64 golang-github-yuin-goldmark-dev all 1.7.13-1 [113 kB]
Get: 103 http://deb.debian.org/debian forky/main amd64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB]
Get: 104 http://deb.debian.org/debian forky/main amd64 golang-github-rogpeppe-go-internal-dev all 1.14.1-1 [103 kB]
Get: 105 http://deb.debian.org/debian forky/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 106 http://deb.debian.org/debian forky/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB]
Get: 107 http://deb.debian.org/debian forky/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 108 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 126 MB in 5s (26.9 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19869 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.26_all.deb ...
Unpacking sensible-utils (0.0.26) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.46-5_amd64.deb ...
Unpacking libmagic-mgc (1:5.46-5) ...
Selecting previously unselected package libmagic1t64:amd64.
Preparing to unpack .../002-libmagic1t64_1%3a5.46-5_amd64.deb ...
Unpacking libmagic1t64:amd64 (1:5.46-5) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.46-5_amd64.deb ...
Unpacking file (1:5.46-5) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-2+b1_amd64.deb ...
Unpacking gettext-base (0.23.1-2+b1) ...
Selecting previously unselected package libuchardet0:amd64.
Preparing to unpack .../005-libuchardet0_0.0.8-2_amd64.deb ...
Unpacking libuchardet0:amd64 (0.0.8-2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-9_amd64.deb ...
Unpacking groff-base (1.23.0-9) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.41.2-4_amd64.deb ...
Unpacking bsdextrautils (2.41.2-4) ...
Selecting previously unselected package libpipeline1:amd64.
Preparing to unpack .../008-libpipeline1_1.5.8-1_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.1-1_amd64.deb ...
Unpacking man-db (2.13.1-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.20-2_amd64.deb ...
Unpacking m4 (1.4.20-2) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3.1_all.deb ...
Unpacking autoconf (2.72-3.1) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20240727.1_all.deb ...
Unpacking autotools-dev (20240727.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.18.1-2_all.deb ...
Unpacking automake (1:1.18.1-2) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-2_all.deb ...
Unpacking autopoint (0.23.1-2) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.28_all.deb ...
Unpacking libdebhelper-perl (13.28) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-7_all.deb ...
Unpacking libtool (2.5.4-7) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_21_all.deb ...
Unpacking dh-autoreconf (21) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.15.0-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.15.0-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.15.0-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.15.0-1) ...
Selecting previously unselected package libelf1t64:amd64.
Preparing to unpack .../021-libelf1t64_0.193-3_amd64.deb ...
Unpacking libelf1t64:amd64 (0.193-3) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.16-2_amd64.deb ...
Unpacking dwz (0.16-2) ...
Selecting previously unselected package libunistring5:amd64.
Preparing to unpack .../023-libunistring5_1.3-2_amd64.deb ...
Unpacking libunistring5:amd64 (1.3-2) ...
Selecting previously unselected package libxml2-16:amd64.
Preparing to unpack .../024-libxml2-16_2.14.6+dfsg-0.1_amd64.deb ...
Unpacking libxml2-16:amd64 (2.14.6+dfsg-0.1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../025-gettext_0.23.1-2+b1_amd64.deb ...
Unpacking gettext (0.23.1-2+b1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../028-debhelper_13.28_all.deb ...
Unpacking debhelper (13.28) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../029-dh-golang_1.63_all.deb ...
Unpacking dh-golang (1.63) ...
Selecting previously unselected package golang-1.24-src.
Preparing to unpack .../030-golang-1.24-src_1.24.9-1_all.deb ...
Unpacking golang-1.24-src (1.24.9-1) ...
Selecting previously unselected package golang-1.24-go.
Preparing to unpack .../031-golang-1.24-go_1.24.9-1_amd64.deb ...
Unpacking golang-1.24-go (1.24.9-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../032-golang-src_2%3a1.24~2_all.deb ...
Unpacking golang-src (2:1.24~2) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../033-golang-go_2%3a1.24~2_amd64.deb ...
Unpacking golang-go:amd64 (2:1.24~2) ...
Selecting previously unselected package golang-any:amd64.
Preparing to unpack .../034-golang-any_2%3a1.24~2_amd64.deb ...
Unpacking golang-any:amd64 (2:1.24~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../038-golang-golang-x-sys-dev_0.36.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.36.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../041-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.4-1_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.4-1) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.7.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.7.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../054-golang-google-protobuf-dev_1.36.7-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.36.7-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-2_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../065-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../066-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../067-golang-github-prometheus-client-model-dev_0.6.2-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.2-1) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../068-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../069-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../070-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../071-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../072-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../073-golang-golang-x-term-dev_0.35.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.35.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../074-golang-golang-x-text-dev_0.29.0-2_all.deb ...
Unpacking golang-golang-x-text-dev (0.29.0-2) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../075-golang-golang-x-crypto-dev_1%3a0.43.0-2_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.43.0-2) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../076-golang-golang-x-net-dev_1%3a0.27.0-2_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-2) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Selecting previously unselected package golang-github-aws-smithy-go-dev.
Preparing to unpack .../078-golang-github-aws-smithy-go-dev_1.20.3-1_all.deb ...
Unpacking golang-github-aws-smithy-go-dev (1.20.3-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../079-golang-gopkg-yaml.v2-dev_2.4.2-1_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.2-1) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../080-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-v2-dev.
Preparing to unpack .../081-golang-github-aws-aws-sdk-go-v2-dev_1.30.3-1_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-v2-dev (1.30.3-1) ...
Selecting previously unselected package golang-github-prometheus-sigv4-dev.
Preparing to unpack .../082-golang-github-prometheus-sigv4-dev_0.2.1-1_all.deb ...
Unpacking golang-github-prometheus-sigv4-dev (0.2.1-1) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../083-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../084-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../085-golang-github-prometheus-common-dev_0.65.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.65.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../086-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../087-golang-github-prometheus-procfs-dev_0.17.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.17.0-1) ...
Selecting previously unselected package golang-uber-goleak-dev.
Preparing to unpack .../088-golang-uber-goleak-dev_1.3.0-1_all.deb ...
Unpacking golang-uber-goleak-dev (1.3.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../089-golang-github-prometheus-client-golang-dev_1.23.0-2_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.23.0-2) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../090-golang-github-armon-go-metrics-dev_0.4.1-2_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-2) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../091-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../092-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../093-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../094-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../095-libsasl2-modules-db_2.1.28+dfsg1-10_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-10) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../096-libsasl2-2_2.1.28+dfsg1-10_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-10) ...
Selecting previously unselected package libssl-dev:amd64.
Preparing to unpack .../097-libssl-dev_3.5.4-1_amd64.deb ...
Unpacking libssl-dev:amd64 (3.5.4-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../098-libsasl2-dev_2.1.28+dfsg1-10_amd64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-10) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../099-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../100-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../101-golang-github-yuin-goldmark-dev_1.7.13-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.13-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../102-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../103-golang-github-rogpeppe-go-internal-dev_1.14.1-1_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.14.1-1) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../104-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../105-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../106-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../107-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-1.24-src (1.24.9-1) ...
Setting up libpipeline1:amd64 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.7.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.2-1) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up bsdextrautils (2.41.2-4) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.46-5) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-aws-smithy-go-dev (1.20.3-1) ...
Setting up libxml2-16:amd64 (2.14.6+dfsg-0.1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.4-1) ...
Setting up libdebhelper-perl (13.28) ...
Setting up golang-golang-x-sys-dev (0.36.0-1) ...
Setting up libmagic1t64:amd64 (1:5.46-5) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-2+b1) ...
Setting up m4 (1.4.20-2) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.35.0-1) ...
Setting up file (1:5.46-5) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-3) ...
Setting up libelf1t64:amd64 (0.193-3) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-10) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20240727.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.36.7-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.10.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.13-1) ...
Setting up libunistring5:amd64 (1.3-2) ...
Setting up libssl-dev:amd64 (3.5.4-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up autopoint (0.23.1-2) ...
Setting up golang-1.24-go (1.24.9-1) ...
Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-10) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3.1) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.16-2) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.26) ...
Setting up golang-golang-x-text-dev (0.29.0-2) ...
Setting up libuchardet0:amd64 (0.0.8-2) ...
Setting up golang-github-prometheus-procfs-dev (0.17.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-uber-goleak-dev (1.3.0-1) ...
Setting up golang-golang-x-crypto-dev (1:0.43.0-2) ...
Setting up golang-github-prometheus-client-model-dev (0.6.2-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.27.0-1) ...
Setting up golang-src (2:1.24~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.18.1-2) ...
update-alternatives: using /usr/bin/automake-1.18 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.15.0-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-2) ...
Setting up gettext (0.23.1-2+b1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-7) ...
Setting up golang-go:amd64 (2:1.24~2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up golang-any:amd64 (2:1.24~2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (21) ...
Setting up libsasl2-dev (2.1.28+dfsg1-10) ...
Setting up golang-golang-x-net-dev (1:0.27.0-2) ...
Setting up dh-strip-nondeterminism (1.15.0-1) ...
Setting up groff-base (1.23.0-9) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Setting up man-db (2.13.1-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-github-aws-aws-sdk-go-v2-dev (1.30.3-1) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.14.1-1) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.28) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.63) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Setting up golang-github-prometheus-sigv4-dev (0.2.1-1) ...
Setting up golang-github-prometheus-common-dev (0.65.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.23.0-2) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-2) ...
Processing triggers for libc-bin (2.41-12) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go install -trimpath -v -p 40 github.com/hashicorp/raft github.com/hashicorp/raft/bench
encoding
internal/goarch
vendor/golang.org/x/crypto/cryptobyte/asn1
crypto/internal/fips140/alias
internal/itoa
unicode/utf16
cmp
container/list
internal/byteorder
log/internal
internal/coverage/rtcov
vendor/golang.org/x/crypto/internal/alias
sync/atomic
internal/nettrace
math/bits
internal/profilerecord
unicode/utf8
internal/goexperiment
internal/goos
crypto/internal/boring/sig
internal/runtime/syscall
internal/unsafeheader
unicode
internal/cpu
internal/msan
internal/runtime/atomic
internal/godebugs
internal/asan
internal/abi
internal/runtime/math
crypto/internal/fips140/subtle
crypto/internal/fips140deps/byteorder
internal/chacha8rand
internal/runtime/sys
internal/runtime/exithook
crypto/internal/fips140deps/cpu
internal/bytealg
math
internal/stringslite
internal/race
internal/sync
internal/runtime/maps
runtime
internal/reflectlite
iter
crypto/subtle
sync
weak
maps
slices
errors
sort
internal/bisect
internal/testlog
internal/singleflight
unique
internal/oserror
github.com/hashicorp/golang-lru/simplelru
io
vendor/golang.org/x/net/dns/dnsmessage
path
math/rand/v2
strconv
runtime/cgo
syscall
internal/godebug
internal/saferio
hash
crypto/internal/randutil
strings
bytes
net/netip
reflect
crypto
hash/crc64
hash/crc32
crypto/internal/fips140deps/godebug
math/rand
vendor/golang.org/x/text/transform
bufio
crypto/internal/fips140
regexp/syntax
crypto/internal/impl
net/http/internal/ascii
github.com/hashicorp/go-immutable-radix
html
crypto/tls/internal/fips140tls
crypto/internal/fips140/sha256
crypto/internal/fips140/sha512
crypto/internal/fips140/sha3
internal/syscall/execenv
internal/syscall/unix
time
crypto/internal/fips140/hmac
crypto/sha3
crypto/internal/fips140/check
crypto/internal/fips140hash
crypto/internal/fips140/edwards25519/field
crypto/internal/fips140/bigmod
crypto/internal/fips140/hkdf
crypto/internal/fips140/tls12
crypto/internal/fips140/aes
crypto/internal/fips140/nistec/fiat
crypto/internal/fips140/tls13
regexp
crypto/internal/fips140/edwards25519
context
io/fs
internal/poll
internal/filepathlite
crypto/internal/fips140/nistec
os
internal/fmtsort
encoding/binary
encoding/base64
vendor/golang.org/x/crypto/internal/poly1305
golang.org/x/sys/unix
encoding/pem
crypto/internal/sysrand
path/filepath
os/signal
io/ioutil
vendor/golang.org/x/sys/cpu
internal/sysinfo
fmt
net
crypto/internal/entropy
crypto/internal/fips140/drbg
crypto/internal/fips140only
crypto/internal/fips140/ecdh
crypto/internal/fips140/ed25519
crypto/internal/fips140/ecdsa
crypto/internal/fips140/aes/gcm
crypto/internal/fips140/rsa
crypto/internal/fips140/mlkem
crypto/rc4
crypto/md5
crypto/cipher
crypto/internal/boring
crypto/des
vendor/golang.org/x/crypto/chacha20
log
encoding/hex
net/url
vendor/golang.org/x/net/http2/hpack
net/http/internal
compress/flate
runtime/trace
mime/quotedprintable
text/template/parse
go/token
flag
mime
encoding/gob
encoding/json
math/big
vendor/golang.org/x/text/unicode/norm
runtime/debug
crypto/aes
crypto/sha1
crypto/sha256
crypto/ecdh
crypto/sha512
crypto/hmac
vendor/golang.org/x/crypto/chacha20poly1305
vendor/golang.org/x/text/unicode/bidi
compress/gzip
testing
vendor/golang.org/x/text/secure/bidirule
text/template
vendor/golang.org/x/net/idna
github.com/mattn/go-isatty
github.com/mattn/go-colorable
github.com/fatih/color
crypto/internal/boring/bbig
crypto/rand
crypto/elliptic
crypto/dsa
encoding/asn1
crypto/ed25519
crypto/internal/hpke
crypto/rsa
github.com/hashicorp/go-hclog
html/template
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
crypto/ecdsa
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-x86_64-linux-gnu && go test -vet=off -v -p 40 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-10-30T18:17:33.723Z [INFO]  creating new snapshot: path=/tmp/raft1301350448/raft1623531546/snapshots/3-10-1761848253723.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2025-10-30T18:17:33.727Z [INFO]  creating new snapshot: path=/tmp/raft1842263271/snapshots/3-10-1761848253727.tmp
2025-10-30T18:17:33.729Z [WARN]  found temporary snapshot: name=3-10-1761848253727.tmp
--- PASS: TestFileSS_CreateSnapshot (0.01s)
=== RUN   TestFileSS_CancelSnapshot
2025-10-30T18:17:33.736Z [INFO]  creating new snapshot: path=/tmp/raft2525840641/snapshots/3-10-1761848253736.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2025-10-30T18:17:33.740Z [INFO]  creating new snapshot: path=/tmp/raft3199211494/snapshots/3-10-1761848253740.tmp
2025-10-30T18:17:33.747Z [INFO]  creating new snapshot: path=/tmp/raft3199211494/snapshots/3-11-1761848253747.tmp
2025-10-30T18:17:33.753Z [INFO]  creating new snapshot: path=/tmp/raft3199211494/snapshots/3-12-1761848253753.tmp
2025-10-30T18:17:33.759Z [INFO]  reaping snapshot: path=/tmp/raft3199211494/snapshots/3-10-1761848253740
2025-10-30T18:17:33.759Z [INFO]  creating new snapshot: path=/tmp/raft3199211494/snapshots/3-13-1761848253759.tmp
2025-10-30T18:17:33.769Z [INFO]  reaping snapshot: path=/tmp/raft3199211494/snapshots/3-11-1761848253747
2025-10-30T18:17:33.769Z [INFO]  creating new snapshot: path=/tmp/raft3199211494/snapshots/3-14-1761848253769.tmp
2025-10-30T18:17:33.774Z [INFO]  reaping snapshot: path=/tmp/raft3199211494/snapshots/3-12-1761848253753
--- PASS: TestFileSS_Retention (0.03s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-10-30T18:17:33.775Z [INFO]  creating new snapshot: path=/tmp/raft1018714603/snapshots/5-130350-1761848253775.tmp
2025-10-30T18:17:33.781Z [INFO]  creating new snapshot: path=/tmp/raft1018714603/snapshots/36-204917-1761848253781.tmp
--- PASS: TestFileSS_Ordering (0.01s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2025-10-30T18:17:33.882Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2025-10-30T18:17:33.933Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
2025-10-30T18:17:33.986Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:33.992Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:33.997Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.008Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.029Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.069Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.150Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.311Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.631Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:35.273Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:36.275Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-10-30T18:17:37.278Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-10-30T18:17:37.995Z [INFO]  server-9b6a8f26-9120-f2d4-7a0a-1d887770fcd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b6a8f26-9120-f2d4-7a0a-1d887770fcd1 Address:9b6a8f26-9120-f2d4-7a0a-1d887770fcd1}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-10-30T18:17:37.996Z [INFO]  server-60e5c947-a6a2-144b-240f-a0de50f4e1b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60e5c947-a6a2-144b-240f-a0de50f4e1b9 Address:60e5c947-a6a2-144b-240f-a0de50f4e1b9}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-10-30T18:17:37.996Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: initial configuration: index=0 servers=[]
2025-10-30T18:17:37.996Z [INFO]  server-b4888378-eb1e-f397-e610-7d6790cf66c5: initial configuration: index=0 servers=[]
2025-10-30T18:17:37.996Z [INFO]  server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7: initial configuration: index=0 servers=[]
2025-10-30T18:17:37.996Z [INFO]  server-b4888378-eb1e-f397-e610-7d6790cf66c5: entering follower state: follower="Node at b4888378-eb1e-f397-e610-7d6790cf66c5 [Follower]" leader-address= leader-id=
2025-10-30T18:17:37.996Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: entering follower state: follower="Node at 27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4 [Follower]" leader-address= leader-id=
2025-10-30T18:17:37.996Z [INFO]  server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7: entering follower state: follower="Node at 69fac77c-a8eb-eaae-d882-c1a0fb928fb7 [Follower]" leader-address= leader-id=
2025-10-30T18:17:37.996Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:38.067Z [WARN]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:38.067Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: entering candidate state: node="Node at 27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4 [Candidate]" term=2
2025-10-30T18:17:38.067Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: election won: term=2 tally=2
2025-10-30T18:17:38.067Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: entering leader state: leader="Node at 27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4 [Leader]"
2025-10-30T18:17:38.067Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: added peer, starting replication: peer=server-b4888378-eb1e-f397-e610-7d6790cf66c5
2025-10-30T18:17:38.067Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: added peer, starting replication: peer=server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7
2025-10-30T18:17:38.067Z [WARN]  server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-10-30T18:17:38.067Z [WARN]  server-b4888378-eb1e-f397-e610-7d6790cf66c5: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-10-30T18:17:38.067Z [WARN]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: appendEntries rejected, sending older logs: peer="{Voter server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7 69fac77c-a8eb-eaae-d882-c1a0fb928fb7}" next=1
2025-10-30T18:17:38.067Z [WARN]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: appendEntries rejected, sending older logs: peer="{Voter server-b4888378-eb1e-f397-e610-7d6790cf66c5 b4888378-eb1e-f397-e610-7d6790cf66c5}" next=1
2025-10-30T18:17:38.067Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: pipelining replication: peer="{Voter server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7 69fac77c-a8eb-eaae-d882-c1a0fb928fb7}"
2025-10-30T18:17:38.067Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: pipelining replication: peer="{Voter server-b4888378-eb1e-f397-e610-7d6790cf66c5 b4888378-eb1e-f397-e610-7d6790cf66c5}"
2025-10-30T18:17:38.172Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:38.067494818 +0000 UTC m=+4.346168729 (2 nodes), 70.567714ms from start of poll, 70.914544ms from cluster start. Timeout at 2025-10-30 18:17:38.172497366 +0000 UTC m=+4.451171274, 105.002545ms after stability
2025-10-30T18:17:38.172Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:38.278Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:38.172853242 +0000 UTC m=+4.451527142 (1 nodes), 2.726µs from start of poll, 176.272957ms from cluster start. Timeout at 2025-10-30 18:17:38.277856571 +0000 UTC m=+4.556530469, 105.003327ms after stability
2025-10-30T18:17:38.289Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: aborting pipeline replication: peer="{Voter server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7 69fac77c-a8eb-eaae-d882-c1a0fb928fb7}"
2025-10-30T18:17:38.289Z [INFO]  server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: aborting pipeline replication: peer="{Voter server-b4888378-eb1e-f397-e610-7d6790cf66c5 b4888378-eb1e-f397-e610-7d6790cf66c5}"
--- PASS: TestRaft_LiveBootstrap (0.29s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-10-30T18:17:38.289Z [INFO]  server-3bc2969f-9d21-f095-ac30-7b886e1c9150: initial configuration: index=0 servers=[]
2025-10-30T18:17:38.290Z [INFO]  server-3d228883-6ce2-d4ae-29dc-540d24c2e147: initial configuration: index=0 servers=[]
2025-10-30T18:17:38.290Z [INFO]  server-3bc2969f-9d21-f095-ac30-7b886e1c9150: entering follower state: follower="Node at 3bc2969f-9d21-f095-ac30-7b886e1c9150 [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.290Z [INFO]  server-3d228883-6ce2-d4ae-29dc-540d24c2e147: entering follower state: follower="Node at 3d228883-6ce2-d4ae-29dc-540d24c2e147 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-10-30T18:17:38.291Z [INFO]  server-e32e11de-8c24-8ddd-6f57-c2afa4630df3: initial configuration: index=0 servers=[]
2025-10-30T18:17:38.291Z [INFO]  server-e32e11de-8c24-8ddd-6f57-c2afa4630df3: entering follower state: follower="Node at e32e11de-8c24-8ddd-6f57-c2afa4630df3 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-10-30T18:17:38.292Z [INFO]  server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]"
2025-10-30T18:17:38.292Z [INFO]  server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: entering follower state: follower="Node at 92be0f91-b5b2-75a3-4841-1bc480c6fa3f [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.292Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]"
2025-10-30T18:17:38.292Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering follower state: follower="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.292Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]"
2025-10-30T18:17:38.292Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:38.292Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: entering follower state: follower="Node at 9401301f-d510-4987-3d54-346790456ef9 [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.355Z [WARN]  server-9401301f-d510-4987-3d54-346790456ef9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:38.355Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: entering candidate state: node="Node at 9401301f-d510-4987-3d54-346790456ef9 [Candidate]" term=2
2025-10-30T18:17:38.355Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: election won: term=2 tally=2
2025-10-30T18:17:38.355Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: entering leader state: leader="Node at 9401301f-d510-4987-3d54-346790456ef9 [Leader]"
2025-10-30T18:17:38.355Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: added peer, starting replication: peer=server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f
2025-10-30T18:17:38.355Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: added peer, starting replication: peer=server-d21182ed-9821-e0a5-d00a-fa1fdf90f057
2025-10-30T18:17:38.355Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: pipelining replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}"
2025-10-30T18:17:38.355Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: pipelining replication: peer="{Voter server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 d21182ed-9821-e0a5-d00a-fa1fdf90f057}"
2025-10-30T18:17:38.461Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:38.355734303 +0000 UTC m=+4.634408211 (1 nodes), 63.02207ms from start of poll, 63.290007ms from cluster start. Timeout at 2025-10-30 18:17:38.460737941 +0000 UTC m=+4.739411838, 105.003627ms after stability
2025-10-30T18:17:38.461Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: aborting pipeline replication: peer="{Voter server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 d21182ed-9821-e0a5-d00a-fa1fdf90f057}"
2025-10-30T18:17:38.461Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: aborting pipeline replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}"
2025-10-30T18:17:38.461Z [INFO]  creating new snapshot: path=/tmp/raft514602561/snapshots/2-2-1761848258461.tmp
2025-10-30T18:17:38.462Z [INFO]  server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: starting restore from snapshot: id=2-2-1761848258461 last-index=2 last-term=2 size-in-bytes=1
2025-10-30T18:17:38.462Z [INFO]  server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: snapshot restore progress: id=2-2-1761848258461 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-10-30T18:17:38.462Z [INFO]  server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: restored from snapshot: id=2-2-1761848258461 last-index=2 last-term=2 size-in-bytes=1
2025-10-30T18:17:38.462Z [INFO]  server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]"
2025-10-30T18:17:38.462Z [INFO]  creating new snapshot: path=/tmp/raft2579846394/snapshots/2-2-1761848258462.tmp
2025-10-30T18:17:38.462Z [INFO]  server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: entering follower state: follower="Node at 92be0f91-b5b2-75a3-4841-1bc480c6fa3f [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.462Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: starting restore from snapshot: id=2-2-1761848258462 last-index=2 last-term=2 size-in-bytes=1
2025-10-30T18:17:38.463Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: snapshot restore progress: id=2-2-1761848258462 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-10-30T18:17:38.463Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: restored from snapshot: id=2-2-1761848258462 last-index=2 last-term=2 size-in-bytes=1
2025-10-30T18:17:38.463Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]"
2025-10-30T18:17:38.463Z [INFO]  creating new snapshot: path=/tmp/raft3251839670/snapshots/2-2-1761848258463.tmp
2025-10-30T18:17:38.463Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering follower state: follower="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.463Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: starting restore from snapshot: id=2-2-1761848258463 last-index=2 last-term=2 size-in-bytes=1
2025-10-30T18:17:38.464Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: snapshot restore progress: id=2-2-1761848258463 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-10-30T18:17:38.464Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: restored from snapshot: id=2-2-1761848258463 last-index=2 last-term=2 size-in-bytes=1
2025-10-30T18:17:38.464Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]"
2025-10-30T18:17:38.464Z [INFO]  server-9401301f-d510-4987-3d54-346790456ef9: entering follower state: follower="Node at 9401301f-d510-4987-3d54-346790456ef9 [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.530Z [WARN]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:38.530Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering candidate state: node="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Candidate]" term=3
2025-10-30T18:17:38.530Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: election won: term=3 tally=2
2025-10-30T18:17:38.530Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering leader state: leader="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Leader]"
2025-10-30T18:17:38.530Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: added peer, starting replication: peer=server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f
2025-10-30T18:17:38.530Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: added peer, starting replication: peer=server-9401301f-d510-4987-3d54-346790456ef9
2025-10-30T18:17:38.531Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: pipelining replication: peer="{Voter server-9401301f-d510-4987-3d54-346790456ef9 9401301f-d510-4987-3d54-346790456ef9}"
2025-10-30T18:17:38.531Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: pipelining replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}"
2025-10-30T18:17:38.779Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:38.885Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:38.779311619 +0000 UTC m=+5.057985519 (1 nodes), 23.978µs from start of poll, 486.867315ms from cluster start. Timeout at 2025-10-30 18:17:38.884314389 +0000 UTC m=+5.162988287, 105.002768ms after stability
2025-10-30T18:17:38.885Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: aborting pipeline replication: peer="{Voter server-9401301f-d510-4987-3d54-346790456ef9 9401301f-d510-4987-3d54-346790456ef9}"
2025-10-30T18:17:38.886Z [INFO]  server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: aborting pipeline replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-10-30T18:17:38.886Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]"
2025-10-30T18:17:38.887Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]"
2025-10-30T18:17:38.887Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering follower state: follower="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.887Z [INFO]  server-5c971473-666b-247a-7af5-7befb1d9604b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]"
2025-10-30T18:17:38.887Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering follower state: follower="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.887Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:38.887Z [INFO]  server-5c971473-666b-247a-7af5-7befb1d9604b: entering follower state: follower="Node at 5c971473-666b-247a-7af5-7befb1d9604b [Follower]" leader-address= leader-id=
2025-10-30T18:17:38.938Z [WARN]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:38.938Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering candidate state: node="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Candidate]" term=2
2025-10-30T18:17:38.938Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: election won: term=2 tally=2
2025-10-30T18:17:38.938Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering leader state: leader="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Leader]"
2025-10-30T18:17:38.938Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: added peer, starting replication: peer=server-c8b39909-a245-775b-ac90-3f65d0dc0b2a
2025-10-30T18:17:38.938Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: added peer, starting replication: peer=server-5c971473-666b-247a-7af5-7befb1d9604b
2025-10-30T18:17:38.939Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: pipelining replication: peer="{Voter server-c8b39909-a245-775b-ac90-3f65d0dc0b2a c8b39909-a245-775b-ac90-3f65d0dc0b2a}"
2025-10-30T18:17:38.939Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: pipelining replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}"
2025-10-30T18:17:39.044Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:38.938953002 +0000 UTC m=+5.217626900 (1 nodes), 51.809402ms from start of poll, 52.0603ms from cluster start. Timeout at 2025-10-30 18:17:39.043954359 +0000 UTC m=+5.322628260, 105.00136ms after stability
2025-10-30T18:17:39.044Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: aborting pipeline replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}"
2025-10-30T18:17:39.044Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: aborting pipeline replication: peer="{Voter server-c8b39909-a245-775b-ac90-3f65d0dc0b2a c8b39909-a245-775b-ac90-3f65d0dc0b2a}"
2025-10-30T18:17:39.044Z [INFO]  creating new snapshot: path=/tmp/raft4199091538/snapshots/2-6-1761848259044.tmp
2025-10-30T18:17:39.044Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: starting restore from snapshot: id=2-6-1761848259044 last-index=6 last-term=2 size-in-bytes=25
2025-10-30T18:17:39.045Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: snapshot restore progress: id=2-6-1761848259044 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-10-30T18:17:39.045Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: restored from snapshot: id=2-6-1761848259044 last-index=6 last-term=2 size-in-bytes=25
2025-10-30T18:17:39.045Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]"
2025-10-30T18:17:39.045Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering follower state: follower="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.045Z [INFO]  creating new snapshot: path=/tmp/raft850242973/snapshots/2-6-1761848259045.tmp
2025-10-30T18:17:39.045Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: starting restore from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25
2025-10-30T18:17:39.045Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: snapshot restore progress: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-10-30T18:17:39.045Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: restored from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25
2025-10-30T18:17:39.045Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]"
2025-10-30T18:17:39.045Z [INFO]  creating new snapshot: path=/tmp/raft2530727206/snapshots/2-6-1761848259045.tmp
2025-10-30T18:17:39.045Z [INFO]  server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering follower state: follower="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.046Z [INFO]  server-5c971473-666b-247a-7af5-7befb1d9604b: starting restore from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25
2025-10-30T18:17:39.046Z [INFO]  server-5c971473-666b-247a-7af5-7befb1d9604b: snapshot restore progress: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-10-30T18:17:39.046Z [INFO]  server-5c971473-666b-247a-7af5-7befb1d9604b: restored from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25
2025-10-30T18:17:39.046Z [INFO]  server-5c971473-666b-247a-7af5-7befb1d9604b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]"
2025-10-30T18:17:39.046Z [INFO]  server-5c971473-666b-247a-7af5-7befb1d9604b: entering follower state: follower="Node at 5c971473-666b-247a-7af5-7befb1d9604b [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.112Z [WARN]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:39.112Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering candidate state: node="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Candidate]" term=3
2025-10-30T18:17:39.112Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: election won: term=3 tally=2
2025-10-30T18:17:39.112Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering leader state: leader="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Leader]"
2025-10-30T18:17:39.112Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: added peer, starting replication: peer=server-4cc2270b-0977-d62a-f36a-642c6b5a18b2
2025-10-30T18:17:39.112Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: added peer, starting replication: peer=server-5c971473-666b-247a-7af5-7befb1d9604b
2025-10-30T18:17:39.112Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: pipelining replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}"
2025-10-30T18:17:39.112Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: pipelining replication: peer="{Voter server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 4cc2270b-0977-d62a-f36a-642c6b5a18b2}"
2025-10-30T18:17:39.361Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:39.467Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:39.36185612 +0000 UTC m=+5.640530017 (1 nodes), 19.775µs from start of poll, 474.963417ms from cluster start. Timeout at 2025-10-30 18:17:39.466862959 +0000 UTC m=+5.745536856, 105.006839ms after stability
2025-10-30T18:17:39.467Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: aborting pipeline replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}"
2025-10-30T18:17:39.467Z [INFO]  server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: aborting pipeline replication: peer="{Voter server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 4cc2270b-0977-d62a-f36a-642c6b5a18b2}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-10-30T18:17:39.468Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]"
2025-10-30T18:17:39.468Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering follower state: follower="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.468Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]"
2025-10-30T18:17:39.468Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering follower state: follower="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.468Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]"
2025-10-30T18:17:39.468Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:39.468Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering follower state: follower="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.527Z [WARN]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:39.527Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering candidate state: node="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Candidate]" term=2
2025-10-30T18:17:39.527Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: election won: term=2 tally=2
2025-10-30T18:17:39.527Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering leader state: leader="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Leader]"
2025-10-30T18:17:39.527Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: added peer, starting replication: peer=server-abbab455-2a1d-ab78-f456-f78ffa09bbea
2025-10-30T18:17:39.527Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: added peer, starting replication: peer=server-d52a0d34-73b1-3c51-d424-a546abfa0ead
2025-10-30T18:17:39.528Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: pipelining replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}"
2025-10-30T18:17:39.528Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: pipelining replication: peer="{Voter server-d52a0d34-73b1-3c51-d424-a546abfa0ead d52a0d34-73b1-3c51-d424-a546abfa0ead}"
2025-10-30T18:17:39.633Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:39.527901003 +0000 UTC m=+5.806574899 (1 nodes), 59.037888ms from start of poll, 59.414779ms from cluster start. Timeout at 2025-10-30 18:17:39.632903401 +0000 UTC m=+5.911577299, 105.0024ms after stability
2025-10-30T18:17:39.635Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: aborting pipeline replication: peer="{Voter server-d52a0d34-73b1-3c51-d424-a546abfa0ead d52a0d34-73b1-3c51-d424-a546abfa0ead}"
2025-10-30T18:17:39.635Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: aborting pipeline replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}"
2025-10-30T18:17:39.635Z [INFO]  creating new snapshot: path=/tmp/raft2050563930/snapshots/2-27-1761848259635.tmp
2025-10-30T18:17:39.635Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: starting restore from snapshot: id=2-27-1761848259635 last-index=27 last-term=2 size-in-bytes=168
2025-10-30T18:17:39.635Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: snapshot restore progress: id=2-27-1761848259635 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-10-30T18:17:39.635Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: restored from snapshot: id=2-27-1761848259635 last-index=27 last-term=2 size-in-bytes=168
2025-10-30T18:17:39.636Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]"
2025-10-30T18:17:39.636Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering follower state: follower="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.636Z [INFO]  creating new snapshot: path=/tmp/raft542924102/snapshots/2-27-1761848259636.tmp
2025-10-30T18:17:39.636Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: starting restore from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168
2025-10-30T18:17:39.636Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: snapshot restore progress: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-10-30T18:17:39.636Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: restored from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168
2025-10-30T18:17:39.636Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]"
2025-10-30T18:17:39.636Z [INFO]  creating new snapshot: path=/tmp/raft1881663974/snapshots/2-27-1761848259636.tmp
2025-10-30T18:17:39.636Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering follower state: follower="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.636Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: starting restore from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168
2025-10-30T18:17:39.637Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: snapshot restore progress: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-10-30T18:17:39.637Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: restored from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168
2025-10-30T18:17:39.637Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]"
2025-10-30T18:17:39.637Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering follower state: follower="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Follower]" leader-address= leader-id=
2025-10-30T18:17:39.709Z [WARN]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:39.709Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering candidate state: node="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Candidate]" term=3
2025-10-30T18:17:39.709Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: election won: term=3 tally=2
2025-10-30T18:17:39.709Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering leader state: leader="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Leader]"
2025-10-30T18:17:39.709Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: added peer, starting replication: peer=server-abbab455-2a1d-ab78-f456-f78ffa09bbea
2025-10-30T18:17:39.709Z [WARN]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:39.709Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering candidate state: node="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Candidate]" term=3
2025-10-30T18:17:39.709Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: duplicate requestVote for same term: term=3
2025-10-30T18:17:39.709Z [INFO]  server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering follower state: follower="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Follower]" leader-address=d52a0d34-73b1-3c51-d424-a546abfa0ead leader-id=server-d52a0d34-73b1-3c51-d424-a546abfa0ead
2025-10-30T18:17:39.709Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: added peer, starting replication: peer=server-6039fdef-4f5f-b129-7db1-72c176ed8608
2025-10-30T18:17:39.709Z [INFO]  server-6039fdef-4f5f-b129-7db1-72c176ed8608: duplicate requestVote for same term: term=3
2025-10-30T18:17:39.709Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: pipelining replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}"
2025-10-30T18:17:39.709Z [WARN]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: rejecting vote request since we have a leader: from=abbab455-2a1d-ab78-f456-f78ffa09bbea leader=d52a0d34-73b1-3c51-d424-a546abfa0ead leader-id=server-d52a0d34-73b1-3c51-d424-a546abfa0ead
2025-10-30T18:17:39.709Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: pipelining replication: peer="{Voter server-6039fdef-4f5f-b129-7db1-72c176ed8608 6039fdef-4f5f-b129-7db1-72c176ed8608}"
2025-10-30T18:17:39.953Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:40.059Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:39.953642724 +0000 UTC m=+6.232316621 (1 nodes), 21.882µs from start of poll, 485.156501ms from cluster start. Timeout at 2025-10-30 18:17:40.058647922 +0000 UTC m=+6.337321819, 105.005198ms after stability
2025-10-30T18:17:40.059Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: aborting pipeline replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}"
2025-10-30T18:17:40.059Z [INFO]  server-d52a0d34-73b1-3c51-d424-a546abfa0ead: aborting pipeline replication: peer="{Voter server-6039fdef-4f5f-b129-7db1-72c176ed8608 6039fdef-4f5f-b129-7db1-72c176ed8608}"
--- PASS: TestRaft_RecoverCluster (1.77s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.58s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.59s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-10-30T18:17:40.059Z [INFO]  server-7189ecfb-ee69-aa78-562d-57a224cd7851: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7189ecfb-ee69-aa78-562d-57a224cd7851 Address:7189ecfb-ee69-aa78-562d-57a224cd7851}]"
2025-10-30T18:17:40.060Z [INFO]  server-7189ecfb-ee69-aa78-562d-57a224cd7851: entering follower state: follower="Node at 7189ecfb-ee69-aa78-562d-57a224cd7851 [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.123Z [WARN]  server-7189ecfb-ee69-aa78-562d-57a224cd7851: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:40.123Z [INFO]  server-7189ecfb-ee69-aa78-562d-57a224cd7851: entering candidate state: node="Node at 7189ecfb-ee69-aa78-562d-57a224cd7851 [Candidate]" term=2
2025-10-30T18:17:40.123Z [INFO]  server-7189ecfb-ee69-aa78-562d-57a224cd7851: election won: term=2 tally=1
2025-10-30T18:17:40.123Z [INFO]  server-7189ecfb-ee69-aa78-562d-57a224cd7851: entering leader state: leader="Node at 7189ecfb-ee69-aa78-562d-57a224cd7851 [Leader]"
--- PASS: TestRaft_SingleNode (0.06s)
=== RUN   TestRaft_TripleNode
2025-10-30T18:17:40.124Z [INFO]  server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 Address:32fe0f14-6e18-47c3-e923-cdb478e3a3b6} {Suffrage:Voter ID:server-544d3a6f-07c8-1695-9728-8192af9796d9 Address:544d3a6f-07c8-1695-9728-8192af9796d9} {Suffrage:Voter ID:server-24b6f6f6-d117-a22d-869b-0ba0aca8f188 Address:24b6f6f6-d117-a22d-869b-0ba0aca8f188}]"
2025-10-30T18:17:40.124Z [INFO]  server-544d3a6f-07c8-1695-9728-8192af9796d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 Address:32fe0f14-6e18-47c3-e923-cdb478e3a3b6} {Suffrage:Voter ID:server-544d3a6f-07c8-1695-9728-8192af9796d9 Address:544d3a6f-07c8-1695-9728-8192af9796d9} {Suffrage:Voter ID:server-24b6f6f6-d117-a22d-869b-0ba0aca8f188 Address:24b6f6f6-d117-a22d-869b-0ba0aca8f188}]"
2025-10-30T18:17:40.124Z [INFO]  server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6: entering follower state: follower="Node at 32fe0f14-6e18-47c3-e923-cdb478e3a3b6 [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.124Z [INFO]  server-544d3a6f-07c8-1695-9728-8192af9796d9: entering follower state: follower="Node at 544d3a6f-07c8-1695-9728-8192af9796d9 [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.124Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 Address:32fe0f14-6e18-47c3-e923-cdb478e3a3b6} {Suffrage:Voter ID:server-544d3a6f-07c8-1695-9728-8192af9796d9 Address:544d3a6f-07c8-1695-9728-8192af9796d9} {Suffrage:Voter ID:server-24b6f6f6-d117-a22d-869b-0ba0aca8f188 Address:24b6f6f6-d117-a22d-869b-0ba0aca8f188}]"
2025-10-30T18:17:40.124Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:40.124Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: entering follower state: follower="Node at 24b6f6f6-d117-a22d-869b-0ba0aca8f188 [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.187Z [WARN]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:40.187Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: entering candidate state: node="Node at 24b6f6f6-d117-a22d-869b-0ba0aca8f188 [Candidate]" term=2
2025-10-30T18:17:40.187Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: election won: term=2 tally=2
2025-10-30T18:17:40.187Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: entering leader state: leader="Node at 24b6f6f6-d117-a22d-869b-0ba0aca8f188 [Leader]"
2025-10-30T18:17:40.187Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: added peer, starting replication: peer=server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6
2025-10-30T18:17:40.187Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: added peer, starting replication: peer=server-544d3a6f-07c8-1695-9728-8192af9796d9
2025-10-30T18:17:40.187Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: pipelining replication: peer="{Voter server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 32fe0f14-6e18-47c3-e923-cdb478e3a3b6}"
2025-10-30T18:17:40.187Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: pipelining replication: peer="{Voter server-544d3a6f-07c8-1695-9728-8192af9796d9 544d3a6f-07c8-1695-9728-8192af9796d9}"
2025-10-30T18:17:40.292Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:40.187220837 +0000 UTC m=+6.465894735 (2 nodes), 62.373027ms from start of poll, 62.626567ms from cluster start. Timeout at 2025-10-30 18:17:40.292222693 +0000 UTC m=+6.570896590, 105.001855ms after stability
2025-10-30T18:17:40.292Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:40.397Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:40.292388294 +0000 UTC m=+6.571062195 (1 nodes), 2.083µs from start of poll, 167.794027ms from cluster start. Timeout at 2025-10-30 18:17:40.397392056 +0000 UTC m=+6.676065952, 105.003757ms after stability
2025-10-30T18:17:40.408Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: aborting pipeline replication: peer="{Voter server-544d3a6f-07c8-1695-9728-8192af9796d9 544d3a6f-07c8-1695-9728-8192af9796d9}"
2025-10-30T18:17:40.408Z [INFO]  server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: aborting pipeline replication: peer="{Voter server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 32fe0f14-6e18-47c3-e923-cdb478e3a3b6}"
--- PASS: TestRaft_TripleNode (0.28s)
=== RUN   TestRaft_LeaderFail
2025-10-30T18:17:40.409Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72284db2-fe2e-3801-8490-c7d6b8c15771 Address:72284db2-fe2e-3801-8490-c7d6b8c15771} {Suffrage:Voter ID:server-7d247416-3214-8134-dd1a-65dc0c50b23b Address:7d247416-3214-8134-dd1a-65dc0c50b23b} {Suffrage:Voter ID:server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b Address:cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}]"
2025-10-30T18:17:40.409Z [INFO]  server-7d247416-3214-8134-dd1a-65dc0c50b23b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72284db2-fe2e-3801-8490-c7d6b8c15771 Address:72284db2-fe2e-3801-8490-c7d6b8c15771} {Suffrage:Voter ID:server-7d247416-3214-8134-dd1a-65dc0c50b23b Address:7d247416-3214-8134-dd1a-65dc0c50b23b} {Suffrage:Voter ID:server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b Address:cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}]"
2025-10-30T18:17:40.409Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering follower state: follower="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.409Z [INFO]  server-7d247416-3214-8134-dd1a-65dc0c50b23b: entering follower state: follower="Node at 7d247416-3214-8134-dd1a-65dc0c50b23b [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.409Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72284db2-fe2e-3801-8490-c7d6b8c15771 Address:72284db2-fe2e-3801-8490-c7d6b8c15771} {Suffrage:Voter ID:server-7d247416-3214-8134-dd1a-65dc0c50b23b Address:7d247416-3214-8134-dd1a-65dc0c50b23b} {Suffrage:Voter ID:server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b Address:cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}]"
2025-10-30T18:17:40.409Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:40.409Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering follower state: follower="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.492Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:40.492Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=2
2025-10-30T18:17:40.492Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: election won: term=2 tally=2
2025-10-30T18:17:40.492Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering leader state: leader="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Leader]"
2025-10-30T18:17:40.492Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: added peer, starting replication: peer=server-72284db2-fe2e-3801-8490-c7d6b8c15771
2025-10-30T18:17:40.492Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: added peer, starting replication: peer=server-7d247416-3214-8134-dd1a-65dc0c50b23b
2025-10-30T18:17:40.492Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: pipelining replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}"
2025-10-30T18:17:40.492Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: pipelining replication: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}"
2025-10-30T18:17:40.598Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:40.492584101 +0000 UTC m=+6.771258010 (2 nodes), 82.806213ms from start of poll, 83.078196ms from cluster start. Timeout at 2025-10-30 18:17:40.597587434 +0000 UTC m=+6.876261332, 105.003322ms after stability
2025-10-30T18:17:40.598Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:40.703Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:40.598126151 +0000 UTC m=+6.876800061 (1 nodes), 1.942µs from start of poll, 188.620247ms from cluster start. Timeout at 2025-10-30 18:17:40.703131699 +0000 UTC m=+6.981805599, 105.005538ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Leader]
2025-10-30T18:17:40.723Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to pipeline appendEntries: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="append pipeline closed"
2025-10-30T18:17:40.723Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: aborting pipeline replication: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}"
2025-10-30T18:17:40.723Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to pipeline appendEntries: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="append pipeline closed"
2025-10-30T18:17:40.723Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: aborting pipeline replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}"
2025-10-30T18:17:40.724Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=10ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.724Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:40.727Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=10ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.731Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.733Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.743Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=10ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.744Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=10ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.751Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.752Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.764Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=10ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.764Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=10ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.765Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact: server-id=server-7d247416-3214-8134-dd1a-65dc0c50b23b time=50.65026ms
2025-10-30T18:17:40.768Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.768Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.776Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact: server-id=server-72284db2-fe2e-3801-8490-c7d6b8c15771 time=58.507745ms
2025-10-30T18:17:40.776Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact: server-id=server-7d247416-3214-8134-dd1a-65dc0c50b23b time=61.601075ms
2025-10-30T18:17:40.776Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact quorum of nodes, stepping down
2025-10-30T18:17:40.776Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering follower state: follower="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.781Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=20ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.783Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=20ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.795Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771"
2025-10-30T18:17:40.795Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b"
2025-10-30T18:17:40.808Z [WARN]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: heartbeat timeout reached, starting election: last-leader-addr=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b last-leader-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b
2025-10-30T18:17:40.808Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering candidate state: node="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Candidate]" term=3
2025-10-30T18:17:40.808Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to make requestVote RPC: target="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" term=3
2025-10-30T18:17:40.808Z [WARN]  server-7d247416-3214-8134-dd1a-65dc0c50b23b: rejecting vote request since we have a leader: from=72284db2-fe2e-3801-8490-c7d6b8c15771 leader=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b leader-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b
2025-10-30T18:17:40.834Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:40.834Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=3
2025-10-30T18:17:40.834Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" term=3
2025-10-30T18:17:40.834Z [WARN]  server-7d247416-3214-8134-dd1a-65dc0c50b23b: heartbeat timeout reached, starting election: last-leader-addr=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b last-leader-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b
2025-10-30T18:17:40.834Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" term=3
2025-10-30T18:17:40.834Z [INFO]  server-7d247416-3214-8134-dd1a-65dc0c50b23b: entering candidate state: node="Node at 7d247416-3214-8134-dd1a-65dc0c50b23b [Candidate]" term=3
2025-10-30T18:17:40.834Z [ERROR] server-7d247416-3214-8134-dd1a-65dc0c50b23b: failed to make requestVote RPC: target="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" term=3
2025-10-30T18:17:40.834Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: duplicate requestVote for same term: term=3
2025-10-30T18:17:40.896Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: Election timeout reached, restarting election
2025-10-30T18:17:40.896Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=4
2025-10-30T18:17:40.896Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" term=4
2025-10-30T18:17:40.896Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" term=4
2025-10-30T18:17:40.907Z [WARN]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: Election timeout reached, restarting election
2025-10-30T18:17:40.907Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering candidate state: node="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Candidate]" term=4
2025-10-30T18:17:40.907Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to make requestVote RPC: target="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" term=4
2025-10-30T18:17:40.907Z [INFO]  server-7d247416-3214-8134-dd1a-65dc0c50b23b: entering follower state: follower="Node at 7d247416-3214-8134-dd1a-65dc0c50b23b [Follower]" leader-address= leader-id=
2025-10-30T18:17:40.907Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: election won: term=4 tally=2
2025-10-30T18:17:40.907Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering leader state: leader="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Leader]"
2025-10-30T18:17:40.907Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-7d247416-3214-8134-dd1a-65dc0c50b23b
2025-10-30T18:17:40.907Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b
2025-10-30T18:17:40.907Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.907Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: pipelining replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}"
2025-10-30T18:17:40.914Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=10ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.924Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.933Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=10ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.943Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.953Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=10ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.957Z [WARN]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to contact: server-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b time=50.173984ms
2025-10-30T18:17:40.966Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: Election timeout reached, restarting election
2025-10-30T18:17:40.966Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=5
2025-10-30T18:17:40.966Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" term=5
2025-10-30T18:17:40.966Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" term=5
2025-10-30T18:17:40.970Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=20ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.974Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:40.996Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=25ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b"
2025-10-30T18:17:41.006Z [WARN]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to contact: server-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b time=99.043092ms
2025-10-30T18:17:41.013Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:40.907601833 +0000 UTC m=+7.186275730 (1 nodes), 182.800078ms from start of poll, 498.095916ms from cluster start. Timeout at 2025-10-30 18:17:41.01260239 +0000 UTC m=+7.291276288, 105.000558ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]
2025-10-30T18:17:41.022Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: peer has newer term, stopping replication: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}"
2025-10-30T18:17:41.022Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering follower state: follower="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.022Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: aborting pipeline replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}"
2025-10-30T18:17:41.066Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: Election timeout reached, restarting election
2025-10-30T18:17:41.066Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=6
2025-10-30T18:17:41.066Z [WARN]  server-7d247416-3214-8134-dd1a-65dc0c50b23b: rejecting vote request since we have a leader: from=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b leader=72284db2-fe2e-3801-8490-c7d6b8c15771 leader-id=server-72284db2-fe2e-3801-8490-c7d6b8c15771
2025-10-30T18:17:41.067Z [WARN]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: rejecting vote request since our last term is greater: candidate=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b last-term=4 last-candidate-term=2
2025-10-30T18:17:41.091Z [WARN]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:41.091Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering candidate state: node="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Candidate]" term=7
2025-10-30T18:17:41.091Z [INFO]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering follower state: follower="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.091Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: election won: term=7 tally=2
2025-10-30T18:17:41.091Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering leader state: leader="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Leader]"
2025-10-30T18:17:41.091Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-7d247416-3214-8134-dd1a-65dc0c50b23b
2025-10-30T18:17:41.091Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b
2025-10-30T18:17:41.092Z [WARN]  server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to get previous log: previous-index=5 last-index=3 error="log not found"
2025-10-30T18:17:41.092Z [WARN]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: appendEntries rejected, sending older logs: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" next=4
2025-10-30T18:17:41.092Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: pipelining replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}"
2025-10-30T18:17:41.092Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: pipelining replication: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}"
2025-10-30T18:17:41.097Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: aborting pipeline replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}"
2025-10-30T18:17:41.097Z [INFO]  server-72284db2-fe2e-3801-8490-c7d6b8c15771: aborting pipeline replication: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}"
--- PASS: TestRaft_LeaderFail (0.69s)
=== RUN   TestRaft_BehindFollower
2025-10-30T18:17:41.098Z [INFO]  server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 Address:821f7b28-91bf-3de7-6f56-2f94f9ddf0e5} {Suffrage:Voter ID:server-9ffd1b46-b571-6cad-22bc-9fb55c496434 Address:9ffd1b46-b571-6cad-22bc-9fb55c496434} {Suffrage:Voter ID:server-1e38d4d5-371c-cbd6-430d-abb6f87207aa Address:1e38d4d5-371c-cbd6-430d-abb6f87207aa}]"
2025-10-30T18:17:41.098Z [INFO]  server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5: entering follower state: follower="Node at 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.098Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 Address:821f7b28-91bf-3de7-6f56-2f94f9ddf0e5} {Suffrage:Voter ID:server-9ffd1b46-b571-6cad-22bc-9fb55c496434 Address:9ffd1b46-b571-6cad-22bc-9fb55c496434} {Suffrage:Voter ID:server-1e38d4d5-371c-cbd6-430d-abb6f87207aa Address:1e38d4d5-371c-cbd6-430d-abb6f87207aa}]"
2025-10-30T18:17:41.098Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: entering follower state: follower="Node at 9ffd1b46-b571-6cad-22bc-9fb55c496434 [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.098Z [INFO]  server-1e38d4d5-371c-cbd6-430d-abb6f87207aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 Address:821f7b28-91bf-3de7-6f56-2f94f9ddf0e5} {Suffrage:Voter ID:server-9ffd1b46-b571-6cad-22bc-9fb55c496434 Address:9ffd1b46-b571-6cad-22bc-9fb55c496434} {Suffrage:Voter ID:server-1e38d4d5-371c-cbd6-430d-abb6f87207aa Address:1e38d4d5-371c-cbd6-430d-abb6f87207aa}]"
2025-10-30T18:17:41.098Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:41.098Z [INFO]  server-1e38d4d5-371c-cbd6-430d-abb6f87207aa: entering follower state: follower="Node at 1e38d4d5-371c-cbd6-430d-abb6f87207aa [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.151Z [WARN]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:41.151Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: entering candidate state: node="Node at 9ffd1b46-b571-6cad-22bc-9fb55c496434 [Candidate]" term=2
2025-10-30T18:17:41.151Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: election won: term=2 tally=2
2025-10-30T18:17:41.151Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: entering leader state: leader="Node at 9ffd1b46-b571-6cad-22bc-9fb55c496434 [Leader]"
2025-10-30T18:17:41.151Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: added peer, starting replication: peer=server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5
2025-10-30T18:17:41.151Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: added peer, starting replication: peer=server-1e38d4d5-371c-cbd6-430d-abb6f87207aa
2025-10-30T18:17:41.151Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: pipelining replication: peer="{Voter server-1e38d4d5-371c-cbd6-430d-abb6f87207aa 1e38d4d5-371c-cbd6-430d-abb6f87207aa}"
2025-10-30T18:17:41.151Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: pipelining replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}"
2025-10-30T18:17:41.257Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:41.151757967 +0000 UTC m=+7.430431864 (1 nodes), 53.178225ms from start of poll, 53.491248ms from cluster start. Timeout at 2025-10-30 18:17:41.256760661 +0000 UTC m=+7.535434559, 105.002695ms after stability
2025-10-30T18:17:41.257Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:41.363Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:41.257735638 +0000 UTC m=+7.536409537 (2 nodes), 1.464µs from start of poll, 159.468921ms from cluster start. Timeout at 2025-10-30 18:17:41.362745368 +0000 UTC m=+7.641419336, 105.009799ms after stability
2025-10-30T18:17:41.363Z [ERROR] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: failed to pipeline appendEntries: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" error="append pipeline closed"
2025-10-30T18:17:41.363Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: aborting pipeline replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}"
2025-10-30T18:17:41.363Z [ERROR] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: failed to appendEntries to: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" error="failed to connect to peer: 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5"
2025-10-30T18:17:41.365Z [ERROR] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: failed to heartbeat to: peer=821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 backoff time=10ms error="failed to connect to peer: 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-10-30T18:17:41.373Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: pipelining replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}"
2025-10-30T18:17:41.384Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:41.490Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:41.384686542 +0000 UTC m=+7.663360439 (1 nodes), 542ns from start of poll, 286.419823ms from cluster start. Timeout at 2025-10-30 18:17:41.489690042 +0000 UTC m=+7.768363939, 105.0035ms after stability
2025-10-30T18:17:41.490Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: aborting pipeline replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}"
2025-10-30T18:17:41.490Z [INFO]  server-9ffd1b46-b571-6cad-22bc-9fb55c496434: aborting pipeline replication: peer="{Voter server-1e38d4d5-371c-cbd6-430d-abb6f87207aa 1e38d4d5-371c-cbd6-430d-abb6f87207aa}"
--- PASS: TestRaft_BehindFollower (0.39s)
=== RUN   TestRaft_ApplyNonLeader
2025-10-30T18:17:41.491Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-182d8643-b95e-ea6e-5c2c-fc19155a90ad Address:182d8643-b95e-ea6e-5c2c-fc19155a90ad} {Suffrage:Voter ID:server-d16cfb49-4715-64c3-a972-38aabb975406 Address:d16cfb49-4715-64c3-a972-38aabb975406} {Suffrage:Voter ID:server-4be31e34-d481-0c47-a717-9820c6d48a3e Address:4be31e34-d481-0c47-a717-9820c6d48a3e}]"
2025-10-30T18:17:41.491Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: entering follower state: follower="Node at 182d8643-b95e-ea6e-5c2c-fc19155a90ad [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.491Z [INFO]  server-d16cfb49-4715-64c3-a972-38aabb975406: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-182d8643-b95e-ea6e-5c2c-fc19155a90ad Address:182d8643-b95e-ea6e-5c2c-fc19155a90ad} {Suffrage:Voter ID:server-d16cfb49-4715-64c3-a972-38aabb975406 Address:d16cfb49-4715-64c3-a972-38aabb975406} {Suffrage:Voter ID:server-4be31e34-d481-0c47-a717-9820c6d48a3e Address:4be31e34-d481-0c47-a717-9820c6d48a3e}]"
2025-10-30T18:17:41.491Z [INFO]  server-d16cfb49-4715-64c3-a972-38aabb975406: entering follower state: follower="Node at d16cfb49-4715-64c3-a972-38aabb975406 [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.491Z [INFO]  server-4be31e34-d481-0c47-a717-9820c6d48a3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-182d8643-b95e-ea6e-5c2c-fc19155a90ad Address:182d8643-b95e-ea6e-5c2c-fc19155a90ad} {Suffrage:Voter ID:server-d16cfb49-4715-64c3-a972-38aabb975406 Address:d16cfb49-4715-64c3-a972-38aabb975406} {Suffrage:Voter ID:server-4be31e34-d481-0c47-a717-9820c6d48a3e Address:4be31e34-d481-0c47-a717-9820c6d48a3e}]"
2025-10-30T18:17:41.491Z [INFO]  server-4be31e34-d481-0c47-a717-9820c6d48a3e: entering follower state: follower="Node at 4be31e34-d481-0c47-a717-9820c6d48a3e [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.491Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:41.569Z [WARN]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:41.569Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: entering candidate state: node="Node at 182d8643-b95e-ea6e-5c2c-fc19155a90ad [Candidate]" term=2
2025-10-30T18:17:41.569Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: election won: term=2 tally=2
2025-10-30T18:17:41.569Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: entering leader state: leader="Node at 182d8643-b95e-ea6e-5c2c-fc19155a90ad [Leader]"
2025-10-30T18:17:41.569Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: added peer, starting replication: peer=server-d16cfb49-4715-64c3-a972-38aabb975406
2025-10-30T18:17:41.569Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: added peer, starting replication: peer=server-4be31e34-d481-0c47-a717-9820c6d48a3e
2025-10-30T18:17:41.569Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: pipelining replication: peer="{Voter server-4be31e34-d481-0c47-a717-9820c6d48a3e 4be31e34-d481-0c47-a717-9820c6d48a3e}"
2025-10-30T18:17:41.570Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: pipelining replication: peer="{Voter server-d16cfb49-4715-64c3-a972-38aabb975406 d16cfb49-4715-64c3-a972-38aabb975406}"
2025-10-30T18:17:41.675Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:41.56996994 +0000 UTC m=+7.848643840 (1 nodes), 78.195768ms from start of poll, 78.99977ms from cluster start. Timeout at 2025-10-30 18:17:41.674972497 +0000 UTC m=+7.953646394, 105.002554ms after stability
2025-10-30T18:17:41.675Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:41.780Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:41.675234975 +0000 UTC m=+7.953908876 (2 nodes), 852ns from start of poll, 184.264806ms from cluster start. Timeout at 2025-10-30 18:17:41.780239191 +0000 UTC m=+8.058913104, 105.004228ms after stability
2025-10-30T18:17:41.780Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: aborting pipeline replication: peer="{Voter server-4be31e34-d481-0c47-a717-9820c6d48a3e 4be31e34-d481-0c47-a717-9820c6d48a3e}"
2025-10-30T18:17:41.780Z [INFO]  server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: aborting pipeline replication: peer="{Voter server-d16cfb49-4715-64c3-a972-38aabb975406 d16cfb49-4715-64c3-a972-38aabb975406}"
--- PASS: TestRaft_ApplyNonLeader (0.29s)
=== RUN   TestRaft_ApplyConcurrent
2025-10-30T18:17:41.781Z [INFO]  server-66edc0d3-3d91-9f9d-750e-6a98d61cf093: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 Address:66edc0d3-3d91-9f9d-750e-6a98d61cf093} {Suffrage:Voter ID:server-ead9fc6b-97cc-9600-d687-c58362a702f9 Address:ead9fc6b-97cc-9600-d687-c58362a702f9} {Suffrage:Voter ID:server-a40ee663-de6f-2f30-67c9-70ff84a126d2 Address:a40ee663-de6f-2f30-67c9-70ff84a126d2}]"
2025-10-30T18:17:41.781Z [INFO]  server-66edc0d3-3d91-9f9d-750e-6a98d61cf093: entering follower state: follower="Node at 66edc0d3-3d91-9f9d-750e-6a98d61cf093 [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.781Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 Address:66edc0d3-3d91-9f9d-750e-6a98d61cf093} {Suffrage:Voter ID:server-ead9fc6b-97cc-9600-d687-c58362a702f9 Address:ead9fc6b-97cc-9600-d687-c58362a702f9} {Suffrage:Voter ID:server-a40ee663-de6f-2f30-67c9-70ff84a126d2 Address:a40ee663-de6f-2f30-67c9-70ff84a126d2}]"
2025-10-30T18:17:41.781Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: entering follower state: follower="Node at ead9fc6b-97cc-9600-d687-c58362a702f9 [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.781Z [INFO]  server-a40ee663-de6f-2f30-67c9-70ff84a126d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 Address:66edc0d3-3d91-9f9d-750e-6a98d61cf093} {Suffrage:Voter ID:server-ead9fc6b-97cc-9600-d687-c58362a702f9 Address:ead9fc6b-97cc-9600-d687-c58362a702f9} {Suffrage:Voter ID:server-a40ee663-de6f-2f30-67c9-70ff84a126d2 Address:a40ee663-de6f-2f30-67c9-70ff84a126d2}]"
2025-10-30T18:17:41.781Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:41.781Z [INFO]  server-a40ee663-de6f-2f30-67c9-70ff84a126d2: entering follower state: follower="Node at a40ee663-de6f-2f30-67c9-70ff84a126d2 [Follower]" leader-address= leader-id=
2025-10-30T18:17:41.908Z [WARN]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:41.908Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: entering candidate state: node="Node at ead9fc6b-97cc-9600-d687-c58362a702f9 [Candidate]" term=2
2025-10-30T18:17:41.908Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: election won: term=2 tally=2
2025-10-30T18:17:41.908Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: entering leader state: leader="Node at ead9fc6b-97cc-9600-d687-c58362a702f9 [Leader]"
2025-10-30T18:17:41.908Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: added peer, starting replication: peer=server-66edc0d3-3d91-9f9d-750e-6a98d61cf093
2025-10-30T18:17:41.908Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: added peer, starting replication: peer=server-a40ee663-de6f-2f30-67c9-70ff84a126d2
2025-10-30T18:17:41.908Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: pipelining replication: peer="{Voter server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 66edc0d3-3d91-9f9d-750e-6a98d61cf093}"
2025-10-30T18:17:41.908Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: pipelining replication: peer="{Voter server-a40ee663-de6f-2f30-67c9-70ff84a126d2 a40ee663-de6f-2f30-67c9-70ff84a126d2}"
2025-10-30T18:17:42.114Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:41.908810143 +0000 UTC m=+8.187484041 (1 nodes), 127.386697ms from start of poll, 127.69849ms from cluster start. Timeout at 2025-10-30 18:17:42.113810582 +0000 UTC m=+8.392484479, 205.000438ms after stability
2025-10-30T18:17:42.128Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: aborting pipeline replication: peer="{Voter server-a40ee663-de6f-2f30-67c9-70ff84a126d2 a40ee663-de6f-2f30-67c9-70ff84a126d2}"
2025-10-30T18:17:42.128Z [INFO]  server-ead9fc6b-97cc-9600-d687-c58362a702f9: aborting pipeline replication: peer="{Voter server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 66edc0d3-3d91-9f9d-750e-6a98d61cf093}"
--- PASS: TestRaft_ApplyConcurrent (0.35s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-10-30T18:17:42.129Z [INFO]  server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 Address:4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1}]"
2025-10-30T18:17:42.129Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:42.129Z [INFO]  server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: entering follower state: follower="Node at 4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 [Follower]" leader-address= leader-id=
2025-10-30T18:17:42.233Z [WARN]  server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:42.233Z [INFO]  server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: entering candidate state: node="Node at 4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 [Candidate]" term=2
2025-10-30T18:17:42.233Z [INFO]  server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: election won: term=2 tally=1
2025-10-30T18:17:42.233Z [INFO]  server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: entering leader state: leader="Node at 4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 [Leader]"
2025-10-30T18:17:42.435Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:42.2333469 +0000 UTC m=+8.512020810 (1 nodes), 103.997156ms from start of poll, 104.439594ms from cluster start. Timeout at 2025-10-30 18:17:42.434351306 +0000 UTC m=+8.713025204, 201.004394ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.31s)
=== RUN   TestRaft_JoinNode
2025-10-30T18:17:42.436Z [INFO]  server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee Address:6dc082e1-6ada-515e-6f4f-106b5c56e7ee} {Suffrage:Voter ID:server-74eaed57-0880-317d-24fa-046bae12554c Address:74eaed57-0880-317d-24fa-046bae12554c}]"
2025-10-30T18:17:42.436Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee Address:6dc082e1-6ada-515e-6f4f-106b5c56e7ee} {Suffrage:Voter ID:server-74eaed57-0880-317d-24fa-046bae12554c Address:74eaed57-0880-317d-24fa-046bae12554c}]"
2025-10-30T18:17:42.436Z [INFO]  server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee: entering follower state: follower="Node at 6dc082e1-6ada-515e-6f4f-106b5c56e7ee [Follower]" leader-address= leader-id=
2025-10-30T18:17:42.436Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: entering follower state: follower="Node at 74eaed57-0880-317d-24fa-046bae12554c [Follower]" leader-address= leader-id=
2025-10-30T18:17:42.437Z [INFO]  server-e40d87c6-80a4-6509-a3fa-850f5b180f88: initial configuration: index=0 servers=[]
2025-10-30T18:17:42.437Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:42.437Z [INFO]  server-e40d87c6-80a4-6509-a3fa-850f5b180f88: entering follower state: follower="Node at e40d87c6-80a4-6509-a3fa-850f5b180f88 [Follower]" leader-address= leader-id=
2025-10-30T18:17:42.490Z [WARN]  server-74eaed57-0880-317d-24fa-046bae12554c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:42.490Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: entering candidate state: node="Node at 74eaed57-0880-317d-24fa-046bae12554c [Candidate]" term=2
2025-10-30T18:17:42.490Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: election won: term=2 tally=2
2025-10-30T18:17:42.490Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: entering leader state: leader="Node at 74eaed57-0880-317d-24fa-046bae12554c [Leader]"
2025-10-30T18:17:42.490Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: added peer, starting replication: peer=server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee
2025-10-30T18:17:42.490Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: pipelining replication: peer="{Voter server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee 6dc082e1-6ada-515e-6f4f-106b5c56e7ee}"
2025-10-30T18:17:42.518Z [WARN]  server-e40d87c6-80a4-6509-a3fa-850f5b180f88: no known peers, aborting election
2025-10-30T18:17:42.596Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:42.49044263 +0000 UTC m=+8.769116538 (1 nodes), 53.274818ms from start of poll, 53.686432ms from cluster start. Timeout at 2025-10-30 18:17:42.595444536 +0000 UTC m=+8.874118434, 105.001896ms after stability
2025-10-30T18:17:42.596Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: updating configuration: command=AddVoter server-id=server-e40d87c6-80a4-6509-a3fa-850f5b180f88 server-addr=e40d87c6-80a4-6509-a3fa-850f5b180f88 servers="[{Suffrage:Voter ID:server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee Address:6dc082e1-6ada-515e-6f4f-106b5c56e7ee} {Suffrage:Voter ID:server-74eaed57-0880-317d-24fa-046bae12554c Address:74eaed57-0880-317d-24fa-046bae12554c} {Suffrage:Voter ID:server-e40d87c6-80a4-6509-a3fa-850f5b180f88 Address:e40d87c6-80a4-6509-a3fa-850f5b180f88}]"
2025-10-30T18:17:42.596Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: added peer, starting replication: peer=server-e40d87c6-80a4-6509-a3fa-850f5b180f88
2025-10-30T18:17:42.596Z [WARN]  server-e40d87c6-80a4-6509-a3fa-850f5b180f88: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-10-30T18:17:42.596Z [WARN]  server-74eaed57-0880-317d-24fa-046bae12554c: appendEntries rejected, sending older logs: peer="{Voter server-e40d87c6-80a4-6509-a3fa-850f5b180f88 e40d87c6-80a4-6509-a3fa-850f5b180f88}" next=1
2025-10-30T18:17:42.596Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: pipelining replication: peer="{Voter server-e40d87c6-80a4-6509-a3fa-850f5b180f88 e40d87c6-80a4-6509-a3fa-850f5b180f88}"
2025-10-30T18:17:42.596Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:42.701Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:42.596440668 +0000 UTC m=+8.875114566 (1 nodes), 571ns from start of poll, 159.68446ms from cluster start. Timeout at 2025-10-30 18:17:42.701448847 +0000 UTC m=+8.980122761, 105.008195ms after stability
2025-10-30T18:17:42.701Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: aborting pipeline replication: peer="{Voter server-e40d87c6-80a4-6509-a3fa-850f5b180f88 e40d87c6-80a4-6509-a3fa-850f5b180f88}"
2025-10-30T18:17:42.701Z [INFO]  server-74eaed57-0880-317d-24fa-046bae12554c: aborting pipeline replication: peer="{Voter server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee 6dc082e1-6ada-515e-6f4f-106b5c56e7ee}"
--- PASS: TestRaft_JoinNode (0.27s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-10-30T18:17:42.702Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f36477ac-c939-3fe3-cad8-9249aacc4a99 Address:f36477ac-c939-3fe3-cad8-9249aacc4a99}]"
2025-10-30T18:17:42.702Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: entering follower state: follower="Node at f36477ac-c939-3fe3-cad8-9249aacc4a99 [Follower]" leader-address= leader-id=
2025-10-30T18:17:42.702Z [INFO]  server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: initial configuration: index=0 servers=[]
2025-10-30T18:17:42.702Z [INFO]  server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: entering follower state: follower="Node at dcc76a28-b151-9ba9-84f7-34704a47b6e4 [Follower]" leader-address= leader-id=
2025-10-30T18:17:42.702Z [INFO]  server-6eb2330a-bd33-58b7-4292-326c07b03977: initial configuration: index=0 servers=[]
2025-10-30T18:17:42.702Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:42.702Z [INFO]  server-6eb2330a-bd33-58b7-4292-326c07b03977: entering follower state: follower="Node at 6eb2330a-bd33-58b7-4292-326c07b03977 [Follower]" leader-address= leader-id=
2025-10-30T18:17:42.784Z [WARN]  server-6eb2330a-bd33-58b7-4292-326c07b03977: no known peers, aborting election
2025-10-30T18:17:42.785Z [WARN]  server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: no known peers, aborting election
2025-10-30T18:17:42.785Z [WARN]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:42.785Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: entering candidate state: node="Node at f36477ac-c939-3fe3-cad8-9249aacc4a99 [Candidate]" term=2
2025-10-30T18:17:42.785Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: election won: term=2 tally=1
2025-10-30T18:17:42.785Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: entering leader state: leader="Node at f36477ac-c939-3fe3-cad8-9249aacc4a99 [Leader]"
2025-10-30T18:17:42.891Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:42.785470408 +0000 UTC m=+9.064144320 (1 nodes), 82.482645ms from start of poll, 83.086063ms from cluster start. Timeout at 2025-10-30 18:17:42.890473913 +0000 UTC m=+9.169147813, 105.003493ms after stability
2025-10-30T18:17:42.891Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: updating configuration: command=AddVoter server-id=server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 server-addr=dcc76a28-b151-9ba9-84f7-34704a47b6e4 servers="[{Suffrage:Voter ID:server-f36477ac-c939-3fe3-cad8-9249aacc4a99 Address:f36477ac-c939-3fe3-cad8-9249aacc4a99} {Suffrage:Voter ID:server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 Address:dcc76a28-b151-9ba9-84f7-34704a47b6e4}]"
2025-10-30T18:17:42.891Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: added peer, starting replication: peer=server-dcc76a28-b151-9ba9-84f7-34704a47b6e4
2025-10-30T18:17:42.891Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:42.891Z [WARN]  server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-10-30T18:17:42.891Z [WARN]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: appendEntries rejected, sending older logs: peer="{Voter server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 dcc76a28-b151-9ba9-84f7-34704a47b6e4}" next=1
2025-10-30T18:17:42.891Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: pipelining replication: peer="{Voter server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 dcc76a28-b151-9ba9-84f7-34704a47b6e4}"
2025-10-30T18:17:42.997Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:42.891583551 +0000 UTC m=+9.170257449 (1 nodes), 792ns from start of poll, 189.199192ms from cluster start. Timeout at 2025-10-30 18:17:42.99659325 +0000 UTC m=+9.275267147, 105.009698ms after stability
2025-10-30T18:17:42.997Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: updating configuration: command=AddVoter server-id=server-6eb2330a-bd33-58b7-4292-326c07b03977 server-addr=6eb2330a-bd33-58b7-4292-326c07b03977 servers="[{Suffrage:Voter ID:server-f36477ac-c939-3fe3-cad8-9249aacc4a99 Address:f36477ac-c939-3fe3-cad8-9249aacc4a99} {Suffrage:Voter ID:server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 Address:dcc76a28-b151-9ba9-84f7-34704a47b6e4} {Suffrage:Voter ID:server-6eb2330a-bd33-58b7-4292-326c07b03977 Address:6eb2330a-bd33-58b7-4292-326c07b03977}]"
2025-10-30T18:17:42.997Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: added peer, starting replication: peer=server-6eb2330a-bd33-58b7-4292-326c07b03977
2025-10-30T18:17:42.997Z [WARN]  server-6eb2330a-bd33-58b7-4292-326c07b03977: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-10-30T18:17:42.997Z [WARN]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: appendEntries rejected, sending older logs: peer="{Voter server-6eb2330a-bd33-58b7-4292-326c07b03977 6eb2330a-bd33-58b7-4292-326c07b03977}" next=1
2025-10-30T18:17:42.997Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:42.998Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: pipelining replication: peer="{Voter server-6eb2330a-bd33-58b7-4292-326c07b03977 6eb2330a-bd33-58b7-4292-326c07b03977}"
2025-10-30T18:17:43.103Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:42.99801285 +0000 UTC m=+9.276686764 (1 nodes), 6.438µs from start of poll, 295.628507ms from cluster start. Timeout at 2025-10-30 18:17:43.103015289 +0000 UTC m=+9.381689186, 105.002422ms after stability
2025-10-30T18:17:43.103Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: aborting pipeline replication: peer="{Voter server-6eb2330a-bd33-58b7-4292-326c07b03977 6eb2330a-bd33-58b7-4292-326c07b03977}"
2025-10-30T18:17:43.103Z [INFO]  server-f36477ac-c939-3fe3-cad8-9249aacc4a99: aborting pipeline replication: peer="{Voter server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 dcc76a28-b151-9ba9-84f7-34704a47b6e4}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.40s)
=== RUN   TestRaft_RemoveFollower
2025-10-30T18:17:43.103Z [INFO]  server-ea27d9af-655f-75e1-d6d5-fd4a0529b188: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 Address:ea27d9af-655f-75e1-d6d5-fd4a0529b188} {Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]"
2025-10-30T18:17:43.104Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 Address:ea27d9af-655f-75e1-d6d5-fd4a0529b188} {Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]"
2025-10-30T18:17:43.104Z [INFO]  server-ea27d9af-655f-75e1-d6d5-fd4a0529b188: entering follower state: follower="Node at ea27d9af-655f-75e1-d6d5-fd4a0529b188 [Follower]" leader-address= leader-id=
2025-10-30T18:17:43.104Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: entering follower state: follower="Node at b08b8bcf-9ef1-57d3-c485-03d0dce82392 [Follower]" leader-address= leader-id=
2025-10-30T18:17:43.104Z [INFO]  server-dc43b1ef-6b4e-c802-b04d-f590d8336d33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 Address:ea27d9af-655f-75e1-d6d5-fd4a0529b188} {Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]"
2025-10-30T18:17:43.104Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:43.104Z [INFO]  server-dc43b1ef-6b4e-c802-b04d-f590d8336d33: entering follower state: follower="Node at dc43b1ef-6b4e-c802-b04d-f590d8336d33 [Follower]" leader-address= leader-id=
2025-10-30T18:17:43.156Z [WARN]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:43.156Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: entering candidate state: node="Node at b08b8bcf-9ef1-57d3-c485-03d0dce82392 [Candidate]" term=2
2025-10-30T18:17:43.157Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: election won: term=2 tally=2
2025-10-30T18:17:43.157Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: entering leader state: leader="Node at b08b8bcf-9ef1-57d3-c485-03d0dce82392 [Leader]"
2025-10-30T18:17:43.157Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: added peer, starting replication: peer=server-ea27d9af-655f-75e1-d6d5-fd4a0529b188
2025-10-30T18:17:43.157Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: added peer, starting replication: peer=server-dc43b1ef-6b4e-c802-b04d-f590d8336d33
2025-10-30T18:17:43.157Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: pipelining replication: peer="{Voter server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 ea27d9af-655f-75e1-d6d5-fd4a0529b188}"
2025-10-30T18:17:43.157Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: pipelining replication: peer="{Voter server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 dc43b1ef-6b4e-c802-b04d-f590d8336d33}"
2025-10-30T18:17:43.262Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:43.157435794 +0000 UTC m=+9.436109696 (1 nodes), 53.316198ms from start of poll, 53.570634ms from cluster start. Timeout at 2025-10-30 18:17:43.262439098 +0000 UTC m=+9.541112997, 105.003301ms after stability
2025-10-30T18:17:43.268Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:43.374Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:43.268233946 +0000 UTC m=+9.546907844 (2 nodes), 2.408µs from start of poll, 164.368782ms from cluster start. Timeout at 2025-10-30 18:17:43.373238298 +0000 UTC m=+9.651912197, 105.004353ms after stability
2025-10-30T18:17:43.374Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: updating configuration: command=RemoveServer server-id=server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 server-addr= servers="[{Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]"
2025-10-30T18:17:43.374Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: removed peer, stopping replication: peer=server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 last-index=3
2025-10-30T18:17:43.374Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: aborting pipeline replication: peer="{Voter server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 ea27d9af-655f-75e1-d6d5-fd4a0529b188}"
2025-10-30T18:17:43.480Z [INFO]  server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: aborting pipeline replication: peer="{Voter server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 dc43b1ef-6b4e-c802-b04d-f590d8336d33}"
--- PASS: TestRaft_RemoveFollower (0.38s)
=== RUN   TestRaft_RemoveLeader
2025-10-30T18:17:43.481Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-64e25359-002d-da89-15dd-c22b9a1aacbf Address:64e25359-002d-da89-15dd-c22b9a1aacbf} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]"
2025-10-30T18:17:43.481Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering follower state: follower="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Follower]" leader-address= leader-id=
2025-10-30T18:17:43.481Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-64e25359-002d-da89-15dd-c22b9a1aacbf Address:64e25359-002d-da89-15dd-c22b9a1aacbf} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]"
2025-10-30T18:17:43.481Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: entering follower state: follower="Node at 64e25359-002d-da89-15dd-c22b9a1aacbf [Follower]" leader-address= leader-id=
2025-10-30T18:17:43.481Z [INFO]  server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-64e25359-002d-da89-15dd-c22b9a1aacbf Address:64e25359-002d-da89-15dd-c22b9a1aacbf} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]"
2025-10-30T18:17:43.481Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:43.481Z [INFO]  server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: entering follower state: follower="Node at fed267d7-71aa-1f7c-0eea-ea089f2b92fa [Follower]" leader-address= leader-id=
2025-10-30T18:17:43.548Z [WARN]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:43.548Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: entering candidate state: node="Node at 64e25359-002d-da89-15dd-c22b9a1aacbf [Candidate]" term=2
2025-10-30T18:17:43.548Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: election won: term=2 tally=2
2025-10-30T18:17:43.548Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: entering leader state: leader="Node at 64e25359-002d-da89-15dd-c22b9a1aacbf [Leader]"
2025-10-30T18:17:43.548Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: added peer, starting replication: peer=server-15cef41c-b5cf-52ae-2fb6-8c893bd80901
2025-10-30T18:17:43.548Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: added peer, starting replication: peer=server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa
2025-10-30T18:17:43.548Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: pipelining replication: peer="{Voter server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 15cef41c-b5cf-52ae-2fb6-8c893bd80901}"
2025-10-30T18:17:43.548Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: pipelining replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}"
2025-10-30T18:17:43.654Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:43.548196502 +0000 UTC m=+9.826870399 (1 nodes), 66.463525ms from start of poll, 66.750341ms from cluster start. Timeout at 2025-10-30 18:17:43.653198081 +0000 UTC m=+9.931871982, 105.001583ms after stability
2025-10-30T18:17:43.659Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:43.765Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:43.659528239 +0000 UTC m=+9.938202149 (2 nodes), 2.258µs from start of poll, 178.082091ms from cluster start. Timeout at 2025-10-30 18:17:43.764532061 +0000 UTC m=+10.043205959, 105.00381ms after stability
2025-10-30T18:17:43.765Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: updating configuration: command=RemoveServer server-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf server-addr= servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]"
2025-10-30T18:17:43.765Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: removed ourself, shutting down
2025-10-30T18:17:43.765Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: aborting pipeline replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}"
2025-10-30T18:17:43.765Z [INFO]  server-64e25359-002d-da89-15dd-c22b9a1aacbf: aborting pipeline replication: peer="{Voter server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 15cef41c-b5cf-52ae-2fb6-8c893bd80901}"
2025-10-30T18:17:43.829Z [WARN]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: heartbeat timeout reached, starting election: last-leader-addr=64e25359-002d-da89-15dd-c22b9a1aacbf last-leader-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf
2025-10-30T18:17:43.829Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering candidate state: node="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Candidate]" term=3
2025-10-30T18:17:43.829Z [WARN]  server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: rejecting vote request since we have a leader: from=15cef41c-b5cf-52ae-2fb6-8c893bd80901 leader=64e25359-002d-da89-15dd-c22b9a1aacbf leader-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf
2025-10-30T18:17:43.885Z [WARN]  server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: heartbeat timeout reached, starting election: last-leader-addr=64e25359-002d-da89-15dd-c22b9a1aacbf last-leader-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf
2025-10-30T18:17:43.885Z [INFO]  server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: entering candidate state: node="Node at fed267d7-71aa-1f7c-0eea-ea089f2b92fa [Candidate]" term=3
2025-10-30T18:17:43.885Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: duplicate requestVote for same term: term=3
2025-10-30T18:17:43.925Z [WARN]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: Election timeout reached, restarting election
2025-10-30T18:17:43.925Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering candidate state: node="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Candidate]" term=4
2025-10-30T18:17:43.925Z [INFO]  server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: entering follower state: follower="Node at fed267d7-71aa-1f7c-0eea-ea089f2b92fa [Follower]" leader-address= leader-id=
2025-10-30T18:17:43.926Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: election won: term=4 tally=2
2025-10-30T18:17:43.926Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering leader state: leader="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Leader]"
2025-10-30T18:17:43.926Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: added peer, starting replication: peer=server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa
2025-10-30T18:17:43.926Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: pipelining replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}"
2025-10-30T18:17:43.977Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:44.082Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:43.977215896 +0000 UTC m=+10.255889805 (1 nodes), 34.111µs from start of poll, 495.769747ms from cluster start. Timeout at 2025-10-30 18:17:44.082219034 +0000 UTC m=+10.360892933, 105.003128ms after stability
2025-10-30T18:17:44.082Z [INFO]  server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: aborting pipeline replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}"
--- PASS: TestRaft_RemoveLeader (0.60s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-10-30T18:17:44.083Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8} {Suffrage:Voter ID:server-c1e1d9ec-e29f-660c-69de-e065a1e69537 Address:c1e1d9ec-e29f-660c-69de-e065a1e69537}]"
2025-10-30T18:17:44.083Z [INFO]  server-9b684f15-58f8-aff8-f15b-c87a818f13a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8} {Suffrage:Voter ID:server-c1e1d9ec-e29f-660c-69de-e065a1e69537 Address:c1e1d9ec-e29f-660c-69de-e065a1e69537}]"
2025-10-30T18:17:44.083Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: entering follower state: follower="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.083Z [INFO]  server-9b684f15-58f8-aff8-f15b-c87a818f13a8: entering follower state: follower="Node at 9b684f15-58f8-aff8-f15b-c87a818f13a8 [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.083Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8} {Suffrage:Voter ID:server-c1e1d9ec-e29f-660c-69de-e065a1e69537 Address:c1e1d9ec-e29f-660c-69de-e065a1e69537}]"
2025-10-30T18:17:44.083Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:44.083Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering follower state: follower="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.149Z [WARN]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:44.149Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering candidate state: node="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Candidate]" term=2
2025-10-30T18:17:44.149Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: election won: term=2 tally=2
2025-10-30T18:17:44.149Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering leader state: leader="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Leader]"
2025-10-30T18:17:44.149Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: added peer, starting replication: peer=server-f267be62-7adb-2a57-19e0-e546a6611cab
2025-10-30T18:17:44.149Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: added peer, starting replication: peer=server-9b684f15-58f8-aff8-f15b-c87a818f13a8
2025-10-30T18:17:44.149Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: pipelining replication: peer="{Voter server-f267be62-7adb-2a57-19e0-e546a6611cab f267be62-7adb-2a57-19e0-e546a6611cab}"
2025-10-30T18:17:44.149Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: pipelining replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}"
2025-10-30T18:17:44.255Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:44.149644525 +0000 UTC m=+10.428318424 (2 nodes), 65.844399ms from start of poll, 66.084214ms from cluster start. Timeout at 2025-10-30 18:17:44.254647951 +0000 UTC m=+10.533321859, 105.003435ms after stability
2025-10-30T18:17:44.255Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:44.360Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:44.255566222 +0000 UTC m=+10.534240120 (1 nodes), 2.137µs from start of poll, 172.00591ms from cluster start. Timeout at 2025-10-30 18:17:44.360571574 +0000 UTC m=+10.639245480, 105.00536ms after stability
2025-10-30T18:17:44.362Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: updating configuration: command=RemoveServer server-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537 server-addr= servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8}]"
2025-10-30T18:17:44.364Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: removed ourself, transitioning to follower
2025-10-30T18:17:44.364Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering follower state: follower="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.364Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: aborting pipeline replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}"
2025-10-30T18:17:44.364Z [INFO]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: aborting pipeline replication: peer="{Voter server-f267be62-7adb-2a57-19e0-e546a6611cab f267be62-7adb-2a57-19e0-e546a6611cab}"
2025-10-30T18:17:44.421Z [WARN]  server-9b684f15-58f8-aff8-f15b-c87a818f13a8: heartbeat timeout reached, starting election: last-leader-addr=c1e1d9ec-e29f-660c-69de-e065a1e69537 last-leader-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537
2025-10-30T18:17:44.421Z [INFO]  server-9b684f15-58f8-aff8-f15b-c87a818f13a8: entering candidate state: node="Node at 9b684f15-58f8-aff8-f15b-c87a818f13a8 [Candidate]" term=3
2025-10-30T18:17:44.421Z [WARN]  server-f267be62-7adb-2a57-19e0-e546a6611cab: rejecting vote request since we have a leader: from=9b684f15-58f8-aff8-f15b-c87a818f13a8 leader=c1e1d9ec-e29f-660c-69de-e065a1e69537 leader-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537
2025-10-30T18:17:44.429Z [WARN]  server-f267be62-7adb-2a57-19e0-e546a6611cab: heartbeat timeout reached, starting election: last-leader-addr=c1e1d9ec-e29f-660c-69de-e065a1e69537 last-leader-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537
2025-10-30T18:17:44.429Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: entering candidate state: node="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Candidate]" term=3
2025-10-30T18:17:44.429Z [INFO]  server-9b684f15-58f8-aff8-f15b-c87a818f13a8: duplicate requestVote for same term: term=3
2025-10-30T18:17:44.435Z [WARN]  server-c1e1d9ec-e29f-660c-69de-e065a1e69537: not part of stable configuration, aborting election
2025-10-30T18:17:44.470Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:44.494Z [WARN]  server-f267be62-7adb-2a57-19e0-e546a6611cab: Election timeout reached, restarting election
2025-10-30T18:17:44.494Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: entering candidate state: node="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Candidate]" term=4
2025-10-30T18:17:44.494Z [INFO]  server-9b684f15-58f8-aff8-f15b-c87a818f13a8: entering follower state: follower="Node at 9b684f15-58f8-aff8-f15b-c87a818f13a8 [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.494Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: election won: term=4 tally=2
2025-10-30T18:17:44.494Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: entering leader state: leader="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Leader]"
2025-10-30T18:17:44.494Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: added peer, starting replication: peer=server-9b684f15-58f8-aff8-f15b-c87a818f13a8
2025-10-30T18:17:44.494Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: pipelining replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}"
2025-10-30T18:17:44.600Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:44.494728577 +0000 UTC m=+10.773402484 (1 nodes), 24.30947ms from start of poll, 411.168274ms from cluster start. Timeout at 2025-10-30 18:17:44.599730528 +0000 UTC m=+10.878404426, 105.001942ms after stability
2025-10-30T18:17:44.705Z [INFO]  server-f267be62-7adb-2a57-19e0-e546a6611cab: aborting pipeline replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-10-30T18:17:44.706Z [INFO]  server-1b28831c-996a-2bea-0305-4c5109095eb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]"
2025-10-30T18:17:44.706Z [INFO]  server-1b28831c-996a-2bea-0305-4c5109095eb6: entering follower state: follower="Node at 1b28831c-996a-2bea-0305-4c5109095eb6 [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.706Z [INFO]  server-e0597101-cc35-a45e-cd75-9fd2031d50fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]"
2025-10-30T18:17:44.706Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]"
2025-10-30T18:17:44.706Z [INFO]  server-e0597101-cc35-a45e-cd75-9fd2031d50fa: entering follower state: follower="Node at e0597101-cc35-a45e-cd75-9fd2031d50fa [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.706Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: entering follower state: follower="Node at 49f68783-af41-9ea5-b459-040d669e337e [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.706Z [INFO]  server-a6720c93-5bd0-bcde-7597-50e4e7104721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]"
2025-10-30T18:17:44.706Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:44.706Z [INFO]  server-a6720c93-5bd0-bcde-7597-50e4e7104721: entering follower state: follower="Node at a6720c93-5bd0-bcde-7597-50e4e7104721 [Follower]" leader-address= leader-id=
2025-10-30T18:17:44.773Z [WARN]  server-49f68783-af41-9ea5-b459-040d669e337e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:44.773Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: entering candidate state: node="Node at 49f68783-af41-9ea5-b459-040d669e337e [Candidate]" term=2
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: election won: term=2 tally=3
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: entering leader state: leader="Node at 49f68783-af41-9ea5-b459-040d669e337e [Leader]"
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: added peer, starting replication: peer=server-1b28831c-996a-2bea-0305-4c5109095eb6
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: added peer, starting replication: peer=server-e0597101-cc35-a45e-cd75-9fd2031d50fa
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: added peer, starting replication: peer=server-a6720c93-5bd0-bcde-7597-50e4e7104721
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: pipelining replication: peer="{Voter server-a6720c93-5bd0-bcde-7597-50e4e7104721 a6720c93-5bd0-bcde-7597-50e4e7104721}"
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: pipelining replication: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}"
2025-10-30T18:17:44.774Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: pipelining replication: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}"
2025-10-30T18:17:44.879Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:44.774090643 +0000 UTC m=+11.052764552 (1 nodes), 67.460259ms from start of poll, 67.854274ms from cluster start. Timeout at 2025-10-30 18:17:44.879092561 +0000 UTC m=+11.157766458, 105.001906ms after stability
2025-10-30T18:17:44.985Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:45.090Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:44.985520123 +0000 UTC m=+11.264194020 (3 nodes), 2.491µs from start of poll, 279.283742ms from cluster start. Timeout at 2025-10-30 18:17:45.090526232 +0000 UTC m=+11.369200130, 105.00611ms after stability
2025-10-30T18:17:45.090Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: updating configuration: command=RemoveServer server-id=server-a6720c93-5bd0-bcde-7597-50e4e7104721 server-addr= servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e}]"
2025-10-30T18:17:45.090Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: removed peer, stopping replication: peer=server-a6720c93-5bd0-bcde-7597-50e4e7104721 last-index=3
2025-10-30T18:17:45.090Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to pipeline appendEntries: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="append pipeline closed"
2025-10-30T18:17:45.090Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to pipeline appendEntries: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="append pipeline closed"
2025-10-30T18:17:45.091Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: aborting pipeline replication: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}"
2025-10-30T18:17:45.090Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: aborting pipeline replication: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}"
2025-10-30T18:17:45.090Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: aborting pipeline replication: peer="{Voter server-a6720c93-5bd0-bcde-7597-50e4e7104721 a6720c93-5bd0-bcde-7597-50e4e7104721}"
2025-10-30T18:17:45.091Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=1b28831c-996a-2bea-0305-4c5109095eb6 backoff time=10ms error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6"
2025-10-30T18:17:45.097Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6"
2025-10-30T18:17:45.097Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa"
2025-10-30T18:17:45.099Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=e0597101-cc35-a45e-cd75-9fd2031d50fa backoff time=10ms error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa"
2025-10-30T18:17:45.109Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=1b28831c-996a-2bea-0305-4c5109095eb6 backoff time=10ms error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6"
2025-10-30T18:17:45.115Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6"
2025-10-30T18:17:45.116Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa"
2025-10-30T18:17:45.117Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=e0597101-cc35-a45e-cd75-9fd2031d50fa backoff time=10ms error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa"
2025-10-30T18:17:45.128Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=1b28831c-996a-2bea-0305-4c5109095eb6 backoff time=10ms error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6"
2025-10-30T18:17:45.135Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa"
2025-10-30T18:17:45.136Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=e0597101-cc35-a45e-cd75-9fd2031d50fa backoff time=10ms error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa"
2025-10-30T18:17:45.136Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6"
2025-10-30T18:17:45.136Z [WARN]  server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact: server-id=server-1b28831c-996a-2bea-0305-4c5109095eb6 time=50.09139ms
2025-10-30T18:17:45.140Z [WARN]  server-e0597101-cc35-a45e-cd75-9fd2031d50fa: heartbeat timeout reached, starting election: last-leader-addr=49f68783-af41-9ea5-b459-040d669e337e last-leader-id=server-49f68783-af41-9ea5-b459-040d669e337e
2025-10-30T18:17:45.140Z [INFO]  server-e0597101-cc35-a45e-cd75-9fd2031d50fa: entering candidate state: node="Node at e0597101-cc35-a45e-cd75-9fd2031d50fa [Candidate]" term=3
2025-10-30T18:17:45.140Z [ERROR] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: failed to make requestVote RPC: target="{Voter server-a6720c93-5bd0-bcde-7597-50e4e7104721 a6720c93-5bd0-bcde-7597-50e4e7104721}" error="failed to connect to peer: a6720c93-5bd0-bcde-7597-50e4e7104721" term=3
2025-10-30T18:17:45.140Z [WARN]  server-1b28831c-996a-2bea-0305-4c5109095eb6: rejecting vote request since we have a leader: from=e0597101-cc35-a45e-cd75-9fd2031d50fa leader=49f68783-af41-9ea5-b459-040d669e337e leader-id=server-49f68783-af41-9ea5-b459-040d669e337e
2025-10-30T18:17:45.140Z [ERROR] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: failed to make requestVote RPC: target="{Voter server-49f68783-af41-9ea5-b459-040d669e337e 49f68783-af41-9ea5-b459-040d669e337e}" error="failed to connect to peer: 49f68783-af41-9ea5-b459-040d669e337e" term=3
2025-10-30T18:17:45.147Z [WARN]  server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact: server-id=server-1b28831c-996a-2bea-0305-4c5109095eb6 time=60.953573ms
2025-10-30T18:17:45.147Z [WARN]  server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact: server-id=server-e0597101-cc35-a45e-cd75-9fd2031d50fa time=57.712392ms
2025-10-30T18:17:45.147Z [WARN]  server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact quorum of nodes, stepping down
2025-10-30T18:17:45.147Z [INFO]  server-49f68783-af41-9ea5-b459-040d669e337e: entering follower state: follower="Node at 49f68783-af41-9ea5-b459-040d669e337e [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.147Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6"
2025-10-30T18:17:45.147Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s)
=== RUN   TestRaft_AddKnownPeer
2025-10-30T18:17:45.148Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]"
2025-10-30T18:17:45.148Z [INFO]  server-204e12a7-76ed-210c-67d1-51083ffdd6ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]"
2025-10-30T18:17:45.148Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: entering follower state: follower="Node at e708b557-baff-aba1-8de0-8885b7cd651b [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.148Z [INFO]  server-5719da0f-81bf-36e5-0290-a9477b806595: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]"
2025-10-30T18:17:45.148Z [INFO]  server-204e12a7-76ed-210c-67d1-51083ffdd6ca: entering follower state: follower="Node at 204e12a7-76ed-210c-67d1-51083ffdd6ca [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.148Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:45.148Z [INFO]  server-5719da0f-81bf-36e5-0290-a9477b806595: entering follower state: follower="Node at 5719da0f-81bf-36e5-0290-a9477b806595 [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.207Z [WARN]  server-e708b557-baff-aba1-8de0-8885b7cd651b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:45.207Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: entering candidate state: node="Node at e708b557-baff-aba1-8de0-8885b7cd651b [Candidate]" term=2
2025-10-30T18:17:45.207Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: election won: term=2 tally=2
2025-10-30T18:17:45.207Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: entering leader state: leader="Node at e708b557-baff-aba1-8de0-8885b7cd651b [Leader]"
2025-10-30T18:17:45.207Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: added peer, starting replication: peer=server-204e12a7-76ed-210c-67d1-51083ffdd6ca
2025-10-30T18:17:45.207Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: added peer, starting replication: peer=server-5719da0f-81bf-36e5-0290-a9477b806595
2025-10-30T18:17:45.207Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: pipelining replication: peer="{Voter server-5719da0f-81bf-36e5-0290-a9477b806595 5719da0f-81bf-36e5-0290-a9477b806595}"
2025-10-30T18:17:45.207Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: pipelining replication: peer="{Voter server-204e12a7-76ed-210c-67d1-51083ffdd6ca 204e12a7-76ed-210c-67d1-51083ffdd6ca}"
2025-10-30T18:17:45.312Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:45.207375765 +0000 UTC m=+11.486049662 (1 nodes), 58.834501ms from start of poll, 59.124858ms from cluster start. Timeout at 2025-10-30 18:17:45.312378772 +0000 UTC m=+11.591052671, 105.003009ms after stability
2025-10-30T18:17:45.312Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:45.418Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:45.312582325 +0000 UTC m=+11.591256222 (2 nodes), 1.374µs from start of poll, 164.331418ms from cluster start. Timeout at 2025-10-30 18:17:45.41758945 +0000 UTC m=+11.696263359, 105.007137ms after stability
2025-10-30T18:17:45.418Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: updating configuration: command=AddVoter server-id=server-204e12a7-76ed-210c-67d1-51083ffdd6ca server-addr=204e12a7-76ed-210c-67d1-51083ffdd6ca servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]"
2025-10-30T18:17:45.418Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: aborting pipeline replication: peer="{Voter server-5719da0f-81bf-36e5-0290-a9477b806595 5719da0f-81bf-36e5-0290-a9477b806595}"
2025-10-30T18:17:45.418Z [INFO]  server-e708b557-baff-aba1-8de0-8885b7cd651b: aborting pipeline replication: peer="{Voter server-204e12a7-76ed-210c-67d1-51083ffdd6ca 204e12a7-76ed-210c-67d1-51083ffdd6ca}"
--- PASS: TestRaft_AddKnownPeer (0.27s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-10-30T18:17:45.419Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]"
2025-10-30T18:17:45.419Z [INFO]  server-a46ee619-979e-32b9-c494-a88bb99679cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]"
2025-10-30T18:17:45.419Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: entering follower state: follower="Node at 3a9fa270-dec9-b506-4b15-4b05860e2ef2 [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.419Z [INFO]  server-af67b78c-1d7c-0107-49fd-38b7e0560803: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]"
2025-10-30T18:17:45.419Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:45.419Z [INFO]  server-a46ee619-979e-32b9-c494-a88bb99679cf: entering follower state: follower="Node at a46ee619-979e-32b9-c494-a88bb99679cf [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.419Z [INFO]  server-af67b78c-1d7c-0107-49fd-38b7e0560803: entering follower state: follower="Node at af67b78c-1d7c-0107-49fd-38b7e0560803 [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.471Z [WARN]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:45.471Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: entering candidate state: node="Node at 3a9fa270-dec9-b506-4b15-4b05860e2ef2 [Candidate]" term=2
2025-10-30T18:17:45.471Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: election won: term=2 tally=2
2025-10-30T18:17:45.471Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: entering leader state: leader="Node at 3a9fa270-dec9-b506-4b15-4b05860e2ef2 [Leader]"
2025-10-30T18:17:45.471Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: added peer, starting replication: peer=server-a46ee619-979e-32b9-c494-a88bb99679cf
2025-10-30T18:17:45.471Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: added peer, starting replication: peer=server-af67b78c-1d7c-0107-49fd-38b7e0560803
2025-10-30T18:17:45.472Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: pipelining replication: peer="{Voter server-af67b78c-1d7c-0107-49fd-38b7e0560803 af67b78c-1d7c-0107-49fd-38b7e0560803}"
2025-10-30T18:17:45.472Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: pipelining replication: peer="{Voter server-a46ee619-979e-32b9-c494-a88bb99679cf a46ee619-979e-32b9-c494-a88bb99679cf}"
2025-10-30T18:17:45.577Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:45.472033551 +0000 UTC m=+11.750707449 (1 nodes), 52.339269ms from start of poll, 52.558451ms from cluster start. Timeout at 2025-10-30 18:17:45.57703952 +0000 UTC m=+11.855713418, 105.005969ms after stability
2025-10-30T18:17:45.577Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: updating configuration: command=RemoveServer server-id=5ba0583a-0d91-ecfd-83ff-5acefca0c5c8 server-addr= servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]"
2025-10-30T18:17:45.577Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: aborting pipeline replication: peer="{Voter server-af67b78c-1d7c-0107-49fd-38b7e0560803 af67b78c-1d7c-0107-49fd-38b7e0560803}"
2025-10-30T18:17:45.577Z [INFO]  server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: aborting pipeline replication: peer="{Voter server-a46ee619-979e-32b9-c494-a88bb99679cf a46ee619-979e-32b9-c494-a88bb99679cf}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2025-10-30T18:17:45.579Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365 Address:baefbeb4-4e30-2a7b-5a74-7ce25eedb365}]"
2025-10-30T18:17:45.579Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:45.579Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: entering follower state: follower="Node at baefbeb4-4e30-2a7b-5a74-7ce25eedb365 [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.668Z [WARN]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:45.668Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: entering candidate state: node="Node at baefbeb4-4e30-2a7b-5a74-7ce25eedb365 [Candidate]" term=2
2025-10-30T18:17:45.668Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: election won: term=2 tally=1
2025-10-30T18:17:45.668Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: entering leader state: leader="Node at baefbeb4-4e30-2a7b-5a74-7ce25eedb365 [Leader]"
2025-10-30T18:17:45.774Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:45.668931776 +0000 UTC m=+11.947605803 (1 nodes), 89.590456ms from start of poll, 90.746214ms from cluster start. Timeout at 2025-10-30 18:17:45.773933737 +0000 UTC m=+12.052607634, 105.001831ms after stability
2025-10-30T18:17:45.775Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: starting snapshot up to: index=102
2025-10-30T18:17:45.775Z [INFO]  creating new snapshot: path=/tmp/raft17063633/snapshots/2-102-1761848265775.tmp
2025-10-30T18:17:45.775Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: compacting logs: from=1 to=92
2025-10-30T18:17:45.775Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: snapshot complete up to: index=102
2025-10-30T18:17:45.776Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: starting restore from snapshot: id=2-102-1761848265775 last-index=102 last-term=2 size-in-bytes=693
2025-10-30T18:17:45.776Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: snapshot restore progress: id=2-102-1761848265775 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-10-30T18:17:45.776Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: restored from snapshot: id=2-102-1761848265775 last-index=102 last-term=2 size-in-bytes=693
2025-10-30T18:17:45.776Z [INFO]  server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365 Address:baefbeb4-4e30-2a7b-5a74-7ce25eedb365}]"
--- PASS: TestRaft_SnapshotRestore (0.20s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-10-30T18:17:45.778Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036 Address:d6f95dcb-2b71-8d5a-c485-cbf3a697f036}]"
2025-10-30T18:17:45.778Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:45.778Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering follower state: follower="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Follower]" leader-address= leader-id=
2025-10-30T18:17:45.833Z [WARN]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:45.833Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering candidate state: node="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Candidate]" term=2
2025-10-30T18:17:45.833Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: election won: term=2 tally=1
2025-10-30T18:17:45.833Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering leader state: leader="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Leader]"
2025-10-30T18:17:45.939Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:45.83384783 +0000 UTC m=+12.112521742 (1 nodes), 55.642154ms from start of poll, 55.732104ms from cluster start. Timeout at 2025-10-30 18:17:45.938850762 +0000 UTC m=+12.217524659, 105.002917ms after stability
2025-10-30T18:17:45.940Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: starting snapshot up to: index=102
2025-10-30T18:17:45.940Z [INFO]  creating new snapshot: path=/tmp/raft993325267/snapshots/2-102-1761848265940.tmp
2025-10-30T18:17:45.941Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: compacting logs: from=1 to=92
2025-10-30T18:17:45.941Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: snapshot complete up to: index=102
2025-10-30T18:17:45.941Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: starting restore from snapshot: id=2-102-1761848265940 last-index=102 last-term=2 size-in-bytes=693
2025-10-30T18:17:45.941Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: snapshot restore progress: id=2-102-1761848265940 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-10-30T18:17:45.941Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: restored from snapshot: id=2-102-1761848265940 last-index=102 last-term=2 size-in-bytes=693
2025-10-30T18:17:45.941Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036 Address:d6f95dcb-2b71-8d5a-c485-cbf3a697f036}]"
2025-10-30T18:17:45.941Z [INFO]  server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering follower state: follower="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.16s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-10-30T18:17:45.942Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb Address:b8e6f099-ad7d-6443-c8c3-5ad28629b8bb}]"
2025-10-30T18:17:45.942Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:45.942Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: entering follower state: follower="Node at b8e6f099-ad7d-6443-c8c3-5ad28629b8bb [Follower]" leader-address= leader-id=
2025-10-30T18:17:46.021Z [WARN]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:46.021Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: entering candidate state: node="Node at b8e6f099-ad7d-6443-c8c3-5ad28629b8bb [Candidate]" term=2
2025-10-30T18:17:46.022Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: election won: term=2 tally=1
2025-10-30T18:17:46.022Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: entering leader state: leader="Node at b8e6f099-ad7d-6443-c8c3-5ad28629b8bb [Leader]"
2025-10-30T18:17:46.127Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:46.022019144 +0000 UTC m=+12.300693051 (1 nodes), 79.838574ms from start of poll, 79.926222ms from cluster start. Timeout at 2025-10-30 18:17:46.127023076 +0000 UTC m=+12.405696975, 105.003924ms after stability
2025-10-30T18:17:46.128Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: starting snapshot up to: index=102
2025-10-30T18:17:46.128Z [INFO]  creating new snapshot: path=/tmp/raft2914980949/snapshots/2-102-1761848266128.tmp
2025-10-30T18:17:46.128Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: compacting logs: from=1 to=92
2025-10-30T18:17:46.128Z [INFO]  server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.19s)
=== RUN   TestRaft_NoRestoreOnStart
2025-10-30T18:17:46.131Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e Address:19aae2d4-c32d-846d-4f26-0a1aa55ad75e}]"
2025-10-30T18:17:46.131Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:46.131Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering follower state: follower="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Follower]" leader-address= leader-id=
2025-10-30T18:17:46.214Z [WARN]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:46.214Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering candidate state: node="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Candidate]" term=2
2025-10-30T18:17:46.214Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: election won: term=2 tally=1
2025-10-30T18:17:46.214Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering leader state: leader="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Leader]"
2025-10-30T18:17:46.320Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:46.214361235 +0000 UTC m=+12.493035132 (1 nodes), 83.24442ms from start of poll, 83.377564ms from cluster start. Timeout at 2025-10-30 18:17:46.31936383 +0000 UTC m=+12.598037730, 105.002598ms after stability
2025-10-30T18:17:46.321Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: starting snapshot up to: index=102
2025-10-30T18:17:46.321Z [INFO]  creating new snapshot: path=/tmp/raft474268794/snapshots/2-102-1761848266321.tmp
2025-10-30T18:17:46.322Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: compacting logs: from=1 to=92
2025-10-30T18:17:46.322Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: snapshot complete up to: index=102
2025-10-30T18:17:46.322Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e Address:19aae2d4-c32d-846d-4f26-0a1aa55ad75e}]"
--- PASS: TestRaft_NoRestoreOnStart (0.19s)
2025-10-30T18:17:46.322Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering follower state: follower="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Follower]" leader-address= leader-id=
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-10-30T18:17:46.322Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 Address:b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56}]"
2025-10-30T18:17:46.322Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:46.322Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: entering follower state: follower="Node at b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 [Follower]" leader-address= leader-id=
2025-10-30T18:17:46.390Z [WARN]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:46.390Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: entering candidate state: node="Node at b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 [Candidate]" term=2
2025-10-30T18:17:46.390Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: election won: term=2 tally=1
2025-10-30T18:17:46.390Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: entering leader state: leader="Node at b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 [Leader]"
2025-10-30T18:17:46.422Z [WARN]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:46.422Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering candidate state: node="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Candidate]" term=3
2025-10-30T18:17:46.422Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: election won: term=3 tally=1
2025-10-30T18:17:46.422Z [INFO]  server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering leader state: leader="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Leader]"
2025-10-30T18:17:46.495Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:46.390510961 +0000 UTC m=+12.669184868 (1 nodes), 67.717156ms from start of poll, 67.842644ms from cluster start. Timeout at 2025-10-30 18:17:46.49551402 +0000 UTC m=+12.774187917, 105.003049ms after stability
2025-10-30T18:17:46.509Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: starting snapshot up to: index=102
2025-10-30T18:17:46.509Z [INFO]  creating new snapshot: path=/tmp/raft4100405487/snapshots/2-102-1761848266509.tmp
2025-10-30T18:17:46.549Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: compacting logs: from=1 to=92
2025-10-30T18:17:46.549Z [INFO]  server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.28s)
=== RUN   TestRaft_UserSnapshot
2025-10-30T18:17:46.603Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c84e118b-1c37-d4b9-5271-cd723944ddd2 Address:c84e118b-1c37-d4b9-5271-cd723944ddd2}]"
2025-10-30T18:17:46.603Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:46.603Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: entering follower state: follower="Node at c84e118b-1c37-d4b9-5271-cd723944ddd2 [Follower]" leader-address= leader-id=
2025-10-30T18:17:46.690Z [WARN]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:46.690Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: entering candidate state: node="Node at c84e118b-1c37-d4b9-5271-cd723944ddd2 [Candidate]" term=2
2025-10-30T18:17:46.690Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: election won: term=2 tally=1
2025-10-30T18:17:46.690Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: entering leader state: leader="Node at c84e118b-1c37-d4b9-5271-cd723944ddd2 [Leader]"
2025-10-30T18:17:46.795Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:46.690496161 +0000 UTC m=+12.969170063 (1 nodes), 86.968628ms from start of poll, 87.07939ms from cluster start. Timeout at 2025-10-30 18:17:46.795497545 +0000 UTC m=+13.074171444, 105.001381ms after stability
2025-10-30T18:17:46.795Z [ERROR] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: failed to take snapshot: error="nothing new to snapshot"
2025-10-30T18:17:46.795Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: starting snapshot up to: index=12
2025-10-30T18:17:46.796Z [INFO]  creating new snapshot: path=/tmp/raft2872604898/snapshots/2-12-1761848266796.tmp
2025-10-30T18:17:46.797Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: compacting logs: from=1 to=2
2025-10-30T18:17:46.797Z [INFO]  server-c84e118b-1c37-d4b9-5271-cd723944ddd2: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.19s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-10-30T18:17:46.797Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45efb12f-99ac-27dc-797c-5f27cce802aa Address:45efb12f-99ac-27dc-797c-5f27cce802aa} {Suffrage:Voter ID:server-7813d007-382a-7aed-fc45-b527ad4cc34c Address:7813d007-382a-7aed-fc45-b527ad4cc34c} {Suffrage:Voter ID:server-2bd4d265-c072-7715-e62a-6f65e0df8015 Address:2bd4d265-c072-7715-e62a-6f65e0df8015}]"
2025-10-30T18:17:46.798Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: entering follower state: follower="Node at 45efb12f-99ac-27dc-797c-5f27cce802aa [Follower]" leader-address= leader-id=
2025-10-30T18:17:46.798Z [INFO]  server-7813d007-382a-7aed-fc45-b527ad4cc34c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45efb12f-99ac-27dc-797c-5f27cce802aa Address:45efb12f-99ac-27dc-797c-5f27cce802aa} {Suffrage:Voter ID:server-7813d007-382a-7aed-fc45-b527ad4cc34c Address:7813d007-382a-7aed-fc45-b527ad4cc34c} {Suffrage:Voter ID:server-2bd4d265-c072-7715-e62a-6f65e0df8015 Address:2bd4d265-c072-7715-e62a-6f65e0df8015}]"
2025-10-30T18:17:46.798Z [INFO]  server-2bd4d265-c072-7715-e62a-6f65e0df8015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45efb12f-99ac-27dc-797c-5f27cce802aa Address:45efb12f-99ac-27dc-797c-5f27cce802aa} {Suffrage:Voter ID:server-7813d007-382a-7aed-fc45-b527ad4cc34c Address:7813d007-382a-7aed-fc45-b527ad4cc34c} {Suffrage:Voter ID:server-2bd4d265-c072-7715-e62a-6f65e0df8015 Address:2bd4d265-c072-7715-e62a-6f65e0df8015}]"
2025-10-30T18:17:46.798Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:46.798Z [INFO]  server-2bd4d265-c072-7715-e62a-6f65e0df8015: entering follower state: follower="Node at 2bd4d265-c072-7715-e62a-6f65e0df8015 [Follower]" leader-address= leader-id=
2025-10-30T18:17:46.798Z [INFO]  server-7813d007-382a-7aed-fc45-b527ad4cc34c: entering follower state: follower="Node at 7813d007-382a-7aed-fc45-b527ad4cc34c [Follower]" leader-address= leader-id=
2025-10-30T18:17:47.384Z [WARN]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:47.384Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: entering candidate state: node="Node at 45efb12f-99ac-27dc-797c-5f27cce802aa [Candidate]" term=2
2025-10-30T18:17:47.384Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: election won: term=2 tally=2
2025-10-30T18:17:47.384Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: entering leader state: leader="Node at 45efb12f-99ac-27dc-797c-5f27cce802aa [Leader]"
2025-10-30T18:17:47.384Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: added peer, starting replication: peer=server-7813d007-382a-7aed-fc45-b527ad4cc34c
2025-10-30T18:17:47.384Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: added peer, starting replication: peer=server-2bd4d265-c072-7715-e62a-6f65e0df8015
2025-10-30T18:17:47.385Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}"
2025-10-30T18:17:47.385Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}"
2025-10-30T18:17:48.391Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:47.385032042 +0000 UTC m=+13.663705939 (1 nodes), 586.938627ms from start of poll, 587.143815ms from cluster start. Timeout at 2025-10-30 18:17:48.390042036 +0000 UTC m=+14.668715947, 1.005010008s after stability
2025-10-30T18:17:48.392Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: starting snapshot up to: index=12
2025-10-30T18:17:48.392Z [INFO]  creating new snapshot: path=/tmp/raft2694164508/snapshots/2-12-1761848268392.tmp
2025-10-30T18:17:48.393Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: snapshot complete up to: index=12
2025-10-30T18:17:48.394Z [INFO]  creating new snapshot: path=/tmp/raft2694164508/snapshots/2-23-1761848268394.tmp
2025-10-30T18:17:48.403Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: copied to local snapshot: bytes=71
2025-10-30T18:17:48.404Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: snapshot restore progress: id=2-23-1761848268394 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:48.404Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: restored user snapshot: index=23
2025-10-30T18:17:48.404Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found"
2025-10-30T18:17:48.404Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}"
2025-10-30T18:17:48.404Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found"
2025-10-30T18:17:48.404Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}"
2025-10-30T18:17:48.414Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found"
2025-10-30T18:17:48.414Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found"
2025-10-30T18:17:48.414Z [INFO]  creating new snapshot: path=/tmp/raft2682460278/snapshots/2-23-1761848268414.tmp
2025-10-30T18:17:48.414Z [INFO]  server-2bd4d265-c072-7715-e62a-6f65e0df8015: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:48.414Z [INFO]  creating new snapshot: path=/tmp/raft1292056249/snapshots/2-23-1761848268414.tmp
2025-10-30T18:17:48.415Z [INFO]  server-2bd4d265-c072-7715-e62a-6f65e0df8015: copied to local snapshot: bytes=71
2025-10-30T18:17:48.415Z [INFO]  server-7813d007-382a-7aed-fc45-b527ad4cc34c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:48.415Z [INFO]  server-2bd4d265-c072-7715-e62a-6f65e0df8015: snapshot restore progress: id=2-23-1761848268414 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:48.415Z [INFO]  server-2bd4d265-c072-7715-e62a-6f65e0df8015: Installed remote snapshot
2025-10-30T18:17:48.415Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}"
2025-10-30T18:17:48.415Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:48.415Z [INFO]  server-7813d007-382a-7aed-fc45-b527ad4cc34c: copied to local snapshot: bytes=71
2025-10-30T18:17:48.416Z [INFO]  server-7813d007-382a-7aed-fc45-b527ad4cc34c: snapshot restore progress: id=2-23-1761848268414 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:48.416Z [INFO]  server-7813d007-382a-7aed-fc45-b527ad4cc34c: Installed remote snapshot
2025-10-30T18:17:48.416Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}"
2025-10-30T18:17:49.421Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:48.415452593 +0000 UTC m=+14.694126491 (2 nodes), 561ns from start of poll, 1.617564367s from cluster start. Timeout at 2025-10-30 18:17:49.420455911 +0000 UTC m=+15.699129808, 1.005003317s after stability
2025-10-30T18:17:49.433Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}"
2025-10-30T18:17:49.433Z [INFO]  server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-10-30T18:17:49.434Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57817e28-20c9-beea-71a8-6c74eeadb438 Address:57817e28-20c9-beea-71a8-6c74eeadb438} {Suffrage:Voter ID:server-608786fe-c4a9-d217-e925-0891ab0573af Address:608786fe-c4a9-d217-e925-0891ab0573af} {Suffrage:Voter ID:server-38d43f82-a7a3-03e6-da08-5604df1b6901 Address:38d43f82-a7a3-03e6-da08-5604df1b6901}]"
2025-10-30T18:17:49.434Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57817e28-20c9-beea-71a8-6c74eeadb438 Address:57817e28-20c9-beea-71a8-6c74eeadb438} {Suffrage:Voter ID:server-608786fe-c4a9-d217-e925-0891ab0573af Address:608786fe-c4a9-d217-e925-0891ab0573af} {Suffrage:Voter ID:server-38d43f82-a7a3-03e6-da08-5604df1b6901 Address:38d43f82-a7a3-03e6-da08-5604df1b6901}]"
2025-10-30T18:17:49.434Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: entering follower state: follower="Node at 57817e28-20c9-beea-71a8-6c74eeadb438 [Follower]" leader-address= leader-id=
2025-10-30T18:17:49.434Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: entering follower state: follower="Node at 608786fe-c4a9-d217-e925-0891ab0573af [Follower]" leader-address= leader-id=
2025-10-30T18:17:49.434Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57817e28-20c9-beea-71a8-6c74eeadb438 Address:57817e28-20c9-beea-71a8-6c74eeadb438} {Suffrage:Voter ID:server-608786fe-c4a9-d217-e925-0891ab0573af Address:608786fe-c4a9-d217-e925-0891ab0573af} {Suffrage:Voter ID:server-38d43f82-a7a3-03e6-da08-5604df1b6901 Address:38d43f82-a7a3-03e6-da08-5604df1b6901}]"
2025-10-30T18:17:49.434Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:49.434Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: entering follower state: follower="Node at 38d43f82-a7a3-03e6-da08-5604df1b6901 [Follower]" leader-address= leader-id=
2025-10-30T18:17:50.098Z [WARN]  server-608786fe-c4a9-d217-e925-0891ab0573af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:50.098Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: entering candidate state: node="Node at 608786fe-c4a9-d217-e925-0891ab0573af [Candidate]" term=2
2025-10-30T18:17:50.098Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: election won: term=2 tally=2
2025-10-30T18:17:50.098Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: entering leader state: leader="Node at 608786fe-c4a9-d217-e925-0891ab0573af [Leader]"
2025-10-30T18:17:50.098Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: added peer, starting replication: peer=server-57817e28-20c9-beea-71a8-6c74eeadb438
2025-10-30T18:17:50.098Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: added peer, starting replication: peer=server-38d43f82-a7a3-03e6-da08-5604df1b6901
2025-10-30T18:17:50.099Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}"
2025-10-30T18:17:50.099Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}"
2025-10-30T18:17:51.104Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:50.09897598 +0000 UTC m=+16.377649878 (1 nodes), 664.683163ms from start of poll, 664.92151ms from cluster start. Timeout at 2025-10-30 18:17:51.103978867 +0000 UTC m=+17.382652765, 1.005002887s after stability
2025-10-30T18:17:51.104Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: starting snapshot up to: index=12
2025-10-30T18:17:51.104Z [INFO]  creating new snapshot: path=/tmp/raft3530938663/snapshots/2-12-1761848271104.tmp
2025-10-30T18:17:51.105Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: snapshot complete up to: index=12
2025-10-30T18:17:51.105Z [INFO]  creating new snapshot: path=/tmp/raft3530938663/snapshots/2-23-1761848271105.tmp
2025-10-30T18:17:51.106Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: copied to local snapshot: bytes=71
2025-10-30T18:17:51.106Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: snapshot restore progress: id=2-23-1761848271105 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:51.106Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: removing all old logs from log store
2025-10-30T18:17:51.106Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: compacting logs: from=1 to=22
2025-10-30T18:17:51.106Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: restored user snapshot: index=23
2025-10-30T18:17:51.106Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found"
2025-10-30T18:17:51.106Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}"
2025-10-30T18:17:51.106Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found"
2025-10-30T18:17:51.106Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}"
2025-10-30T18:17:51.113Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found"
2025-10-30T18:17:51.113Z [INFO]  creating new snapshot: path=/tmp/raft205117741/snapshots/2-23-1761848271113.tmp
2025-10-30T18:17:51.114Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:51.114Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: copied to local snapshot: bytes=71
2025-10-30T18:17:51.114Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found"
2025-10-30T18:17:51.114Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: snapshot restore progress: id=2-23-1761848271113 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:51.114Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: removing all old logs from log store
2025-10-30T18:17:51.114Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: compacting logs: from=1 to=22
2025-10-30T18:17:51.114Z [INFO]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: Installed remote snapshot
2025-10-30T18:17:51.114Z [WARN]  server-38d43f82-a7a3-03e6-da08-5604df1b6901: skipping application of old log: index=22
2025-10-30T18:17:51.114Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}"
2025-10-30T18:17:51.114Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:51.115Z [INFO]  creating new snapshot: path=/tmp/raft1192626479/snapshots/2-23-1761848271115.tmp
2025-10-30T18:17:51.115Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:51.115Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: copied to local snapshot: bytes=71
2025-10-30T18:17:51.115Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: snapshot restore progress: id=2-23-1761848271115 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:51.115Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: removing all old logs from log store
2025-10-30T18:17:51.115Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: compacting logs: from=1 to=22
2025-10-30T18:17:51.115Z [INFO]  server-57817e28-20c9-beea-71a8-6c74eeadb438: Installed remote snapshot
2025-10-30T18:17:51.115Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}"
2025-10-30T18:17:52.120Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:51.114991304 +0000 UTC m=+17.393665210 (2 nodes), 615ns from start of poll, 1.680936842s from cluster start. Timeout at 2025-10-30 18:17:52.11999406 +0000 UTC m=+18.398667958, 1.005002748s after stability
2025-10-30T18:17:52.132Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}"
2025-10-30T18:17:52.132Z [INFO]  server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-10-30T18:17:52.134Z [INFO]  server-85255c93-7cbb-829a-97b1-be16cf8dc383: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85255c93-7cbb-829a-97b1-be16cf8dc383 Address:85255c93-7cbb-829a-97b1-be16cf8dc383} {Suffrage:Voter ID:server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 Address:c95bd67b-03c8-e2f4-643f-4a8dedcf5f67} {Suffrage:Voter ID:server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 Address:94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}]"
2025-10-30T18:17:52.134Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85255c93-7cbb-829a-97b1-be16cf8dc383 Address:85255c93-7cbb-829a-97b1-be16cf8dc383} {Suffrage:Voter ID:server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 Address:c95bd67b-03c8-e2f4-643f-4a8dedcf5f67} {Suffrage:Voter ID:server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 Address:94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}]"
2025-10-30T18:17:52.134Z [INFO]  server-85255c93-7cbb-829a-97b1-be16cf8dc383: entering follower state: follower="Node at 85255c93-7cbb-829a-97b1-be16cf8dc383 [Follower]" leader-address= leader-id=
2025-10-30T18:17:52.134Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: entering follower state: follower="Node at c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 [Follower]" leader-address= leader-id=
2025-10-30T18:17:52.134Z [INFO]  server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85255c93-7cbb-829a-97b1-be16cf8dc383 Address:85255c93-7cbb-829a-97b1-be16cf8dc383} {Suffrage:Voter ID:server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 Address:c95bd67b-03c8-e2f4-643f-4a8dedcf5f67} {Suffrage:Voter ID:server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 Address:94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}]"
2025-10-30T18:17:52.134Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:52.134Z [INFO]  server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869: entering follower state: follower="Node at 94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 [Follower]" leader-address= leader-id=
2025-10-30T18:17:52.679Z [WARN]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:52.679Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: entering candidate state: node="Node at c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 [Candidate]" term=2
2025-10-30T18:17:52.679Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: election won: term=2 tally=2
2025-10-30T18:17:52.679Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: entering leader state: leader="Node at c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 [Leader]"
2025-10-30T18:17:52.679Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: added peer, starting replication: peer=server-85255c93-7cbb-829a-97b1-be16cf8dc383
2025-10-30T18:17:52.679Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: added peer, starting replication: peer=server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869
2025-10-30T18:17:52.679Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: pipelining replication: peer="{Voter server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}"
2025-10-30T18:17:52.679Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: pipelining replication: peer="{Voter server-85255c93-7cbb-829a-97b1-be16cf8dc383 85255c93-7cbb-829a-97b1-be16cf8dc383}"
2025-10-30T18:17:53.685Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:52.679742649 +0000 UTC m=+18.958416566 (1 nodes), 545.421524ms from start of poll, 545.686385ms from cluster start. Timeout at 2025-10-30 18:17:53.684743611 +0000 UTC m=+19.963417508, 1.005000942s after stability
2025-10-30T18:17:53.686Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: starting snapshot up to: index=12
2025-10-30T18:17:53.686Z [INFO]  creating new snapshot: path=/tmp/raft383714054/snapshots/2-12-1761848273686.tmp
2025-10-30T18:17:53.687Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: snapshot complete up to: index=12
2025-10-30T18:17:53.687Z [INFO]  server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 Address:f99e0af7-b17b-34c6-5053-ee1c133a3e03} {Suffrage:Voter ID:server-cde94500-b21c-20c9-98e1-e9f97439f399 Address:cde94500-b21c-20c9-98e1-e9f97439f399} {Suffrage:Voter ID:server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca Address:2e6de459-da77-8a6d-61ca-f7ccc4b33eca}]"
2025-10-30T18:17:53.687Z [INFO]  server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: entering follower state: follower="Node at f99e0af7-b17b-34c6-5053-ee1c133a3e03 [Follower]" leader-address= leader-id=
2025-10-30T18:17:53.687Z [INFO]  server-cde94500-b21c-20c9-98e1-e9f97439f399: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 Address:f99e0af7-b17b-34c6-5053-ee1c133a3e03} {Suffrage:Voter ID:server-cde94500-b21c-20c9-98e1-e9f97439f399 Address:cde94500-b21c-20c9-98e1-e9f97439f399} {Suffrage:Voter ID:server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca Address:2e6de459-da77-8a6d-61ca-f7ccc4b33eca}]"
2025-10-30T18:17:53.688Z [INFO]  server-cde94500-b21c-20c9-98e1-e9f97439f399: entering follower state: follower="Node at cde94500-b21c-20c9-98e1-e9f97439f399 [Follower]" leader-address= leader-id=
2025-10-30T18:17:53.688Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 Address:f99e0af7-b17b-34c6-5053-ee1c133a3e03} {Suffrage:Voter ID:server-cde94500-b21c-20c9-98e1-e9f97439f399 Address:cde94500-b21c-20c9-98e1-e9f97439f399} {Suffrage:Voter ID:server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca Address:2e6de459-da77-8a6d-61ca-f7ccc4b33eca}]"
2025-10-30T18:17:53.688Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:53.688Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: entering follower state: follower="Node at 2e6de459-da77-8a6d-61ca-f7ccc4b33eca [Follower]" leader-address= leader-id=
2025-10-30T18:17:54.256Z [WARN]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:54.256Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: entering candidate state: node="Node at 2e6de459-da77-8a6d-61ca-f7ccc4b33eca [Candidate]" term=2
2025-10-30T18:17:54.256Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: election won: term=2 tally=2
2025-10-30T18:17:54.256Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: entering leader state: leader="Node at 2e6de459-da77-8a6d-61ca-f7ccc4b33eca [Leader]"
2025-10-30T18:17:54.256Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: added peer, starting replication: peer=server-f99e0af7-b17b-34c6-5053-ee1c133a3e03
2025-10-30T18:17:54.256Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: added peer, starting replication: peer=server-cde94500-b21c-20c9-98e1-e9f97439f399
2025-10-30T18:17:54.257Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-cde94500-b21c-20c9-98e1-e9f97439f399 cde94500-b21c-20c9-98e1-e9f97439f399}"
2025-10-30T18:17:54.257Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 f99e0af7-b17b-34c6-5053-ee1c133a3e03}"
2025-10-30T18:17:55.263Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:54.257473978 +0000 UTC m=+20.536147876 (1 nodes), 569.277941ms from start of poll, 569.712348ms from cluster start. Timeout at 2025-10-30 18:17:55.262476031 +0000 UTC m=+21.541149928, 1.005002052s after stability
2025-10-30T18:17:55.263Z [INFO]  creating new snapshot: path=/tmp/raft2247454591/snapshots/2-13-1761848275263.tmp
2025-10-30T18:17:55.366Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: copied to local snapshot: bytes=71
2025-10-30T18:17:55.366Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: snapshot restore progress: id=2-13-1761848275263 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:55.366Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: restored user snapshot: index=13
2025-10-30T18:17:55.366Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found"
2025-10-30T18:17:55.366Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found"
2025-10-30T18:17:55.366Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: aborting pipeline replication: peer="{Voter server-cde94500-b21c-20c9-98e1-e9f97439f399 cde94500-b21c-20c9-98e1-e9f97439f399}"
2025-10-30T18:17:55.366Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: aborting pipeline replication: peer="{Voter server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 f99e0af7-b17b-34c6-5053-ee1c133a3e03}"
2025-10-30T18:17:55.373Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found"
2025-10-30T18:17:55.374Z [INFO]  creating new snapshot: path=/tmp/raft2761616345/snapshots/2-13-1761848275374.tmp
2025-10-30T18:17:55.374Z [INFO]  server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:55.375Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found"
2025-10-30T18:17:55.375Z [INFO]  creating new snapshot: path=/tmp/raft4078536202/snapshots/2-13-1761848275375.tmp
2025-10-30T18:17:55.375Z [INFO]  server-cde94500-b21c-20c9-98e1-e9f97439f399: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:55.523Z [INFO]  server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: copied to local snapshot: bytes=71
2025-10-30T18:17:55.523Z [INFO]  server-cde94500-b21c-20c9-98e1-e9f97439f399: copied to local snapshot: bytes=71
2025-10-30T18:17:55.523Z [INFO]  server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: snapshot restore progress: id=2-13-1761848275374 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:55.523Z [INFO]  server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: Installed remote snapshot
2025-10-30T18:17:55.523Z [INFO]  server-cde94500-b21c-20c9-98e1-e9f97439f399: snapshot restore progress: id=2-13-1761848275375 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:17:55.523Z [INFO]  server-cde94500-b21c-20c9-98e1-e9f97439f399: Installed remote snapshot
2025-10-30T18:17:55.523Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 f99e0af7-b17b-34c6-5053-ee1c133a3e03}"
2025-10-30T18:17:55.523Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:17:55.523Z [INFO]  server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-cde94500-b21c-20c9-98e1-e9f97439f399 cde94500-b21c-20c9-98e1-e9f97439f399}"
2025-10-30T18:17:56.529Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:17:55.523675657 +0000 UTC m=+21.802349567 (2 nodes), 2.126µs from start of poll, 1.835914039s from cluster start. Timeout at 2025-10-30 18:17:56.528679292 +0000 UTC m=+22.807353189, 1.005003622s after stability
2025-10-30T18:17:56.545Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: aborting pipeline replication: peer="{Voter server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}"
2025-10-30T18:17:56.546Z [INFO]  server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: aborting pipeline replication: peer="{Voter server-85255c93-7cbb-829a-97b1-be16cf8dc383 85255c93-7cbb-829a-97b1-be16cf8dc383}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-10-30T18:17:56.546Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb0a50c-2338-4c38-9356-035502ac6b85 Address:5fb0a50c-2338-4c38-9356-035502ac6b85} {Suffrage:Voter ID:server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 Address:5ac2565b-6f42-222c-779b-cb1b4ead74b2} {Suffrage:Voter ID:server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 Address:6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}]"
2025-10-30T18:17:56.546Z [INFO]  server-5ac2565b-6f42-222c-779b-cb1b4ead74b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb0a50c-2338-4c38-9356-035502ac6b85 Address:5fb0a50c-2338-4c38-9356-035502ac6b85} {Suffrage:Voter ID:server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 Address:5ac2565b-6f42-222c-779b-cb1b4ead74b2} {Suffrage:Voter ID:server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 Address:6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}]"
2025-10-30T18:17:56.546Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: entering follower state: follower="Node at 5fb0a50c-2338-4c38-9356-035502ac6b85 [Follower]" leader-address= leader-id=
2025-10-30T18:17:56.546Z [INFO]  server-5ac2565b-6f42-222c-779b-cb1b4ead74b2: entering follower state: follower="Node at 5ac2565b-6f42-222c-779b-cb1b4ead74b2 [Follower]" leader-address= leader-id=
2025-10-30T18:17:56.547Z [INFO]  server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb0a50c-2338-4c38-9356-035502ac6b85 Address:5fb0a50c-2338-4c38-9356-035502ac6b85} {Suffrage:Voter ID:server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 Address:5ac2565b-6f42-222c-779b-cb1b4ead74b2} {Suffrage:Voter ID:server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 Address:6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}]"
2025-10-30T18:17:56.547Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:56.547Z [INFO]  server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850: entering follower state: follower="Node at 6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 [Follower]" leader-address= leader-id=
2025-10-30T18:17:57.072Z [WARN]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:57.072Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: entering candidate state: node="Node at 5fb0a50c-2338-4c38-9356-035502ac6b85 [Candidate]" term=2
2025-10-30T18:17:57.073Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: election won: term=2 tally=2
2025-10-30T18:17:57.073Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: entering leader state: leader="Node at 5fb0a50c-2338-4c38-9356-035502ac6b85 [Leader]"
2025-10-30T18:17:57.073Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: added peer, starting replication: peer=server-5ac2565b-6f42-222c-779b-cb1b4ead74b2
2025-10-30T18:17:57.073Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: added peer, starting replication: peer=server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850
2025-10-30T18:17:57.073Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: pipelining replication: peer="{Voter server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}"
2025-10-30T18:17:57.073Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: pipelining replication: peer="{Voter server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 5ac2565b-6f42-222c-779b-cb1b4ead74b2}"
2025-10-30T18:17:58.078Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:57.073097908 +0000 UTC m=+23.351771805 (1 nodes), 525.983431ms from start of poll, 526.298966ms from cluster start. Timeout at 2025-10-30 18:17:58.078102332 +0000 UTC m=+24.356776229, 1.005004424s after stability
2025-10-30T18:17:58.079Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: starting snapshot up to: index=12
2025-10-30T18:17:58.079Z [INFO]  creating new snapshot: path=/tmp/raft2888405852/snapshots/2-12-1761848278079.tmp
2025-10-30T18:17:58.259Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: snapshot complete up to: index=12
2025-10-30T18:17:58.259Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83251805-6ece-3215-0036-8804211416f2 Address:83251805-6ece-3215-0036-8804211416f2} {Suffrage:Voter ID:server-7c958cae-37cf-4466-86a9-3d991e3b0a49 Address:7c958cae-37cf-4466-86a9-3d991e3b0a49} {Suffrage:Voter ID:server-6277ade7-660e-73bf-b161-d75bf3a57370 Address:6277ade7-660e-73bf-b161-d75bf3a57370}]"
2025-10-30T18:17:58.259Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: entering follower state: follower="Node at 83251805-6ece-3215-0036-8804211416f2 [Follower]" leader-address= leader-id=
2025-10-30T18:17:58.259Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83251805-6ece-3215-0036-8804211416f2 Address:83251805-6ece-3215-0036-8804211416f2} {Suffrage:Voter ID:server-7c958cae-37cf-4466-86a9-3d991e3b0a49 Address:7c958cae-37cf-4466-86a9-3d991e3b0a49} {Suffrage:Voter ID:server-6277ade7-660e-73bf-b161-d75bf3a57370 Address:6277ade7-660e-73bf-b161-d75bf3a57370}]"
2025-10-30T18:17:58.260Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83251805-6ece-3215-0036-8804211416f2 Address:83251805-6ece-3215-0036-8804211416f2} {Suffrage:Voter ID:server-7c958cae-37cf-4466-86a9-3d991e3b0a49 Address:7c958cae-37cf-4466-86a9-3d991e3b0a49} {Suffrage:Voter ID:server-6277ade7-660e-73bf-b161-d75bf3a57370 Address:6277ade7-660e-73bf-b161-d75bf3a57370}]"
2025-10-30T18:17:58.260Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: entering follower state: follower="Node at 7c958cae-37cf-4466-86a9-3d991e3b0a49 [Follower]" leader-address= leader-id=
2025-10-30T18:17:58.260Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:17:58.260Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: entering follower state: follower="Node at 6277ade7-660e-73bf-b161-d75bf3a57370 [Follower]" leader-address= leader-id=
2025-10-30T18:17:58.861Z [WARN]  server-83251805-6ece-3215-0036-8804211416f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:17:58.861Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: entering candidate state: node="Node at 83251805-6ece-3215-0036-8804211416f2 [Candidate]" term=2
2025-10-30T18:17:58.861Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: election won: term=2 tally=2
2025-10-30T18:17:58.861Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: entering leader state: leader="Node at 83251805-6ece-3215-0036-8804211416f2 [Leader]"
2025-10-30T18:17:58.861Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: added peer, starting replication: peer=server-7c958cae-37cf-4466-86a9-3d991e3b0a49
2025-10-30T18:17:58.861Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: added peer, starting replication: peer=server-6277ade7-660e-73bf-b161-d75bf3a57370
2025-10-30T18:17:58.861Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-6277ade7-660e-73bf-b161-d75bf3a57370 6277ade7-660e-73bf-b161-d75bf3a57370}"
2025-10-30T18:17:58.861Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-7c958cae-37cf-4466-86a9-3d991e3b0a49 7c958cae-37cf-4466-86a9-3d991e3b0a49}"
2025-10-30T18:17:59.867Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:17:58.861302354 +0000 UTC m=+25.139976267 (1 nodes), 601.226586ms from start of poll, 601.504743ms from cluster start. Timeout at 2025-10-30 18:17:59.866305725 +0000 UTC m=+26.144979623, 1.005003356s after stability
2025-10-30T18:17:59.998Z [INFO]  creating new snapshot: path=/tmp/raft3159913288/snapshots/2-13-1761848279998.tmp
2025-10-30T18:18:00.055Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: copied to local snapshot: bytes=71
2025-10-30T18:18:00.056Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: snapshot restore progress: id=2-13-1761848279998 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:00.056Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: removing all old logs from log store
2025-10-30T18:18:00.056Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: compacting logs: from=1 to=2
2025-10-30T18:18:00.056Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: restored user snapshot: index=13
2025-10-30T18:18:00.056Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:00.056Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: aborting pipeline replication: peer="{Voter server-7c958cae-37cf-4466-86a9-3d991e3b0a49 7c958cae-37cf-4466-86a9-3d991e3b0a49}"
2025-10-30T18:18:00.056Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:00.056Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: aborting pipeline replication: peer="{Voter server-6277ade7-660e-73bf-b161-d75bf3a57370 6277ade7-660e-73bf-b161-d75bf3a57370}"
2025-10-30T18:18:00.063Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:00.063Z [INFO]  creating new snapshot: path=/tmp/raft3833466995/snapshots/2-13-1761848280063.tmp
2025-10-30T18:18:00.063Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:00.065Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:00.065Z [INFO]  creating new snapshot: path=/tmp/raft366783963/snapshots/2-13-1761848280065.tmp
2025-10-30T18:18:00.065Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:00.151Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: copied to local snapshot: bytes=71
2025-10-30T18:18:00.151Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: copied to local snapshot: bytes=71
2025-10-30T18:18:00.152Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: snapshot restore progress: id=2-13-1761848280063 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:00.152Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: snapshot restore progress: id=2-13-1761848280065 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:00.152Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: removing all old logs from log store
2025-10-30T18:18:00.152Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: removing all old logs from log store
2025-10-30T18:18:00.152Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: compacting logs: from=1 to=2
2025-10-30T18:18:00.152Z [INFO]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: Installed remote snapshot
2025-10-30T18:18:00.152Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: compacting logs: from=1 to=2
2025-10-30T18:18:00.152Z [INFO]  server-6277ade7-660e-73bf-b161-d75bf3a57370: Installed remote snapshot
2025-10-30T18:18:00.152Z [WARN]  server-7c958cae-37cf-4466-86a9-3d991e3b0a49: skipping application of old log: index=13
2025-10-30T18:18:00.152Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-6277ade7-660e-73bf-b161-d75bf3a57370 6277ade7-660e-73bf-b161-d75bf3a57370}"
2025-10-30T18:18:00.152Z [INFO]  server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-7c958cae-37cf-4466-86a9-3d991e3b0a49 7c958cae-37cf-4466-86a9-3d991e3b0a49}"
2025-10-30T18:18:00.152Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:01.157Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:00.152301623 +0000 UTC m=+26.430975521 (2 nodes), 492ns from start of poll, 1.892503997s from cluster start. Timeout at 2025-10-30 18:18:01.157314803 +0000 UTC m=+27.435988710, 1.005013189s after stability
2025-10-30T18:18:01.168Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: aborting pipeline replication: peer="{Voter server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}"
2025-10-30T18:18:01.168Z [INFO]  server-5fb0a50c-2338-4c38-9356-035502ac6b85: aborting pipeline replication: peer="{Voter server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 5ac2565b-6f42-222c-779b-cb1b4ead74b2}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-10-30T18:18:01.169Z [INFO]  server-cb0ee475-1152-2541-4716-ac0c8d654162: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb0ee475-1152-2541-4716-ac0c8d654162 Address:cb0ee475-1152-2541-4716-ac0c8d654162} {Suffrage:Voter ID:server-31b4661f-d3ae-2b72-0536-84782ccc54bc Address:31b4661f-d3ae-2b72-0536-84782ccc54bc} {Suffrage:Voter ID:server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837 Address:40f9ef41-3e46-3c01-d6e9-e57f47ad9837}]"
2025-10-30T18:18:01.169Z [INFO]  server-31b4661f-d3ae-2b72-0536-84782ccc54bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb0ee475-1152-2541-4716-ac0c8d654162 Address:cb0ee475-1152-2541-4716-ac0c8d654162} {Suffrage:Voter ID:server-31b4661f-d3ae-2b72-0536-84782ccc54bc Address:31b4661f-d3ae-2b72-0536-84782ccc54bc} {Suffrage:Voter ID:server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837 Address:40f9ef41-3e46-3c01-d6e9-e57f47ad9837}]"
2025-10-30T18:18:01.169Z [INFO]  server-cb0ee475-1152-2541-4716-ac0c8d654162: entering follower state: follower="Node at cb0ee475-1152-2541-4716-ac0c8d654162 [Follower]" leader-address= leader-id=
2025-10-30T18:18:01.169Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb0ee475-1152-2541-4716-ac0c8d654162 Address:cb0ee475-1152-2541-4716-ac0c8d654162} {Suffrage:Voter ID:server-31b4661f-d3ae-2b72-0536-84782ccc54bc Address:31b4661f-d3ae-2b72-0536-84782ccc54bc} {Suffrage:Voter ID:server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837 Address:40f9ef41-3e46-3c01-d6e9-e57f47ad9837}]"
2025-10-30T18:18:01.169Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:01.169Z [INFO]  server-31b4661f-d3ae-2b72-0536-84782ccc54bc: entering follower state: follower="Node at 31b4661f-d3ae-2b72-0536-84782ccc54bc [Follower]" leader-address= leader-id=
2025-10-30T18:18:01.169Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: entering follower state: follower="Node at 40f9ef41-3e46-3c01-d6e9-e57f47ad9837 [Follower]" leader-address= leader-id=
2025-10-30T18:18:01.699Z [WARN]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:01.699Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: entering candidate state: node="Node at 40f9ef41-3e46-3c01-d6e9-e57f47ad9837 [Candidate]" term=2
2025-10-30T18:18:01.699Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: election won: term=2 tally=2
2025-10-30T18:18:01.699Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: entering leader state: leader="Node at 40f9ef41-3e46-3c01-d6e9-e57f47ad9837 [Leader]"
2025-10-30T18:18:01.699Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: added peer, starting replication: peer=server-cb0ee475-1152-2541-4716-ac0c8d654162
2025-10-30T18:18:01.699Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: added peer, starting replication: peer=server-31b4661f-d3ae-2b72-0536-84782ccc54bc
2025-10-30T18:18:01.699Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}"
2025-10-30T18:18:01.699Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}"
2025-10-30T18:18:02.705Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:01.699742306 +0000 UTC m=+27.978416211 (1 nodes), 530.117757ms from start of poll, 530.349879ms from cluster start. Timeout at 2025-10-30 18:18:02.704744249 +0000 UTC m=+28.983418147, 1.005001936s after stability
2025-10-30T18:18:02.706Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: starting snapshot up to: index=12
2025-10-30T18:18:02.706Z [INFO]  creating new snapshot: path=/tmp/raft3765395359/snapshots/2-12-1761848282706.tmp
2025-10-30T18:18:02.707Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: snapshot complete up to: index=12
2025-10-30T18:18:02.707Z [INFO]  creating new snapshot: path=/tmp/raft3765395359/snapshots/2-23-1761848282707.tmp
2025-10-30T18:18:02.708Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: copied to local snapshot: bytes=71
2025-10-30T18:18:02.708Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: snapshot restore progress: id=2-23-1761848282707 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:02.708Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: restored user snapshot: index=23
2025-10-30T18:18:02.708Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found"
2025-10-30T18:18:02.708Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found"
2025-10-30T18:18:02.708Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}"
2025-10-30T18:18:02.708Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}"
2025-10-30T18:18:02.714Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found"
2025-10-30T18:18:02.714Z [INFO]  creating new snapshot: path=/tmp/raft3638930787/snapshots/2-23-1761848282714.tmp
2025-10-30T18:18:02.714Z [INFO]  server-cb0ee475-1152-2541-4716-ac0c8d654162: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:02.715Z [INFO]  server-cb0ee475-1152-2541-4716-ac0c8d654162: copied to local snapshot: bytes=71
2025-10-30T18:18:02.715Z [INFO]  server-cb0ee475-1152-2541-4716-ac0c8d654162: snapshot restore progress: id=2-23-1761848282714 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:02.715Z [INFO]  server-cb0ee475-1152-2541-4716-ac0c8d654162: Installed remote snapshot
2025-10-30T18:18:02.715Z [WARN]  server-cb0ee475-1152-2541-4716-ac0c8d654162: skipping application of old log: index=22
2025-10-30T18:18:02.715Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}"
2025-10-30T18:18:02.715Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:02.716Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found"
2025-10-30T18:18:02.716Z [INFO]  creating new snapshot: path=/tmp/raft800884271/snapshots/2-23-1761848282716.tmp
2025-10-30T18:18:02.716Z [INFO]  server-31b4661f-d3ae-2b72-0536-84782ccc54bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:02.717Z [INFO]  server-31b4661f-d3ae-2b72-0536-84782ccc54bc: copied to local snapshot: bytes=71
2025-10-30T18:18:02.717Z [INFO]  server-31b4661f-d3ae-2b72-0536-84782ccc54bc: snapshot restore progress: id=2-23-1761848282716 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:02.717Z [INFO]  server-31b4661f-d3ae-2b72-0536-84782ccc54bc: Installed remote snapshot
2025-10-30T18:18:02.717Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}"
2025-10-30T18:18:03.720Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:02.715493414 +0000 UTC m=+28.994167314 (2 nodes), 484ns from start of poll, 1.546100982s from cluster start. Timeout at 2025-10-30 18:18:03.720505029 +0000 UTC m=+29.999178944, 1.00501163s after stability
2025-10-30T18:18:03.732Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}"
2025-10-30T18:18:03.732Z [INFO]  server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-10-30T18:18:03.733Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561a8915-c417-41d8-cec1-85c4f701c32b Address:561a8915-c417-41d8-cec1-85c4f701c32b} {Suffrage:Voter ID:server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca Address:b7edf66e-f2e7-2b50-89dc-f37cccabb3ca} {Suffrage:Voter ID:server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac Address:8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}]"
2025-10-30T18:18:03.733Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561a8915-c417-41d8-cec1-85c4f701c32b Address:561a8915-c417-41d8-cec1-85c4f701c32b} {Suffrage:Voter ID:server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca Address:b7edf66e-f2e7-2b50-89dc-f37cccabb3ca} {Suffrage:Voter ID:server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac Address:8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}]"
2025-10-30T18:18:03.733Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: entering follower state: follower="Node at 561a8915-c417-41d8-cec1-85c4f701c32b [Follower]" leader-address= leader-id=
2025-10-30T18:18:03.733Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: entering follower state: follower="Node at b7edf66e-f2e7-2b50-89dc-f37cccabb3ca [Follower]" leader-address= leader-id=
2025-10-30T18:18:03.733Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561a8915-c417-41d8-cec1-85c4f701c32b Address:561a8915-c417-41d8-cec1-85c4f701c32b} {Suffrage:Voter ID:server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca Address:b7edf66e-f2e7-2b50-89dc-f37cccabb3ca} {Suffrage:Voter ID:server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac Address:8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}]"
2025-10-30T18:18:03.733Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:03.733Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: entering follower state: follower="Node at 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac [Follower]" leader-address= leader-id=
2025-10-30T18:18:04.326Z [WARN]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:04.326Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: entering candidate state: node="Node at b7edf66e-f2e7-2b50-89dc-f37cccabb3ca [Candidate]" term=2
2025-10-30T18:18:04.326Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: election won: term=2 tally=2
2025-10-30T18:18:04.326Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: entering leader state: leader="Node at b7edf66e-f2e7-2b50-89dc-f37cccabb3ca [Leader]"
2025-10-30T18:18:04.326Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: added peer, starting replication: peer=server-561a8915-c417-41d8-cec1-85c4f701c32b
2025-10-30T18:18:04.326Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: added peer, starting replication: peer=server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac
2025-10-30T18:18:04.326Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}"
2025-10-30T18:18:04.326Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}"
2025-10-30T18:18:05.332Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:04.326540601 +0000 UTC m=+30.605214499 (1 nodes), 593.131286ms from start of poll, 593.41544ms from cluster start. Timeout at 2025-10-30 18:18:05.331543008 +0000 UTC m=+31.610216905, 1.005002406s after stability
2025-10-30T18:18:05.332Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: starting snapshot up to: index=12
2025-10-30T18:18:05.332Z [INFO]  creating new snapshot: path=/tmp/raft1758275657/snapshots/2-12-1761848285332.tmp
2025-10-30T18:18:05.333Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: snapshot complete up to: index=12
2025-10-30T18:18:05.333Z [INFO]  creating new snapshot: path=/tmp/raft1758275657/snapshots/2-23-1761848285333.tmp
2025-10-30T18:18:05.333Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: copied to local snapshot: bytes=71
2025-10-30T18:18:05.334Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: snapshot restore progress: id=2-23-1761848285333 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:05.334Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: removing all old logs from log store
2025-10-30T18:18:05.334Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: compacting logs: from=1 to=22
2025-10-30T18:18:05.334Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: restored user snapshot: index=23
2025-10-30T18:18:05.334Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found"
2025-10-30T18:18:05.334Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}"
2025-10-30T18:18:05.334Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found"
2025-10-30T18:18:05.334Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}"
2025-10-30T18:18:05.342Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found"
2025-10-30T18:18:05.342Z [INFO]  creating new snapshot: path=/tmp/raft3093969630/snapshots/2-23-1761848285342.tmp
2025-10-30T18:18:05.343Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:05.343Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: copied to local snapshot: bytes=71
2025-10-30T18:18:05.343Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: snapshot restore progress: id=2-23-1761848285342 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:05.343Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: removing all old logs from log store
2025-10-30T18:18:05.343Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: compacting logs: from=1 to=22
2025-10-30T18:18:05.343Z [INFO]  server-561a8915-c417-41d8-cec1-85c4f701c32b: Installed remote snapshot
2025-10-30T18:18:05.343Z [WARN]  server-561a8915-c417-41d8-cec1-85c4f701c32b: skipping application of old log: index=22
2025-10-30T18:18:05.343Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}"
2025-10-30T18:18:05.343Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:05.344Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found"
2025-10-30T18:18:05.345Z [INFO]  creating new snapshot: path=/tmp/raft4163550248/snapshots/2-23-1761848285345.tmp
2025-10-30T18:18:05.345Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:05.345Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: copied to local snapshot: bytes=71
2025-10-30T18:18:05.346Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: snapshot restore progress: id=2-23-1761848285345 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:05.346Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: removing all old logs from log store
2025-10-30T18:18:05.346Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: compacting logs: from=1 to=22
2025-10-30T18:18:05.346Z [INFO]  server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: Installed remote snapshot
2025-10-30T18:18:05.347Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}"
2025-10-30T18:18:06.349Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:05.343608178 +0000 UTC m=+31.622282087 (2 nodes), 707ns from start of poll, 1.610483028s from cluster start. Timeout at 2025-10-30 18:18:06.348610943 +0000 UTC m=+32.627284840, 1.005002753s after stability
2025-10-30T18:18:06.365Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}"
2025-10-30T18:18:06.365Z [INFO]  server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-10-30T18:18:06.366Z [INFO]  server-e1cb99c7-b8ea-92a4-ee56-8255791394cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cb99c7-b8ea-92a4-ee56-8255791394cb Address:e1cb99c7-b8ea-92a4-ee56-8255791394cb} {Suffrage:Voter ID:server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef Address:8334db8f-9bd0-cbc8-d840-4dd7803f22ef} {Suffrage:Voter ID:server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 Address:4cd8842c-131a-aaec-75a1-e60cd5b8f838}]"
2025-10-30T18:18:06.366Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cb99c7-b8ea-92a4-ee56-8255791394cb Address:e1cb99c7-b8ea-92a4-ee56-8255791394cb} {Suffrage:Voter ID:server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef Address:8334db8f-9bd0-cbc8-d840-4dd7803f22ef} {Suffrage:Voter ID:server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 Address:4cd8842c-131a-aaec-75a1-e60cd5b8f838}]"
2025-10-30T18:18:06.366Z [INFO]  server-e1cb99c7-b8ea-92a4-ee56-8255791394cb: entering follower state: follower="Node at e1cb99c7-b8ea-92a4-ee56-8255791394cb [Follower]" leader-address= leader-id=
2025-10-30T18:18:06.366Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: entering follower state: follower="Node at 8334db8f-9bd0-cbc8-d840-4dd7803f22ef [Follower]" leader-address= leader-id=
2025-10-30T18:18:06.366Z [INFO]  server-4cd8842c-131a-aaec-75a1-e60cd5b8f838: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cb99c7-b8ea-92a4-ee56-8255791394cb Address:e1cb99c7-b8ea-92a4-ee56-8255791394cb} {Suffrage:Voter ID:server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef Address:8334db8f-9bd0-cbc8-d840-4dd7803f22ef} {Suffrage:Voter ID:server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 Address:4cd8842c-131a-aaec-75a1-e60cd5b8f838}]"
2025-10-30T18:18:06.366Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:06.367Z [INFO]  server-4cd8842c-131a-aaec-75a1-e60cd5b8f838: entering follower state: follower="Node at 4cd8842c-131a-aaec-75a1-e60cd5b8f838 [Follower]" leader-address= leader-id=
2025-10-30T18:18:07.046Z [WARN]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:07.046Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: entering candidate state: node="Node at 8334db8f-9bd0-cbc8-d840-4dd7803f22ef [Candidate]" term=2
2025-10-30T18:18:07.047Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: election won: term=2 tally=2
2025-10-30T18:18:07.047Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: entering leader state: leader="Node at 8334db8f-9bd0-cbc8-d840-4dd7803f22ef [Leader]"
2025-10-30T18:18:07.047Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: added peer, starting replication: peer=server-e1cb99c7-b8ea-92a4-ee56-8255791394cb
2025-10-30T18:18:07.047Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: added peer, starting replication: peer=server-4cd8842c-131a-aaec-75a1-e60cd5b8f838
2025-10-30T18:18:07.047Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: pipelining replication: peer="{Voter server-e1cb99c7-b8ea-92a4-ee56-8255791394cb e1cb99c7-b8ea-92a4-ee56-8255791394cb}"
2025-10-30T18:18:07.047Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: pipelining replication: peer="{Voter server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 4cd8842c-131a-aaec-75a1-e60cd5b8f838}"
2025-10-30T18:18:08.052Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:07.047138003 +0000 UTC m=+33.325811903 (1 nodes), 680.14279ms from start of poll, 680.385909ms from cluster start. Timeout at 2025-10-30 18:18:08.052202772 +0000 UTC m=+34.330876680, 1.005064777s after stability
2025-10-30T18:18:08.052Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: starting snapshot up to: index=12
2025-10-30T18:18:08.052Z [INFO]  creating new snapshot: path=/tmp/raft3073081136/snapshots/2-12-1761848288052.tmp
2025-10-30T18:18:08.053Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: snapshot complete up to: index=12
2025-10-30T18:18:08.055Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-485d23e4-0629-68c4-893b-2d99bc4981e7 Address:485d23e4-0629-68c4-893b-2d99bc4981e7} {Suffrage:Voter ID:server-9ebf4678-a528-5952-ef5e-40e366e0542e Address:9ebf4678-a528-5952-ef5e-40e366e0542e} {Suffrage:Voter ID:server-d7338044-b577-8dda-b3bc-fcec970a1452 Address:d7338044-b577-8dda-b3bc-fcec970a1452}]"
2025-10-30T18:18:08.055Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: entering follower state: follower="Node at 485d23e4-0629-68c4-893b-2d99bc4981e7 [Follower]" leader-address= leader-id=
2025-10-30T18:18:08.055Z [INFO]  server-9ebf4678-a528-5952-ef5e-40e366e0542e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-485d23e4-0629-68c4-893b-2d99bc4981e7 Address:485d23e4-0629-68c4-893b-2d99bc4981e7} {Suffrage:Voter ID:server-9ebf4678-a528-5952-ef5e-40e366e0542e Address:9ebf4678-a528-5952-ef5e-40e366e0542e} {Suffrage:Voter ID:server-d7338044-b577-8dda-b3bc-fcec970a1452 Address:d7338044-b577-8dda-b3bc-fcec970a1452}]"
2025-10-30T18:18:08.055Z [INFO]  server-9ebf4678-a528-5952-ef5e-40e366e0542e: entering follower state: follower="Node at 9ebf4678-a528-5952-ef5e-40e366e0542e [Follower]" leader-address= leader-id=
2025-10-30T18:18:08.055Z [INFO]  server-d7338044-b577-8dda-b3bc-fcec970a1452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-485d23e4-0629-68c4-893b-2d99bc4981e7 Address:485d23e4-0629-68c4-893b-2d99bc4981e7} {Suffrage:Voter ID:server-9ebf4678-a528-5952-ef5e-40e366e0542e Address:9ebf4678-a528-5952-ef5e-40e366e0542e} {Suffrage:Voter ID:server-d7338044-b577-8dda-b3bc-fcec970a1452 Address:d7338044-b577-8dda-b3bc-fcec970a1452}]"
2025-10-30T18:18:08.055Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:08.055Z [INFO]  server-d7338044-b577-8dda-b3bc-fcec970a1452: entering follower state: follower="Node at d7338044-b577-8dda-b3bc-fcec970a1452 [Follower]" leader-address= leader-id=
2025-10-30T18:18:08.765Z [WARN]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:08.765Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: entering candidate state: node="Node at 485d23e4-0629-68c4-893b-2d99bc4981e7 [Candidate]" term=2
2025-10-30T18:18:08.765Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: election won: term=2 tally=2
2025-10-30T18:18:08.765Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: entering leader state: leader="Node at 485d23e4-0629-68c4-893b-2d99bc4981e7 [Leader]"
2025-10-30T18:18:08.765Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: added peer, starting replication: peer=server-9ebf4678-a528-5952-ef5e-40e366e0542e
2025-10-30T18:18:08.765Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: added peer, starting replication: peer=server-d7338044-b577-8dda-b3bc-fcec970a1452
2025-10-30T18:18:08.765Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-9ebf4678-a528-5952-ef5e-40e366e0542e 9ebf4678-a528-5952-ef5e-40e366e0542e}"
2025-10-30T18:18:08.765Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-d7338044-b577-8dda-b3bc-fcec970a1452 d7338044-b577-8dda-b3bc-fcec970a1452}"
2025-10-30T18:18:09.771Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:08.765418557 +0000 UTC m=+35.044092470 (1 nodes), 709.832405ms from start of poll, 710.20486ms from cluster start. Timeout at 2025-10-30 18:18:09.770421739 +0000 UTC m=+36.049095636, 1.005003166s after stability
2025-10-30T18:18:09.771Z [INFO]  creating new snapshot: path=/tmp/raft1091931536/snapshots/2-14-1761848289771.tmp
2025-10-30T18:18:09.771Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: copied to local snapshot: bytes=71
2025-10-30T18:18:09.771Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: snapshot restore progress: id=2-14-1761848289771 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:09.771Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: restored user snapshot: index=14
2025-10-30T18:18:09.771Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found"
2025-10-30T18:18:09.771Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: aborting pipeline replication: peer="{Voter server-9ebf4678-a528-5952-ef5e-40e366e0542e 9ebf4678-a528-5952-ef5e-40e366e0542e}"
2025-10-30T18:18:09.771Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found"
2025-10-30T18:18:09.772Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: aborting pipeline replication: peer="{Voter server-d7338044-b577-8dda-b3bc-fcec970a1452 d7338044-b577-8dda-b3bc-fcec970a1452}"
2025-10-30T18:18:09.777Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found"
2025-10-30T18:18:09.778Z [INFO]  creating new snapshot: path=/tmp/raft3004682100/snapshots/2-14-1761848289778.tmp
2025-10-30T18:18:09.778Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found"
2025-10-30T18:18:09.778Z [INFO]  creating new snapshot: path=/tmp/raft1054771965/snapshots/2-14-1761848289778.tmp
2025-10-30T18:18:09.778Z [INFO]  server-d7338044-b577-8dda-b3bc-fcec970a1452: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:09.778Z [INFO]  server-9ebf4678-a528-5952-ef5e-40e366e0542e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:09.778Z [INFO]  server-d7338044-b577-8dda-b3bc-fcec970a1452: copied to local snapshot: bytes=71
2025-10-30T18:18:09.778Z [INFO]  server-9ebf4678-a528-5952-ef5e-40e366e0542e: copied to local snapshot: bytes=71
2025-10-30T18:18:09.778Z [INFO]  server-d7338044-b577-8dda-b3bc-fcec970a1452: snapshot restore progress: id=2-14-1761848289778 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:09.778Z [INFO]  server-d7338044-b577-8dda-b3bc-fcec970a1452: Installed remote snapshot
2025-10-30T18:18:09.779Z [INFO]  server-9ebf4678-a528-5952-ef5e-40e366e0542e: snapshot restore progress: id=2-14-1761848289778 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:09.780Z [INFO]  server-9ebf4678-a528-5952-ef5e-40e366e0542e: Installed remote snapshot
2025-10-30T18:18:09.780Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-d7338044-b577-8dda-b3bc-fcec970a1452 d7338044-b577-8dda-b3bc-fcec970a1452}"
2025-10-30T18:18:09.780Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:09.780Z [INFO]  server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-9ebf4678-a528-5952-ef5e-40e366e0542e 9ebf4678-a528-5952-ef5e-40e366e0542e}"
2025-10-30T18:18:10.785Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:09.780150254 +0000 UTC m=+36.058824151 (2 nodes), 1.065µs from start of poll, 1.724936541s from cluster start. Timeout at 2025-10-30 18:18:10.785153921 +0000 UTC m=+37.063827818, 1.005003667s after stability
2025-10-30T18:18:10.796Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: aborting pipeline replication: peer="{Voter server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 4cd8842c-131a-aaec-75a1-e60cd5b8f838}"
2025-10-30T18:18:10.796Z [INFO]  server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: aborting pipeline replication: peer="{Voter server-e1cb99c7-b8ea-92a4-ee56-8255791394cb e1cb99c7-b8ea-92a4-ee56-8255791394cb}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-10-30T18:18:10.797Z [INFO]  server-c2fa9d15-4939-de29-8cf8-0304635b83b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2fa9d15-4939-de29-8cf8-0304635b83b8 Address:c2fa9d15-4939-de29-8cf8-0304635b83b8} {Suffrage:Voter ID:server-ec908aba-30e1-0112-6c5a-573d8880cc2d Address:ec908aba-30e1-0112-6c5a-573d8880cc2d} {Suffrage:Voter ID:server-8403c93b-15bc-75c0-dea7-22426435b181 Address:8403c93b-15bc-75c0-dea7-22426435b181}]"
2025-10-30T18:18:10.797Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2fa9d15-4939-de29-8cf8-0304635b83b8 Address:c2fa9d15-4939-de29-8cf8-0304635b83b8} {Suffrage:Voter ID:server-ec908aba-30e1-0112-6c5a-573d8880cc2d Address:ec908aba-30e1-0112-6c5a-573d8880cc2d} {Suffrage:Voter ID:server-8403c93b-15bc-75c0-dea7-22426435b181 Address:8403c93b-15bc-75c0-dea7-22426435b181}]"
2025-10-30T18:18:10.797Z [INFO]  server-c2fa9d15-4939-de29-8cf8-0304635b83b8: entering follower state: follower="Node at c2fa9d15-4939-de29-8cf8-0304635b83b8 [Follower]" leader-address= leader-id=
2025-10-30T18:18:10.798Z [INFO]  server-8403c93b-15bc-75c0-dea7-22426435b181: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2fa9d15-4939-de29-8cf8-0304635b83b8 Address:c2fa9d15-4939-de29-8cf8-0304635b83b8} {Suffrage:Voter ID:server-ec908aba-30e1-0112-6c5a-573d8880cc2d Address:ec908aba-30e1-0112-6c5a-573d8880cc2d} {Suffrage:Voter ID:server-8403c93b-15bc-75c0-dea7-22426435b181 Address:8403c93b-15bc-75c0-dea7-22426435b181}]"
2025-10-30T18:18:10.798Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: entering follower state: follower="Node at ec908aba-30e1-0112-6c5a-573d8880cc2d [Follower]" leader-address= leader-id=
2025-10-30T18:18:10.798Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:10.798Z [INFO]  server-8403c93b-15bc-75c0-dea7-22426435b181: entering follower state: follower="Node at 8403c93b-15bc-75c0-dea7-22426435b181 [Follower]" leader-address= leader-id=
2025-10-30T18:18:11.309Z [WARN]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:11.309Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: entering candidate state: node="Node at ec908aba-30e1-0112-6c5a-573d8880cc2d [Candidate]" term=2
2025-10-30T18:18:11.309Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: election won: term=2 tally=2
2025-10-30T18:18:11.309Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: entering leader state: leader="Node at ec908aba-30e1-0112-6c5a-573d8880cc2d [Leader]"
2025-10-30T18:18:11.309Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: added peer, starting replication: peer=server-c2fa9d15-4939-de29-8cf8-0304635b83b8
2025-10-30T18:18:11.309Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: added peer, starting replication: peer=server-8403c93b-15bc-75c0-dea7-22426435b181
2025-10-30T18:18:11.309Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: pipelining replication: peer="{Voter server-8403c93b-15bc-75c0-dea7-22426435b181 8403c93b-15bc-75c0-dea7-22426435b181}"
2025-10-30T18:18:11.309Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: pipelining replication: peer="{Voter server-c2fa9d15-4939-de29-8cf8-0304635b83b8 c2fa9d15-4939-de29-8cf8-0304635b83b8}"
2025-10-30T18:18:12.315Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:11.309637834 +0000 UTC m=+37.588311733 (1 nodes), 511.581109ms from start of poll, 511.808744ms from cluster start. Timeout at 2025-10-30 18:18:12.314642303 +0000 UTC m=+38.593316203, 1.00500447s after stability
2025-10-30T18:18:12.315Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: starting snapshot up to: index=12
2025-10-30T18:18:12.315Z [INFO]  creating new snapshot: path=/tmp/raft1662924434/snapshots/2-12-1761848292315.tmp
2025-10-30T18:18:12.316Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: snapshot complete up to: index=12
2025-10-30T18:18:12.316Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2 Address:e3d4ef1d-3da5-6ccf-031c-f9401887edf2} {Suffrage:Voter ID:server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c Address:c3206cd3-2dd0-6fe0-0091-3e917e94ce2c} {Suffrage:Voter ID:server-410c7af0-44b1-9b2a-d251-cb3c0583806f Address:410c7af0-44b1-9b2a-d251-cb3c0583806f}]"
2025-10-30T18:18:12.316Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: entering follower state: follower="Node at e3d4ef1d-3da5-6ccf-031c-f9401887edf2 [Follower]" leader-address= leader-id=
2025-10-30T18:18:12.316Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2 Address:e3d4ef1d-3da5-6ccf-031c-f9401887edf2} {Suffrage:Voter ID:server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c Address:c3206cd3-2dd0-6fe0-0091-3e917e94ce2c} {Suffrage:Voter ID:server-410c7af0-44b1-9b2a-d251-cb3c0583806f Address:410c7af0-44b1-9b2a-d251-cb3c0583806f}]"
2025-10-30T18:18:12.317Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: entering follower state: follower="Node at c3206cd3-2dd0-6fe0-0091-3e917e94ce2c [Follower]" leader-address= leader-id=
2025-10-30T18:18:12.317Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2 Address:e3d4ef1d-3da5-6ccf-031c-f9401887edf2} {Suffrage:Voter ID:server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c Address:c3206cd3-2dd0-6fe0-0091-3e917e94ce2c} {Suffrage:Voter ID:server-410c7af0-44b1-9b2a-d251-cb3c0583806f Address:410c7af0-44b1-9b2a-d251-cb3c0583806f}]"
2025-10-30T18:18:12.317Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:12.317Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: entering follower state: follower="Node at 410c7af0-44b1-9b2a-d251-cb3c0583806f [Follower]" leader-address= leader-id=
2025-10-30T18:18:12.843Z [WARN]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:12.843Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: entering candidate state: node="Node at e3d4ef1d-3da5-6ccf-031c-f9401887edf2 [Candidate]" term=2
2025-10-30T18:18:12.843Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: election won: term=2 tally=2
2025-10-30T18:18:12.843Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: entering leader state: leader="Node at e3d4ef1d-3da5-6ccf-031c-f9401887edf2 [Leader]"
2025-10-30T18:18:12.843Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: added peer, starting replication: peer=server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c
2025-10-30T18:18:12.843Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: added peer, starting replication: peer=server-410c7af0-44b1-9b2a-d251-cb3c0583806f
2025-10-30T18:18:12.843Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-410c7af0-44b1-9b2a-d251-cb3c0583806f 410c7af0-44b1-9b2a-d251-cb3c0583806f}"
2025-10-30T18:18:12.843Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c c3206cd3-2dd0-6fe0-0091-3e917e94ce2c}"
2025-10-30T18:18:13.849Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:12.843229667 +0000 UTC m=+39.121903564 (1 nodes), 526.046674ms from start of poll, 526.378551ms from cluster start. Timeout at 2025-10-30 18:18:13.848231739 +0000 UTC m=+40.126905638, 1.005002074s after stability
2025-10-30T18:18:13.849Z [INFO]  creating new snapshot: path=/tmp/raft1194384435/snapshots/2-14-1761848293849.tmp
2025-10-30T18:18:13.849Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: copied to local snapshot: bytes=71
2025-10-30T18:18:13.850Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: snapshot restore progress: id=2-14-1761848293849 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:13.850Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: removing all old logs from log store
2025-10-30T18:18:13.850Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: compacting logs: from=1 to=2
2025-10-30T18:18:13.850Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: restored user snapshot: index=14
2025-10-30T18:18:13.850Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:13.850Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: aborting pipeline replication: peer="{Voter server-410c7af0-44b1-9b2a-d251-cb3c0583806f 410c7af0-44b1-9b2a-d251-cb3c0583806f}"
2025-10-30T18:18:13.850Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:13.850Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: aborting pipeline replication: peer="{Voter server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c c3206cd3-2dd0-6fe0-0091-3e917e94ce2c}"
2025-10-30T18:18:13.855Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:13.855Z [INFO]  creating new snapshot: path=/tmp/raft1098516377/snapshots/2-14-1761848293855.tmp
2025-10-30T18:18:13.856Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:13.856Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: copied to local snapshot: bytes=71
2025-10-30T18:18:13.856Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: snapshot restore progress: id=2-14-1761848293855 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:13.856Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: removing all old logs from log store
2025-10-30T18:18:13.856Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: compacting logs: from=1 to=2
2025-10-30T18:18:13.856Z [INFO]  server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: Installed remote snapshot
2025-10-30T18:18:13.856Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c c3206cd3-2dd0-6fe0-0091-3e917e94ce2c}"
2025-10-30T18:18:13.856Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:13.859Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found"
2025-10-30T18:18:13.859Z [INFO]  creating new snapshot: path=/tmp/raft1159472978/snapshots/2-14-1761848293859.tmp
2025-10-30T18:18:13.859Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:13.859Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: copied to local snapshot: bytes=71
2025-10-30T18:18:13.859Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: snapshot restore progress: id=2-14-1761848293859 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:13.860Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: removing all old logs from log store
2025-10-30T18:18:13.860Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: compacting logs: from=1 to=2
2025-10-30T18:18:13.860Z [INFO]  server-410c7af0-44b1-9b2a-d251-cb3c0583806f: Installed remote snapshot
2025-10-30T18:18:13.860Z [INFO]  server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-410c7af0-44b1-9b2a-d251-cb3c0583806f 410c7af0-44b1-9b2a-d251-cb3c0583806f}"
2025-10-30T18:18:14.862Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:13.856723201 +0000 UTC m=+40.135397110 (2 nodes), 784ns from start of poll, 1.539872097s from cluster start. Timeout at 2025-10-30 18:18:14.861737512 +0000 UTC m=+41.140411418, 1.005014308s after stability
2025-10-30T18:18:14.873Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: aborting pipeline replication: peer="{Voter server-8403c93b-15bc-75c0-dea7-22426435b181 8403c93b-15bc-75c0-dea7-22426435b181}"
2025-10-30T18:18:14.873Z [INFO]  server-ec908aba-30e1-0112-6c5a-573d8880cc2d: aborting pipeline replication: peer="{Voter server-c2fa9d15-4939-de29-8cf8-0304635b83b8 c2fa9d15-4939-de29-8cf8-0304635b83b8}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-10-30T18:18:14.874Z [INFO]  server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a11aa13-229c-7a1e-5a97-6d583dc3914a Address:8a11aa13-229c-7a1e-5a97-6d583dc3914a} {Suffrage:Voter ID:server-8bd36cd1-dce9-28b1-169e-ae212f20b997 Address:8bd36cd1-dce9-28b1-169e-ae212f20b997} {Suffrage:Voter ID:server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600 Address:e410ad3b-cb93-be72-6fe1-e17c4ddb6600}]"
2025-10-30T18:18:14.874Z [INFO]  server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: entering follower state: follower="Node at 8a11aa13-229c-7a1e-5a97-6d583dc3914a [Follower]" leader-address= leader-id=
2025-10-30T18:18:14.874Z [INFO]  server-8bd36cd1-dce9-28b1-169e-ae212f20b997: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a11aa13-229c-7a1e-5a97-6d583dc3914a Address:8a11aa13-229c-7a1e-5a97-6d583dc3914a} {Suffrage:Voter ID:server-8bd36cd1-dce9-28b1-169e-ae212f20b997 Address:8bd36cd1-dce9-28b1-169e-ae212f20b997} {Suffrage:Voter ID:server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600 Address:e410ad3b-cb93-be72-6fe1-e17c4ddb6600}]"
2025-10-30T18:18:14.874Z [INFO]  server-8bd36cd1-dce9-28b1-169e-ae212f20b997: entering follower state: follower="Node at 8bd36cd1-dce9-28b1-169e-ae212f20b997 [Follower]" leader-address= leader-id=
2025-10-30T18:18:14.874Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a11aa13-229c-7a1e-5a97-6d583dc3914a Address:8a11aa13-229c-7a1e-5a97-6d583dc3914a} {Suffrage:Voter ID:server-8bd36cd1-dce9-28b1-169e-ae212f20b997 Address:8bd36cd1-dce9-28b1-169e-ae212f20b997} {Suffrage:Voter ID:server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600 Address:e410ad3b-cb93-be72-6fe1-e17c4ddb6600}]"
2025-10-30T18:18:14.874Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:14.874Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: entering follower state: follower="Node at e410ad3b-cb93-be72-6fe1-e17c4ddb6600 [Follower]" leader-address= leader-id=
2025-10-30T18:18:15.550Z [WARN]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:15.550Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: entering candidate state: node="Node at e410ad3b-cb93-be72-6fe1-e17c4ddb6600 [Candidate]" term=2
2025-10-30T18:18:15.550Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: election won: term=2 tally=2
2025-10-30T18:18:15.550Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: entering leader state: leader="Node at e410ad3b-cb93-be72-6fe1-e17c4ddb6600 [Leader]"
2025-10-30T18:18:15.550Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: added peer, starting replication: peer=server-8a11aa13-229c-7a1e-5a97-6d583dc3914a
2025-10-30T18:18:15.550Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: added peer, starting replication: peer=server-8bd36cd1-dce9-28b1-169e-ae212f20b997
2025-10-30T18:18:15.550Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}"
2025-10-30T18:18:15.550Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}"
2025-10-30T18:18:16.556Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:15.550426788 +0000 UTC m=+41.829100684 (1 nodes), 675.82852ms from start of poll, 676.145288ms from cluster start. Timeout at 2025-10-30 18:18:16.555434062 +0000 UTC m=+42.834107959, 1.005007275s after stability
2025-10-30T18:18:16.556Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: starting snapshot up to: index=12
2025-10-30T18:18:16.556Z [INFO]  creating new snapshot: path=/tmp/raft3053875945/snapshots/2-12-1761848296556.tmp
2025-10-30T18:18:16.557Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: snapshot complete up to: index=12
2025-10-30T18:18:16.557Z [INFO]  creating new snapshot: path=/tmp/raft3053875945/snapshots/2-23-1761848296557.tmp
2025-10-30T18:18:16.558Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: copied to local snapshot: bytes=71
2025-10-30T18:18:16.558Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: snapshot restore progress: id=2-23-1761848296557 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:16.558Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: restored user snapshot: index=23
2025-10-30T18:18:16.558Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found"
2025-10-30T18:18:16.558Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}"
2025-10-30T18:18:16.558Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found"
2025-10-30T18:18:16.558Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}"
2025-10-30T18:18:16.568Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found"
2025-10-30T18:18:16.568Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found"
2025-10-30T18:18:16.568Z [INFO]  creating new snapshot: path=/tmp/raft4159777038/snapshots/2-23-1761848296568.tmp
2025-10-30T18:18:16.568Z [INFO]  creating new snapshot: path=/tmp/raft471217910/snapshots/2-23-1761848296568.tmp
2025-10-30T18:18:16.568Z [INFO]  server-8bd36cd1-dce9-28b1-169e-ae212f20b997: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:16.568Z [INFO]  server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:16.568Z [INFO]  server-8bd36cd1-dce9-28b1-169e-ae212f20b997: copied to local snapshot: bytes=71
2025-10-30T18:18:16.568Z [INFO]  server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: copied to local snapshot: bytes=71
2025-10-30T18:18:16.568Z [INFO]  server-8bd36cd1-dce9-28b1-169e-ae212f20b997: snapshot restore progress: id=2-23-1761848296568 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:16.569Z [INFO]  server-8bd36cd1-dce9-28b1-169e-ae212f20b997: Installed remote snapshot
2025-10-30T18:18:16.569Z [INFO]  server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: snapshot restore progress: id=2-23-1761848296568 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:16.569Z [WARN]  server-8bd36cd1-dce9-28b1-169e-ae212f20b997: skipping application of old log: index=22
2025-10-30T18:18:16.569Z [INFO]  server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: Installed remote snapshot
2025-10-30T18:18:16.569Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}"
2025-10-30T18:18:16.569Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:16.569Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}"
2025-10-30T18:18:17.574Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:16.569199374 +0000 UTC m=+42.847873275 (2 nodes), 822ns from start of poll, 1.694917879s from cluster start. Timeout at 2025-10-30 18:18:17.574208981 +0000 UTC m=+43.852882890, 1.005009615s after stability
2025-10-30T18:18:17.586Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}"
2025-10-30T18:18:17.586Z [INFO]  server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-10-30T18:18:17.587Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68eef77e-3208-b381-278a-45983e61fee5 Address:68eef77e-3208-b381-278a-45983e61fee5} {Suffrage:Voter ID:server-c623027d-1b46-13b1-30f2-99bf616d7e3b Address:c623027d-1b46-13b1-30f2-99bf616d7e3b} {Suffrage:Voter ID:server-ee861805-a79a-24cd-a55c-27b5dece4f98 Address:ee861805-a79a-24cd-a55c-27b5dece4f98}]"
2025-10-30T18:18:17.588Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68eef77e-3208-b381-278a-45983e61fee5 Address:68eef77e-3208-b381-278a-45983e61fee5} {Suffrage:Voter ID:server-c623027d-1b46-13b1-30f2-99bf616d7e3b Address:c623027d-1b46-13b1-30f2-99bf616d7e3b} {Suffrage:Voter ID:server-ee861805-a79a-24cd-a55c-27b5dece4f98 Address:ee861805-a79a-24cd-a55c-27b5dece4f98}]"
2025-10-30T18:18:17.588Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: entering follower state: follower="Node at 68eef77e-3208-b381-278a-45983e61fee5 [Follower]" leader-address= leader-id=
2025-10-30T18:18:17.588Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: entering follower state: follower="Node at c623027d-1b46-13b1-30f2-99bf616d7e3b [Follower]" leader-address= leader-id=
2025-10-30T18:18:17.588Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68eef77e-3208-b381-278a-45983e61fee5 Address:68eef77e-3208-b381-278a-45983e61fee5} {Suffrage:Voter ID:server-c623027d-1b46-13b1-30f2-99bf616d7e3b Address:c623027d-1b46-13b1-30f2-99bf616d7e3b} {Suffrage:Voter ID:server-ee861805-a79a-24cd-a55c-27b5dece4f98 Address:ee861805-a79a-24cd-a55c-27b5dece4f98}]"
2025-10-30T18:18:17.588Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:17.588Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: entering follower state: follower="Node at ee861805-a79a-24cd-a55c-27b5dece4f98 [Follower]" leader-address= leader-id=
2025-10-30T18:18:18.316Z [WARN]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:18.316Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: entering candidate state: node="Node at c623027d-1b46-13b1-30f2-99bf616d7e3b [Candidate]" term=2
2025-10-30T18:18:18.316Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: election won: term=2 tally=2
2025-10-30T18:18:18.316Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: entering leader state: leader="Node at c623027d-1b46-13b1-30f2-99bf616d7e3b [Leader]"
2025-10-30T18:18:18.316Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: added peer, starting replication: peer=server-68eef77e-3208-b381-278a-45983e61fee5
2025-10-30T18:18:18.316Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: added peer, starting replication: peer=server-ee861805-a79a-24cd-a55c-27b5dece4f98
2025-10-30T18:18:18.316Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}"
2025-10-30T18:18:18.316Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}"
2025-10-30T18:18:19.321Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:18.316583877 +0000 UTC m=+44.595257774 (1 nodes), 728.32873ms from start of poll, 728.705482ms from cluster start. Timeout at 2025-10-30 18:18:19.321585357 +0000 UTC m=+45.600259254, 1.00500148s after stability
2025-10-30T18:18:19.322Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: starting snapshot up to: index=12
2025-10-30T18:18:19.322Z [INFO]  creating new snapshot: path=/tmp/raft529528452/snapshots/2-12-1761848299322.tmp
2025-10-30T18:18:19.322Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: snapshot complete up to: index=12
2025-10-30T18:18:19.322Z [INFO]  creating new snapshot: path=/tmp/raft529528452/snapshots/2-23-1761848299322.tmp
2025-10-30T18:18:19.323Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: copied to local snapshot: bytes=71
2025-10-30T18:18:19.323Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: snapshot restore progress: id=2-23-1761848299322 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:19.323Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: removing all old logs from log store
2025-10-30T18:18:19.323Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: compacting logs: from=1 to=22
2025-10-30T18:18:19.323Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: restored user snapshot: index=23
2025-10-30T18:18:19.323Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found"
2025-10-30T18:18:19.323Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found"
2025-10-30T18:18:19.323Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}"
2025-10-30T18:18:19.323Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}"
2025-10-30T18:18:19.332Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found"
2025-10-30T18:18:19.332Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found"
2025-10-30T18:18:19.333Z [INFO]  creating new snapshot: path=/tmp/raft792270234/snapshots/2-23-1761848299333.tmp
2025-10-30T18:18:19.333Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:19.333Z [INFO]  creating new snapshot: path=/tmp/raft3375558584/snapshots/2-23-1761848299333.tmp
2025-10-30T18:18:19.334Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:19.334Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: copied to local snapshot: bytes=71
2025-10-30T18:18:19.334Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: copied to local snapshot: bytes=71
2025-10-30T18:18:19.334Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: snapshot restore progress: id=2-23-1761848299333 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:19.334Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: removing all old logs from log store
2025-10-30T18:18:19.334Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: compacting logs: from=1 to=22
2025-10-30T18:18:19.334Z [INFO]  server-68eef77e-3208-b381-278a-45983e61fee5: Installed remote snapshot
2025-10-30T18:18:19.335Z [WARN]  server-68eef77e-3208-b381-278a-45983e61fee5: skipping application of old log: index=22
2025-10-30T18:18:19.335Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: snapshot restore progress: id=2-23-1761848299333 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:19.335Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: removing all old logs from log store
2025-10-30T18:18:19.335Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: compacting logs: from=1 to=22
2025-10-30T18:18:19.335Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}"
2025-10-30T18:18:19.335Z [INFO]  server-ee861805-a79a-24cd-a55c-27b5dece4f98: Installed remote snapshot
2025-10-30T18:18:19.335Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:19.335Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}"
2025-10-30T18:18:20.340Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:19.335104056 +0000 UTC m=+45.613777955 (2 nodes), 998ns from start of poll, 1.747225663s from cluster start. Timeout at 2025-10-30 18:18:20.340114096 +0000 UTC m=+46.618788006, 1.005010051s after stability
2025-10-30T18:18:20.351Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}"
2025-10-30T18:18:20.351Z [INFO]  server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-10-30T18:18:20.352Z [INFO]  server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e Address:e354a99b-aaa3-c6e2-f235-3dfab84fe83e} {Suffrage:Voter ID:server-dac695b8-a066-2646-30f8-d8b9c0078a83 Address:dac695b8-a066-2646-30f8-d8b9c0078a83} {Suffrage:Voter ID:server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049 Address:d96195c3-9dec-9dd9-a246-a0fc2c3a2049}]"
2025-10-30T18:18:20.352Z [INFO]  server-dac695b8-a066-2646-30f8-d8b9c0078a83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e Address:e354a99b-aaa3-c6e2-f235-3dfab84fe83e} {Suffrage:Voter ID:server-dac695b8-a066-2646-30f8-d8b9c0078a83 Address:dac695b8-a066-2646-30f8-d8b9c0078a83} {Suffrage:Voter ID:server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049 Address:d96195c3-9dec-9dd9-a246-a0fc2c3a2049}]"
2025-10-30T18:18:20.352Z [INFO]  server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e: entering follower state: follower="Node at e354a99b-aaa3-c6e2-f235-3dfab84fe83e [Follower]" leader-address= leader-id=
2025-10-30T18:18:20.352Z [INFO]  server-dac695b8-a066-2646-30f8-d8b9c0078a83: entering follower state: follower="Node at dac695b8-a066-2646-30f8-d8b9c0078a83 [Follower]" leader-address= leader-id=
2025-10-30T18:18:20.352Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e Address:e354a99b-aaa3-c6e2-f235-3dfab84fe83e} {Suffrage:Voter ID:server-dac695b8-a066-2646-30f8-d8b9c0078a83 Address:dac695b8-a066-2646-30f8-d8b9c0078a83} {Suffrage:Voter ID:server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049 Address:d96195c3-9dec-9dd9-a246-a0fc2c3a2049}]"
2025-10-30T18:18:20.352Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:20.352Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: entering follower state: follower="Node at d96195c3-9dec-9dd9-a246-a0fc2c3a2049 [Follower]" leader-address= leader-id=
2025-10-30T18:18:21.001Z [WARN]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:21.001Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: entering candidate state: node="Node at d96195c3-9dec-9dd9-a246-a0fc2c3a2049 [Candidate]" term=2
2025-10-30T18:18:21.001Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: election won: term=2 tally=2
2025-10-30T18:18:21.001Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: entering leader state: leader="Node at d96195c3-9dec-9dd9-a246-a0fc2c3a2049 [Leader]"
2025-10-30T18:18:21.001Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: added peer, starting replication: peer=server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e
2025-10-30T18:18:21.001Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: added peer, starting replication: peer=server-dac695b8-a066-2646-30f8-d8b9c0078a83
2025-10-30T18:18:21.002Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: pipelining replication: peer="{Voter server-dac695b8-a066-2646-30f8-d8b9c0078a83 dac695b8-a066-2646-30f8-d8b9c0078a83}"
2025-10-30T18:18:21.002Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: pipelining replication: peer="{Voter server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e e354a99b-aaa3-c6e2-f235-3dfab84fe83e}"
2025-10-30T18:18:22.007Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:21.001953009 +0000 UTC m=+47.280626906 (1 nodes), 649.587675ms from start of poll, 649.817248ms from cluster start. Timeout at 2025-10-30 18:18:22.00695676 +0000 UTC m=+48.285630657, 1.005003751s after stability
2025-10-30T18:18:22.008Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: starting snapshot up to: index=12
2025-10-30T18:18:22.008Z [INFO]  creating new snapshot: path=/tmp/raft3054392106/snapshots/2-12-1761848302008.tmp
2025-10-30T18:18:22.008Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: snapshot complete up to: index=12
2025-10-30T18:18:22.010Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acb4f508-92ed-0d88-7103-52bc09f9abd4 Address:acb4f508-92ed-0d88-7103-52bc09f9abd4} {Suffrage:Voter ID:server-15259e9c-4869-743d-fed4-4e0d522d7c7f Address:15259e9c-4869-743d-fed4-4e0d522d7c7f} {Suffrage:Voter ID:server-4389cd1c-3106-5587-c166-6e11581f620b Address:4389cd1c-3106-5587-c166-6e11581f620b}]"
2025-10-30T18:18:22.010Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: entering follower state: follower="Node at acb4f508-92ed-0d88-7103-52bc09f9abd4 [Follower]" leader-address= leader-id=
2025-10-30T18:18:22.010Z [INFO]  server-15259e9c-4869-743d-fed4-4e0d522d7c7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acb4f508-92ed-0d88-7103-52bc09f9abd4 Address:acb4f508-92ed-0d88-7103-52bc09f9abd4} {Suffrage:Voter ID:server-15259e9c-4869-743d-fed4-4e0d522d7c7f Address:15259e9c-4869-743d-fed4-4e0d522d7c7f} {Suffrage:Voter ID:server-4389cd1c-3106-5587-c166-6e11581f620b Address:4389cd1c-3106-5587-c166-6e11581f620b}]"
2025-10-30T18:18:22.010Z [INFO]  server-15259e9c-4869-743d-fed4-4e0d522d7c7f: entering follower state: follower="Node at 15259e9c-4869-743d-fed4-4e0d522d7c7f [Follower]" leader-address= leader-id=
2025-10-30T18:18:22.010Z [INFO]  server-4389cd1c-3106-5587-c166-6e11581f620b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acb4f508-92ed-0d88-7103-52bc09f9abd4 Address:acb4f508-92ed-0d88-7103-52bc09f9abd4} {Suffrage:Voter ID:server-15259e9c-4869-743d-fed4-4e0d522d7c7f Address:15259e9c-4869-743d-fed4-4e0d522d7c7f} {Suffrage:Voter ID:server-4389cd1c-3106-5587-c166-6e11581f620b Address:4389cd1c-3106-5587-c166-6e11581f620b}]"
2025-10-30T18:18:22.010Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:22.010Z [INFO]  server-4389cd1c-3106-5587-c166-6e11581f620b: entering follower state: follower="Node at 4389cd1c-3106-5587-c166-6e11581f620b [Follower]" leader-address= leader-id=
2025-10-30T18:18:22.679Z [WARN]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:22.679Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: entering candidate state: node="Node at acb4f508-92ed-0d88-7103-52bc09f9abd4 [Candidate]" term=2
2025-10-30T18:18:22.680Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: election won: term=2 tally=2
2025-10-30T18:18:22.680Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: entering leader state: leader="Node at acb4f508-92ed-0d88-7103-52bc09f9abd4 [Leader]"
2025-10-30T18:18:22.680Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: added peer, starting replication: peer=server-15259e9c-4869-743d-fed4-4e0d522d7c7f
2025-10-30T18:18:22.680Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: added peer, starting replication: peer=server-4389cd1c-3106-5587-c166-6e11581f620b
2025-10-30T18:18:22.680Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-15259e9c-4869-743d-fed4-4e0d522d7c7f 15259e9c-4869-743d-fed4-4e0d522d7c7f}"
2025-10-30T18:18:22.680Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-4389cd1c-3106-5587-c166-6e11581f620b 4389cd1c-3106-5587-c166-6e11581f620b}"
2025-10-30T18:18:23.685Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:22.680090206 +0000 UTC m=+48.958764102 (1 nodes), 669.394425ms from start of poll, 669.768725ms from cluster start. Timeout at 2025-10-30 18:18:23.685090459 +0000 UTC m=+49.963764355, 1.005000253s after stability
2025-10-30T18:18:23.685Z [INFO]  creating new snapshot: path=/tmp/raft779741449/snapshots/2-15-1761848303685.tmp
2025-10-30T18:18:23.701Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: copied to local snapshot: bytes=71
2025-10-30T18:18:23.701Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: snapshot restore progress: id=2-15-1761848303685 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:23.701Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: restored user snapshot: index=15
2025-10-30T18:18:23.701Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found"
2025-10-30T18:18:23.701Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found"
2025-10-30T18:18:23.701Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: aborting pipeline replication: peer="{Voter server-4389cd1c-3106-5587-c166-6e11581f620b 4389cd1c-3106-5587-c166-6e11581f620b}"
2025-10-30T18:18:23.701Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: aborting pipeline replication: peer="{Voter server-15259e9c-4869-743d-fed4-4e0d522d7c7f 15259e9c-4869-743d-fed4-4e0d522d7c7f}"
2025-10-30T18:18:23.708Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found"
2025-10-30T18:18:23.711Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found"
2025-10-30T18:18:23.711Z [INFO]  creating new snapshot: path=/tmp/raft3968030613/snapshots/2-15-1761848303711.tmp
2025-10-30T18:18:23.794Z [INFO]  server-15259e9c-4869-743d-fed4-4e0d522d7c7f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:23.794Z [INFO]  creating new snapshot: path=/tmp/raft2820671455/snapshots/2-15-1761848303794.tmp
2025-10-30T18:18:23.794Z [INFO]  server-15259e9c-4869-743d-fed4-4e0d522d7c7f: copied to local snapshot: bytes=71
2025-10-30T18:18:23.794Z [INFO]  server-4389cd1c-3106-5587-c166-6e11581f620b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:23.795Z [INFO]  server-15259e9c-4869-743d-fed4-4e0d522d7c7f: snapshot restore progress: id=2-15-1761848303711 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:23.795Z [INFO]  server-15259e9c-4869-743d-fed4-4e0d522d7c7f: Installed remote snapshot
2025-10-30T18:18:23.795Z [INFO]  server-4389cd1c-3106-5587-c166-6e11581f620b: copied to local snapshot: bytes=71
2025-10-30T18:18:23.795Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-15259e9c-4869-743d-fed4-4e0d522d7c7f 15259e9c-4869-743d-fed4-4e0d522d7c7f}"
2025-10-30T18:18:23.795Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:23.795Z [INFO]  server-4389cd1c-3106-5587-c166-6e11581f620b: snapshot restore progress: id=2-15-1761848303794 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:23.795Z [INFO]  server-4389cd1c-3106-5587-c166-6e11581f620b: Installed remote snapshot
2025-10-30T18:18:23.795Z [INFO]  server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-4389cd1c-3106-5587-c166-6e11581f620b 4389cd1c-3106-5587-c166-6e11581f620b}"
2025-10-30T18:18:24.800Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:23.795219208 +0000 UTC m=+50.073893107 (2 nodes), 1.023µs from start of poll, 1.78489773s from cluster start. Timeout at 2025-10-30 18:18:24.80022228 +0000 UTC m=+51.078896177, 1.00500307s after stability
2025-10-30T18:18:24.811Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: aborting pipeline replication: peer="{Voter server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e e354a99b-aaa3-c6e2-f235-3dfab84fe83e}"
2025-10-30T18:18:24.811Z [INFO]  server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: aborting pipeline replication: peer="{Voter server-dac695b8-a066-2646-30f8-d8b9c0078a83 dac695b8-a066-2646-30f8-d8b9c0078a83}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-10-30T18:18:24.812Z [INFO]  server-28444ed3-dcd0-621a-7f09-9e964b12f478: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28444ed3-dcd0-621a-7f09-9e964b12f478 Address:28444ed3-dcd0-621a-7f09-9e964b12f478} {Suffrage:Voter ID:server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 Address:8b667098-7eed-2842-f2d9-40fd2a8c15a6} {Suffrage:Voter ID:server-f9af27c7-42d3-8b29-1523-3a4c4e758488 Address:f9af27c7-42d3-8b29-1523-3a4c4e758488}]"
2025-10-30T18:18:24.812Z [INFO]  server-8b667098-7eed-2842-f2d9-40fd2a8c15a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28444ed3-dcd0-621a-7f09-9e964b12f478 Address:28444ed3-dcd0-621a-7f09-9e964b12f478} {Suffrage:Voter ID:server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 Address:8b667098-7eed-2842-f2d9-40fd2a8c15a6} {Suffrage:Voter ID:server-f9af27c7-42d3-8b29-1523-3a4c4e758488 Address:f9af27c7-42d3-8b29-1523-3a4c4e758488}]"
2025-10-30T18:18:24.812Z [INFO]  server-28444ed3-dcd0-621a-7f09-9e964b12f478: entering follower state: follower="Node at 28444ed3-dcd0-621a-7f09-9e964b12f478 [Follower]" leader-address= leader-id=
2025-10-30T18:18:24.812Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28444ed3-dcd0-621a-7f09-9e964b12f478 Address:28444ed3-dcd0-621a-7f09-9e964b12f478} {Suffrage:Voter ID:server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 Address:8b667098-7eed-2842-f2d9-40fd2a8c15a6} {Suffrage:Voter ID:server-f9af27c7-42d3-8b29-1523-3a4c4e758488 Address:f9af27c7-42d3-8b29-1523-3a4c4e758488}]"
2025-10-30T18:18:24.812Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:24.812Z [INFO]  server-8b667098-7eed-2842-f2d9-40fd2a8c15a6: entering follower state: follower="Node at 8b667098-7eed-2842-f2d9-40fd2a8c15a6 [Follower]" leader-address= leader-id=
2025-10-30T18:18:24.812Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: entering follower state: follower="Node at f9af27c7-42d3-8b29-1523-3a4c4e758488 [Follower]" leader-address= leader-id=
2025-10-30T18:18:25.336Z [WARN]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:25.336Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: entering candidate state: node="Node at f9af27c7-42d3-8b29-1523-3a4c4e758488 [Candidate]" term=2
2025-10-30T18:18:25.336Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: election won: term=2 tally=2
2025-10-30T18:18:25.336Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: entering leader state: leader="Node at f9af27c7-42d3-8b29-1523-3a4c4e758488 [Leader]"
2025-10-30T18:18:25.336Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: added peer, starting replication: peer=server-28444ed3-dcd0-621a-7f09-9e964b12f478
2025-10-30T18:18:25.336Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: added peer, starting replication: peer=server-8b667098-7eed-2842-f2d9-40fd2a8c15a6
2025-10-30T18:18:25.336Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: pipelining replication: peer="{Voter server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 8b667098-7eed-2842-f2d9-40fd2a8c15a6}"
2025-10-30T18:18:25.336Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: pipelining replication: peer="{Voter server-28444ed3-dcd0-621a-7f09-9e964b12f478 28444ed3-dcd0-621a-7f09-9e964b12f478}"
2025-10-30T18:18:26.342Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:25.337002453 +0000 UTC m=+51.615676358 (1 nodes), 524.330736ms from start of poll, 524.515068ms from cluster start. Timeout at 2025-10-30 18:18:26.342005844 +0000 UTC m=+52.620679741, 1.005003383s after stability
2025-10-30T18:18:26.343Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: starting snapshot up to: index=12
2025-10-30T18:18:26.343Z [INFO]  creating new snapshot: path=/tmp/raft1615071074/snapshots/2-12-1761848306343.tmp
2025-10-30T18:18:26.398Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: snapshot complete up to: index=12
2025-10-30T18:18:26.399Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d475804d-0f75-c1fe-7644-20bd6b8e50c4 Address:d475804d-0f75-c1fe-7644-20bd6b8e50c4} {Suffrage:Voter ID:server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad Address:7bb1fef8-5766-efd8-52fd-24a1cd26f2ad} {Suffrage:Voter ID:server-2f12b407-dbe3-3e38-9511-92ed05505016 Address:2f12b407-dbe3-3e38-9511-92ed05505016}]"
2025-10-30T18:18:26.399Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d475804d-0f75-c1fe-7644-20bd6b8e50c4 Address:d475804d-0f75-c1fe-7644-20bd6b8e50c4} {Suffrage:Voter ID:server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad Address:7bb1fef8-5766-efd8-52fd-24a1cd26f2ad} {Suffrage:Voter ID:server-2f12b407-dbe3-3e38-9511-92ed05505016 Address:2f12b407-dbe3-3e38-9511-92ed05505016}]"
2025-10-30T18:18:26.399Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: entering follower state: follower="Node at d475804d-0f75-c1fe-7644-20bd6b8e50c4 [Follower]" leader-address= leader-id=
2025-10-30T18:18:26.399Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: entering follower state: follower="Node at 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad [Follower]" leader-address= leader-id=
2025-10-30T18:18:26.399Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d475804d-0f75-c1fe-7644-20bd6b8e50c4 Address:d475804d-0f75-c1fe-7644-20bd6b8e50c4} {Suffrage:Voter ID:server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad Address:7bb1fef8-5766-efd8-52fd-24a1cd26f2ad} {Suffrage:Voter ID:server-2f12b407-dbe3-3e38-9511-92ed05505016 Address:2f12b407-dbe3-3e38-9511-92ed05505016}]"
2025-10-30T18:18:26.399Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:26.399Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: entering follower state: follower="Node at 2f12b407-dbe3-3e38-9511-92ed05505016 [Follower]" leader-address= leader-id=
2025-10-30T18:18:26.951Z [WARN]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:26.951Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: entering candidate state: node="Node at d475804d-0f75-c1fe-7644-20bd6b8e50c4 [Candidate]" term=2
2025-10-30T18:18:26.951Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: election won: term=2 tally=2
2025-10-30T18:18:26.951Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: entering leader state: leader="Node at d475804d-0f75-c1fe-7644-20bd6b8e50c4 [Leader]"
2025-10-30T18:18:26.951Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: added peer, starting replication: peer=server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad
2025-10-30T18:18:26.951Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: added peer, starting replication: peer=server-2f12b407-dbe3-3e38-9511-92ed05505016
2025-10-30T18:18:26.951Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad}"
2025-10-30T18:18:26.951Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-2f12b407-dbe3-3e38-9511-92ed05505016 2f12b407-dbe3-3e38-9511-92ed05505016}"
2025-10-30T18:18:27.957Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:26.951260439 +0000 UTC m=+53.229934336 (1 nodes), 551.448618ms from start of poll, 551.713368ms from cluster start. Timeout at 2025-10-30 18:18:27.956264579 +0000 UTC m=+54.234938476, 1.00500414s after stability
2025-10-30T18:18:27.957Z [INFO]  creating new snapshot: path=/tmp/raft555533492/snapshots/2-15-1761848307957.tmp
2025-10-30T18:18:28.001Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: copied to local snapshot: bytes=71
2025-10-30T18:18:28.001Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: snapshot restore progress: id=2-15-1761848307957 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:28.001Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: removing all old logs from log store
2025-10-30T18:18:28.001Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: compacting logs: from=1 to=2
2025-10-30T18:18:28.001Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: restored user snapshot: index=15
2025-10-30T18:18:28.001Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found"
2025-10-30T18:18:28.001Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: aborting pipeline replication: peer="{Voter server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad}"
2025-10-30T18:18:28.001Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found"
2025-10-30T18:18:28.001Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: aborting pipeline replication: peer="{Voter server-2f12b407-dbe3-3e38-9511-92ed05505016 2f12b407-dbe3-3e38-9511-92ed05505016}"
2025-10-30T18:18:28.008Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found"
2025-10-30T18:18:28.008Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found"
2025-10-30T18:18:28.008Z [INFO]  creating new snapshot: path=/tmp/raft939547717/snapshots/2-15-1761848308008.tmp
2025-10-30T18:18:28.008Z [INFO]  creating new snapshot: path=/tmp/raft563965688/snapshots/2-15-1761848308008.tmp
2025-10-30T18:18:28.008Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:28.008Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:28.101Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: copied to local snapshot: bytes=71
2025-10-30T18:18:28.101Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: copied to local snapshot: bytes=71
2025-10-30T18:18:28.101Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: snapshot restore progress: id=2-15-1761848308008 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:28.101Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: snapshot restore progress: id=2-15-1761848308008 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:28.101Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: removing all old logs from log store
2025-10-30T18:18:28.101Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: compacting logs: from=1 to=2
2025-10-30T18:18:28.101Z [INFO]  server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: Installed remote snapshot
2025-10-30T18:18:28.101Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: removing all old logs from log store
2025-10-30T18:18:28.101Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: compacting logs: from=1 to=2
2025-10-30T18:18:28.101Z [INFO]  server-2f12b407-dbe3-3e38-9511-92ed05505016: Installed remote snapshot
2025-10-30T18:18:28.101Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad}"
2025-10-30T18:18:28.101Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:28.101Z [INFO]  server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-2f12b407-dbe3-3e38-9511-92ed05505016 2f12b407-dbe3-3e38-9511-92ed05505016}"
2025-10-30T18:18:29.107Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:28.101544281 +0000 UTC m=+54.380218178 (2 nodes), 771ns from start of poll, 1.70199721s from cluster start. Timeout at 2025-10-30 18:18:29.106558365 +0000 UTC m=+55.385232280, 1.005014102s after stability
2025-10-30T18:18:29.118Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: aborting pipeline replication: peer="{Voter server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 8b667098-7eed-2842-f2d9-40fd2a8c15a6}"
2025-10-30T18:18:29.118Z [INFO]  server-f9af27c7-42d3-8b29-1523-3a4c4e758488: aborting pipeline replication: peer="{Voter server-28444ed3-dcd0-621a-7f09-9e964b12f478 28444ed3-dcd0-621a-7f09-9e964b12f478}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-10-30T18:18:29.119Z [INFO]  server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd Address:720f909d-2ddb-b4e7-ab96-41817bb3c6fd} {Suffrage:Voter ID:server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 Address:6cb3b18d-9674-c289-dd01-d381a2c35fa6} {Suffrage:Voter ID:server-6980861d-ae8c-eac8-e365-f435c847cd8c Address:6980861d-ae8c-eac8-e365-f435c847cd8c}]"
2025-10-30T18:18:29.119Z [INFO]  server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: entering follower state: follower="Node at 720f909d-2ddb-b4e7-ab96-41817bb3c6fd [Follower]" leader-address= leader-id=
2025-10-30T18:18:29.119Z [INFO]  server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd Address:720f909d-2ddb-b4e7-ab96-41817bb3c6fd} {Suffrage:Voter ID:server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 Address:6cb3b18d-9674-c289-dd01-d381a2c35fa6} {Suffrage:Voter ID:server-6980861d-ae8c-eac8-e365-f435c847cd8c Address:6980861d-ae8c-eac8-e365-f435c847cd8c}]"
2025-10-30T18:18:29.119Z [INFO]  server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: entering follower state: follower="Node at 6cb3b18d-9674-c289-dd01-d381a2c35fa6 [Follower]" leader-address= leader-id=
2025-10-30T18:18:29.120Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd Address:720f909d-2ddb-b4e7-ab96-41817bb3c6fd} {Suffrage:Voter ID:server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 Address:6cb3b18d-9674-c289-dd01-d381a2c35fa6} {Suffrage:Voter ID:server-6980861d-ae8c-eac8-e365-f435c847cd8c Address:6980861d-ae8c-eac8-e365-f435c847cd8c}]"
2025-10-30T18:18:29.120Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:29.120Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: entering follower state: follower="Node at 6980861d-ae8c-eac8-e365-f435c847cd8c [Follower]" leader-address= leader-id=
2025-10-30T18:18:29.787Z [WARN]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:29.787Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: entering candidate state: node="Node at 6980861d-ae8c-eac8-e365-f435c847cd8c [Candidate]" term=2
2025-10-30T18:18:29.787Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: election won: term=2 tally=2
2025-10-30T18:18:29.787Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: entering leader state: leader="Node at 6980861d-ae8c-eac8-e365-f435c847cd8c [Leader]"
2025-10-30T18:18:29.787Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: added peer, starting replication: peer=server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd
2025-10-30T18:18:29.787Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: added peer, starting replication: peer=server-6cb3b18d-9674-c289-dd01-d381a2c35fa6
2025-10-30T18:18:29.787Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}"
2025-10-30T18:18:29.787Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}"
2025-10-30T18:18:30.793Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:29.78786796 +0000 UTC m=+56.066541858 (1 nodes), 667.747592ms from start of poll, 668.130229ms from cluster start. Timeout at 2025-10-30 18:18:30.792873827 +0000 UTC m=+57.071547725, 1.005005867s after stability
2025-10-30T18:18:30.794Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: starting snapshot up to: index=12
2025-10-30T18:18:30.794Z [INFO]  creating new snapshot: path=/tmp/raft4184190451/snapshots/2-12-1761848310794.tmp
2025-10-30T18:18:30.811Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: snapshot complete up to: index=12
2025-10-30T18:18:30.812Z [INFO]  creating new snapshot: path=/tmp/raft4184190451/snapshots/2-113-1761848310812.tmp
2025-10-30T18:18:30.858Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: copied to local snapshot: bytes=71
2025-10-30T18:18:30.859Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: snapshot restore progress: id=2-113-1761848310812 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:30.859Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: restored user snapshot: index=113
2025-10-30T18:18:30.859Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found"
2025-10-30T18:18:30.859Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}"
2025-10-30T18:18:30.859Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found"
2025-10-30T18:18:30.859Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}"
2025-10-30T18:18:30.868Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found"
2025-10-30T18:18:30.868Z [INFO]  creating new snapshot: path=/tmp/raft841580653/snapshots/2-113-1761848310868.tmp
2025-10-30T18:18:30.868Z [INFO]  server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:30.870Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found"
2025-10-30T18:18:30.870Z [INFO]  creating new snapshot: path=/tmp/raft2020485141/snapshots/2-113-1761848310870.tmp
2025-10-30T18:18:30.870Z [INFO]  server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:30.965Z [INFO]  server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: copied to local snapshot: bytes=71
2025-10-30T18:18:30.965Z [INFO]  server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: copied to local snapshot: bytes=71
2025-10-30T18:18:30.965Z [INFO]  server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: snapshot restore progress: id=2-113-1761848310870 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:30.965Z [INFO]  server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: snapshot restore progress: id=2-113-1761848310868 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:30.965Z [INFO]  server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: Installed remote snapshot
2025-10-30T18:18:30.965Z [INFO]  server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: Installed remote snapshot
2025-10-30T18:18:30.965Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}"
2025-10-30T18:18:30.965Z [WARN]  server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: skipping application of old log: index=113
2025-10-30T18:18:30.965Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:30.965Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}"
2025-10-30T18:18:31.971Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:30.965662919 +0000 UTC m=+57.244336817 (2 nodes), 528ns from start of poll, 1.845925188s from cluster start. Timeout at 2025-10-30 18:18:31.970678629 +0000 UTC m=+58.249352544, 1.005015727s after stability
2025-10-30T18:18:31.982Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}"
2025-10-30T18:18:31.982Z [INFO]  server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-10-30T18:18:31.983Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6 Address:c2e2d911-c06f-431e-2fcc-81fd79c71ba6} {Suffrage:Voter ID:server-1b4ab6b8-de7e-133a-41a5-12fb93926aad Address:1b4ab6b8-de7e-133a-41a5-12fb93926aad} {Suffrage:Voter ID:server-f60ce911-6562-57aa-4bdc-18d4ec329d48 Address:f60ce911-6562-57aa-4bdc-18d4ec329d48}]"
2025-10-30T18:18:31.983Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: entering follower state: follower="Node at c2e2d911-c06f-431e-2fcc-81fd79c71ba6 [Follower]" leader-address= leader-id=
2025-10-30T18:18:31.983Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6 Address:c2e2d911-c06f-431e-2fcc-81fd79c71ba6} {Suffrage:Voter ID:server-1b4ab6b8-de7e-133a-41a5-12fb93926aad Address:1b4ab6b8-de7e-133a-41a5-12fb93926aad} {Suffrage:Voter ID:server-f60ce911-6562-57aa-4bdc-18d4ec329d48 Address:f60ce911-6562-57aa-4bdc-18d4ec329d48}]"
2025-10-30T18:18:31.983Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: entering follower state: follower="Node at 1b4ab6b8-de7e-133a-41a5-12fb93926aad [Follower]" leader-address= leader-id=
2025-10-30T18:18:31.983Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6 Address:c2e2d911-c06f-431e-2fcc-81fd79c71ba6} {Suffrage:Voter ID:server-1b4ab6b8-de7e-133a-41a5-12fb93926aad Address:1b4ab6b8-de7e-133a-41a5-12fb93926aad} {Suffrage:Voter ID:server-f60ce911-6562-57aa-4bdc-18d4ec329d48 Address:f60ce911-6562-57aa-4bdc-18d4ec329d48}]"
2025-10-30T18:18:31.983Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:31.983Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: entering follower state: follower="Node at f60ce911-6562-57aa-4bdc-18d4ec329d48 [Follower]" leader-address= leader-id=
2025-10-30T18:18:32.600Z [WARN]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:32.600Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: entering candidate state: node="Node at c2e2d911-c06f-431e-2fcc-81fd79c71ba6 [Candidate]" term=2
2025-10-30T18:18:32.600Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: election won: term=2 tally=2
2025-10-30T18:18:32.600Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: entering leader state: leader="Node at c2e2d911-c06f-431e-2fcc-81fd79c71ba6 [Leader]"
2025-10-30T18:18:32.600Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: added peer, starting replication: peer=server-1b4ab6b8-de7e-133a-41a5-12fb93926aad
2025-10-30T18:18:32.600Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: added peer, starting replication: peer=server-f60ce911-6562-57aa-4bdc-18d4ec329d48
2025-10-30T18:18:32.600Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}"
2025-10-30T18:18:32.600Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}"
2025-10-30T18:18:33.605Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:32.600594344 +0000 UTC m=+58.879268244 (1 nodes), 616.700385ms from start of poll, 617.10079ms from cluster start. Timeout at 2025-10-30 18:18:33.605594704 +0000 UTC m=+59.884268601, 1.005000357s after stability
2025-10-30T18:18:33.606Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: starting snapshot up to: index=12
2025-10-30T18:18:33.606Z [INFO]  creating new snapshot: path=/tmp/raft3068251824/snapshots/2-12-1761848313606.tmp
2025-10-30T18:18:33.606Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: snapshot complete up to: index=12
2025-10-30T18:18:33.607Z [INFO]  creating new snapshot: path=/tmp/raft3068251824/snapshots/2-113-1761848313607.tmp
2025-10-30T18:18:33.607Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: copied to local snapshot: bytes=71
2025-10-30T18:18:33.607Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: snapshot restore progress: id=2-113-1761848313607 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:33.607Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: removing all old logs from log store
2025-10-30T18:18:33.607Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: compacting logs: from=1 to=22
2025-10-30T18:18:33.607Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: restored user snapshot: index=113
2025-10-30T18:18:33.607Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found"
2025-10-30T18:18:33.607Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found"
2025-10-30T18:18:33.607Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}"
2025-10-30T18:18:33.607Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}"
2025-10-30T18:18:33.617Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found"
2025-10-30T18:18:33.617Z [INFO]  creating new snapshot: path=/tmp/raft1078824647/snapshots/2-113-1761848313617.tmp
2025-10-30T18:18:33.617Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found"
2025-10-30T18:18:33.617Z [INFO]  creating new snapshot: path=/tmp/raft2376243238/snapshots/2-113-1761848313617.tmp
2025-10-30T18:18:33.617Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:33.617Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:33.617Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: copied to local snapshot: bytes=71
2025-10-30T18:18:33.618Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: copied to local snapshot: bytes=71
2025-10-30T18:18:33.618Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: snapshot restore progress: id=2-113-1761848313617 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:33.618Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: removing all old logs from log store
2025-10-30T18:18:33.618Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: compacting logs: from=1 to=22
2025-10-30T18:18:33.618Z [INFO]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: Installed remote snapshot
2025-10-30T18:18:33.618Z [WARN]  server-f60ce911-6562-57aa-4bdc-18d4ec329d48: skipping application of old log: index=22
2025-10-30T18:18:33.618Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: snapshot restore progress: id=2-113-1761848313617 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:33.618Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: removing all old logs from log store
2025-10-30T18:18:33.618Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: compacting logs: from=1 to=22
2025-10-30T18:18:33.618Z [INFO]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: Installed remote snapshot
2025-10-30T18:18:33.618Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}"
2025-10-30T18:18:33.618Z [WARN]  server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: skipping application of old log: index=113
2025-10-30T18:18:33.618Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:33.618Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}"
2025-10-30T18:18:34.623Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:33.618344875 +0000 UTC m=+59.897018780 (2 nodes), 760ns from start of poll, 1.634851326s from cluster start. Timeout at 2025-10-30 18:18:34.623354412 +0000 UTC m=+60.902028327, 1.005009547s after stability
2025-10-30T18:18:34.635Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}"
2025-10-30T18:18:34.635Z [INFO]  server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-10-30T18:18:34.636Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca17eb3-d82f-eacc-3286-52c6424582e9 Address:1ca17eb3-d82f-eacc-3286-52c6424582e9} {Suffrage:Voter ID:server-826570be-a109-033c-b9c2-b73e0c64edcf Address:826570be-a109-033c-b9c2-b73e0c64edcf} {Suffrage:Voter ID:server-30cc0303-00d8-773c-a2e0-2b97575db913 Address:30cc0303-00d8-773c-a2e0-2b97575db913}]"
2025-10-30T18:18:34.636Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: entering follower state: follower="Node at 1ca17eb3-d82f-eacc-3286-52c6424582e9 [Follower]" leader-address= leader-id=
2025-10-30T18:18:34.636Z [INFO]  server-826570be-a109-033c-b9c2-b73e0c64edcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca17eb3-d82f-eacc-3286-52c6424582e9 Address:1ca17eb3-d82f-eacc-3286-52c6424582e9} {Suffrage:Voter ID:server-826570be-a109-033c-b9c2-b73e0c64edcf Address:826570be-a109-033c-b9c2-b73e0c64edcf} {Suffrage:Voter ID:server-30cc0303-00d8-773c-a2e0-2b97575db913 Address:30cc0303-00d8-773c-a2e0-2b97575db913}]"
2025-10-30T18:18:34.636Z [INFO]  server-826570be-a109-033c-b9c2-b73e0c64edcf: entering follower state: follower="Node at 826570be-a109-033c-b9c2-b73e0c64edcf [Follower]" leader-address= leader-id=
2025-10-30T18:18:34.636Z [INFO]  server-30cc0303-00d8-773c-a2e0-2b97575db913: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca17eb3-d82f-eacc-3286-52c6424582e9 Address:1ca17eb3-d82f-eacc-3286-52c6424582e9} {Suffrage:Voter ID:server-826570be-a109-033c-b9c2-b73e0c64edcf Address:826570be-a109-033c-b9c2-b73e0c64edcf} {Suffrage:Voter ID:server-30cc0303-00d8-773c-a2e0-2b97575db913 Address:30cc0303-00d8-773c-a2e0-2b97575db913}]"
2025-10-30T18:18:34.636Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:34.636Z [INFO]  server-30cc0303-00d8-773c-a2e0-2b97575db913: entering follower state: follower="Node at 30cc0303-00d8-773c-a2e0-2b97575db913 [Follower]" leader-address= leader-id=
2025-10-30T18:18:35.567Z [WARN]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:35.567Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: entering candidate state: node="Node at 1ca17eb3-d82f-eacc-3286-52c6424582e9 [Candidate]" term=2
2025-10-30T18:18:35.567Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: election won: term=2 tally=2
2025-10-30T18:18:35.567Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: entering leader state: leader="Node at 1ca17eb3-d82f-eacc-3286-52c6424582e9 [Leader]"
2025-10-30T18:18:35.567Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: added peer, starting replication: peer=server-826570be-a109-033c-b9c2-b73e0c64edcf
2025-10-30T18:18:35.567Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: added peer, starting replication: peer=server-30cc0303-00d8-773c-a2e0-2b97575db913
2025-10-30T18:18:35.567Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: pipelining replication: peer="{Voter server-826570be-a109-033c-b9c2-b73e0c64edcf 826570be-a109-033c-b9c2-b73e0c64edcf}"
2025-10-30T18:18:35.567Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: pipelining replication: peer="{Voter server-30cc0303-00d8-773c-a2e0-2b97575db913 30cc0303-00d8-773c-a2e0-2b97575db913}"
2025-10-30T18:18:36.572Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:35.567249867 +0000 UTC m=+61.845923767 (1 nodes), 930.732881ms from start of poll, 931.057274ms from cluster start. Timeout at 2025-10-30 18:18:36.572250355 +0000 UTC m=+62.850924253, 1.005000486s after stability
2025-10-30T18:18:36.574Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: starting snapshot up to: index=12
2025-10-30T18:18:36.574Z [INFO]  creating new snapshot: path=/tmp/raft2548335724/snapshots/2-12-1761848316574.tmp
2025-10-30T18:18:36.574Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: snapshot complete up to: index=12
2025-10-30T18:18:36.575Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4c036b-584c-1601-4c5e-b08a2c94effd Address:8b4c036b-584c-1601-4c5e-b08a2c94effd} {Suffrage:Voter ID:server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 Address:bd6b162b-0bd1-3481-49ca-d3e438a8cab0} {Suffrage:Voter ID:server-1ac23055-c326-0117-dc20-6cdfcbc0804a Address:1ac23055-c326-0117-dc20-6cdfcbc0804a}]"
2025-10-30T18:18:36.575Z [INFO]  server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4c036b-584c-1601-4c5e-b08a2c94effd Address:8b4c036b-584c-1601-4c5e-b08a2c94effd} {Suffrage:Voter ID:server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 Address:bd6b162b-0bd1-3481-49ca-d3e438a8cab0} {Suffrage:Voter ID:server-1ac23055-c326-0117-dc20-6cdfcbc0804a Address:1ac23055-c326-0117-dc20-6cdfcbc0804a}]"
2025-10-30T18:18:36.575Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: entering follower state: follower="Node at 8b4c036b-584c-1601-4c5e-b08a2c94effd [Follower]" leader-address= leader-id=
2025-10-30T18:18:36.575Z [INFO]  server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: entering follower state: follower="Node at bd6b162b-0bd1-3481-49ca-d3e438a8cab0 [Follower]" leader-address= leader-id=
2025-10-30T18:18:36.575Z [INFO]  server-1ac23055-c326-0117-dc20-6cdfcbc0804a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4c036b-584c-1601-4c5e-b08a2c94effd Address:8b4c036b-584c-1601-4c5e-b08a2c94effd} {Suffrage:Voter ID:server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 Address:bd6b162b-0bd1-3481-49ca-d3e438a8cab0} {Suffrage:Voter ID:server-1ac23055-c326-0117-dc20-6cdfcbc0804a Address:1ac23055-c326-0117-dc20-6cdfcbc0804a}]"
2025-10-30T18:18:36.575Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:36.575Z [INFO]  server-1ac23055-c326-0117-dc20-6cdfcbc0804a: entering follower state: follower="Node at 1ac23055-c326-0117-dc20-6cdfcbc0804a [Follower]" leader-address= leader-id=
2025-10-30T18:18:37.192Z [WARN]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:37.192Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: entering candidate state: node="Node at 8b4c036b-584c-1601-4c5e-b08a2c94effd [Candidate]" term=2
2025-10-30T18:18:37.192Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: election won: term=2 tally=2
2025-10-30T18:18:37.192Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: entering leader state: leader="Node at 8b4c036b-584c-1601-4c5e-b08a2c94effd [Leader]"
2025-10-30T18:18:37.192Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: added peer, starting replication: peer=server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0
2025-10-30T18:18:37.192Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: added peer, starting replication: peer=server-1ac23055-c326-0117-dc20-6cdfcbc0804a
2025-10-30T18:18:37.192Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-1ac23055-c326-0117-dc20-6cdfcbc0804a 1ac23055-c326-0117-dc20-6cdfcbc0804a}"
2025-10-30T18:18:37.192Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 bd6b162b-0bd1-3481-49ca-d3e438a8cab0}"
2025-10-30T18:18:38.198Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:37.192355474 +0000 UTC m=+63.471029372 (1 nodes), 616.627045ms from start of poll, 616.965171ms from cluster start. Timeout at 2025-10-30 18:18:38.197359416 +0000 UTC m=+64.476033316, 1.005003944s after stability
2025-10-30T18:18:38.198Z [INFO]  creating new snapshot: path=/tmp/raft3474413342/snapshots/2-113-1761848318198.tmp
2025-10-30T18:18:38.303Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: copied to local snapshot: bytes=71
2025-10-30T18:18:38.303Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: snapshot restore progress: id=2-113-1761848318198 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:38.303Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: restored user snapshot: index=113
2025-10-30T18:18:38.303Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found"
2025-10-30T18:18:38.303Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: aborting pipeline replication: peer="{Voter server-1ac23055-c326-0117-dc20-6cdfcbc0804a 1ac23055-c326-0117-dc20-6cdfcbc0804a}"
2025-10-30T18:18:38.303Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found"
2025-10-30T18:18:38.303Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: aborting pipeline replication: peer="{Voter server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 bd6b162b-0bd1-3481-49ca-d3e438a8cab0}"
2025-10-30T18:18:38.310Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found"
2025-10-30T18:18:38.310Z [INFO]  creating new snapshot: path=/tmp/raft729453438/snapshots/2-113-1761848318310.tmp
2025-10-30T18:18:38.310Z [INFO]  server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:38.311Z [INFO]  server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: copied to local snapshot: bytes=71
2025-10-30T18:18:38.311Z [INFO]  server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: snapshot restore progress: id=2-113-1761848318310 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:38.311Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found"
2025-10-30T18:18:38.311Z [INFO]  server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: Installed remote snapshot
2025-10-30T18:18:38.311Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 bd6b162b-0bd1-3481-49ca-d3e438a8cab0}"
2025-10-30T18:18:38.311Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:38.311Z [INFO]  creating new snapshot: path=/tmp/raft2921723766/snapshots/2-113-1761848318311.tmp
2025-10-30T18:18:38.311Z [INFO]  server-1ac23055-c326-0117-dc20-6cdfcbc0804a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:38.311Z [INFO]  server-1ac23055-c326-0117-dc20-6cdfcbc0804a: copied to local snapshot: bytes=71
2025-10-30T18:18:38.312Z [INFO]  server-1ac23055-c326-0117-dc20-6cdfcbc0804a: snapshot restore progress: id=2-113-1761848318311 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:38.312Z [INFO]  server-1ac23055-c326-0117-dc20-6cdfcbc0804a: Installed remote snapshot
2025-10-30T18:18:38.312Z [INFO]  server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-1ac23055-c326-0117-dc20-6cdfcbc0804a 1ac23055-c326-0117-dc20-6cdfcbc0804a}"
2025-10-30T18:18:39.316Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:38.311315189 +0000 UTC m=+64.589989087 (2 nodes), 708ns from start of poll, 1.735924886s from cluster start. Timeout at 2025-10-30 18:18:39.316332764 +0000 UTC m=+65.595006692, 1.005017605s after stability
2025-10-30T18:18:39.327Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: aborting pipeline replication: peer="{Voter server-826570be-a109-033c-b9c2-b73e0c64edcf 826570be-a109-033c-b9c2-b73e0c64edcf}"
2025-10-30T18:18:39.327Z [INFO]  server-1ca17eb3-d82f-eacc-3286-52c6424582e9: aborting pipeline replication: peer="{Voter server-30cc0303-00d8-773c-a2e0-2b97575db913 30cc0303-00d8-773c-a2e0-2b97575db913}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-10-30T18:18:39.330Z [INFO]  server-748ca023-7768-a08f-8cd7-cf6701cd2f1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-748ca023-7768-a08f-8cd7-cf6701cd2f1a Address:748ca023-7768-a08f-8cd7-cf6701cd2f1a} {Suffrage:Voter ID:server-1818bd9b-16af-9652-9038-d6af42c2c520 Address:1818bd9b-16af-9652-9038-d6af42c2c520} {Suffrage:Voter ID:server-76a58140-ece6-e957-356d-750a6badabda Address:76a58140-ece6-e957-356d-750a6badabda}]"
2025-10-30T18:18:39.330Z [INFO]  server-748ca023-7768-a08f-8cd7-cf6701cd2f1a: entering follower state: follower="Node at 748ca023-7768-a08f-8cd7-cf6701cd2f1a [Follower]" leader-address= leader-id=
2025-10-30T18:18:39.330Z [INFO]  server-1818bd9b-16af-9652-9038-d6af42c2c520: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-748ca023-7768-a08f-8cd7-cf6701cd2f1a Address:748ca023-7768-a08f-8cd7-cf6701cd2f1a} {Suffrage:Voter ID:server-1818bd9b-16af-9652-9038-d6af42c2c520 Address:1818bd9b-16af-9652-9038-d6af42c2c520} {Suffrage:Voter ID:server-76a58140-ece6-e957-356d-750a6badabda Address:76a58140-ece6-e957-356d-750a6badabda}]"
2025-10-30T18:18:39.330Z [INFO]  server-1818bd9b-16af-9652-9038-d6af42c2c520: entering follower state: follower="Node at 1818bd9b-16af-9652-9038-d6af42c2c520 [Follower]" leader-address= leader-id=
2025-10-30T18:18:39.330Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-748ca023-7768-a08f-8cd7-cf6701cd2f1a Address:748ca023-7768-a08f-8cd7-cf6701cd2f1a} {Suffrage:Voter ID:server-1818bd9b-16af-9652-9038-d6af42c2c520 Address:1818bd9b-16af-9652-9038-d6af42c2c520} {Suffrage:Voter ID:server-76a58140-ece6-e957-356d-750a6badabda Address:76a58140-ece6-e957-356d-750a6badabda}]"
2025-10-30T18:18:39.330Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:39.330Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: entering follower state: follower="Node at 76a58140-ece6-e957-356d-750a6badabda [Follower]" leader-address= leader-id=
2025-10-30T18:18:40.055Z [WARN]  server-76a58140-ece6-e957-356d-750a6badabda: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:40.055Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: entering candidate state: node="Node at 76a58140-ece6-e957-356d-750a6badabda [Candidate]" term=2
2025-10-30T18:18:40.055Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: election won: term=2 tally=2
2025-10-30T18:18:40.055Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: entering leader state: leader="Node at 76a58140-ece6-e957-356d-750a6badabda [Leader]"
2025-10-30T18:18:40.055Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: added peer, starting replication: peer=server-748ca023-7768-a08f-8cd7-cf6701cd2f1a
2025-10-30T18:18:40.055Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: added peer, starting replication: peer=server-1818bd9b-16af-9652-9038-d6af42c2c520
2025-10-30T18:18:40.055Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: pipelining replication: peer="{Voter server-1818bd9b-16af-9652-9038-d6af42c2c520 1818bd9b-16af-9652-9038-d6af42c2c520}"
2025-10-30T18:18:40.055Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: pipelining replication: peer="{Voter server-748ca023-7768-a08f-8cd7-cf6701cd2f1a 748ca023-7768-a08f-8cd7-cf6701cd2f1a}"
2025-10-30T18:18:41.060Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:40.055461955 +0000 UTC m=+66.334135868 (1 nodes), 725.164764ms from start of poll, 725.51142ms from cluster start. Timeout at 2025-10-30 18:18:41.060467108 +0000 UTC m=+67.339141012, 1.005005144s after stability
2025-10-30T18:18:41.061Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: starting snapshot up to: index=12
2025-10-30T18:18:41.061Z [INFO]  creating new snapshot: path=/tmp/raft3208121412/snapshots/2-12-1761848321061.tmp
2025-10-30T18:18:41.061Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: snapshot complete up to: index=12
2025-10-30T18:18:41.062Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2c34d70-1753-4cbe-33c3-edb13f282d02 Address:d2c34d70-1753-4cbe-33c3-edb13f282d02} {Suffrage:Voter ID:server-e54b2fd9-af1a-74be-3888-dc953cc87ebd Address:e54b2fd9-af1a-74be-3888-dc953cc87ebd} {Suffrage:Voter ID:server-59d300b7-30d6-0976-d683-9115ee0b89ea Address:59d300b7-30d6-0976-d683-9115ee0b89ea}]"
2025-10-30T18:18:41.062Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2c34d70-1753-4cbe-33c3-edb13f282d02 Address:d2c34d70-1753-4cbe-33c3-edb13f282d02} {Suffrage:Voter ID:server-e54b2fd9-af1a-74be-3888-dc953cc87ebd Address:e54b2fd9-af1a-74be-3888-dc953cc87ebd} {Suffrage:Voter ID:server-59d300b7-30d6-0976-d683-9115ee0b89ea Address:59d300b7-30d6-0976-d683-9115ee0b89ea}]"
2025-10-30T18:18:41.062Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: entering follower state: follower="Node at d2c34d70-1753-4cbe-33c3-edb13f282d02 [Follower]" leader-address= leader-id=
2025-10-30T18:18:41.062Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2c34d70-1753-4cbe-33c3-edb13f282d02 Address:d2c34d70-1753-4cbe-33c3-edb13f282d02} {Suffrage:Voter ID:server-e54b2fd9-af1a-74be-3888-dc953cc87ebd Address:e54b2fd9-af1a-74be-3888-dc953cc87ebd} {Suffrage:Voter ID:server-59d300b7-30d6-0976-d683-9115ee0b89ea Address:59d300b7-30d6-0976-d683-9115ee0b89ea}]"
2025-10-30T18:18:41.062Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:41.062Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: entering follower state: follower="Node at e54b2fd9-af1a-74be-3888-dc953cc87ebd [Follower]" leader-address= leader-id=
2025-10-30T18:18:41.062Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: entering follower state: follower="Node at 59d300b7-30d6-0976-d683-9115ee0b89ea [Follower]" leader-address= leader-id=
2025-10-30T18:18:41.665Z [WARN]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:41.665Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: entering candidate state: node="Node at e54b2fd9-af1a-74be-3888-dc953cc87ebd [Candidate]" term=2
2025-10-30T18:18:41.665Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: election won: term=2 tally=2
2025-10-30T18:18:41.665Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: entering leader state: leader="Node at e54b2fd9-af1a-74be-3888-dc953cc87ebd [Leader]"
2025-10-30T18:18:41.665Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: added peer, starting replication: peer=server-d2c34d70-1753-4cbe-33c3-edb13f282d02
2025-10-30T18:18:41.665Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: added peer, starting replication: peer=server-59d300b7-30d6-0976-d683-9115ee0b89ea
2025-10-30T18:18:41.665Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-59d300b7-30d6-0976-d683-9115ee0b89ea 59d300b7-30d6-0976-d683-9115ee0b89ea}"
2025-10-30T18:18:41.666Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-d2c34d70-1753-4cbe-33c3-edb13f282d02 d2c34d70-1753-4cbe-33c3-edb13f282d02}"
2025-10-30T18:18:42.671Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:41.665985715 +0000 UTC m=+67.944659627 (1 nodes), 603.69682ms from start of poll, 603.873856ms from cluster start. Timeout at 2025-10-30 18:18:42.67099003 +0000 UTC m=+68.949663930, 1.005004303s after stability
2025-10-30T18:18:42.671Z [INFO]  creating new snapshot: path=/tmp/raft2680253167/snapshots/2-113-1761848322671.tmp
2025-10-30T18:18:42.672Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: copied to local snapshot: bytes=71
2025-10-30T18:18:42.672Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: snapshot restore progress: id=2-113-1761848322671 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:42.672Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: removing all old logs from log store
2025-10-30T18:18:42.672Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: compacting logs: from=1 to=2
2025-10-30T18:18:42.672Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: restored user snapshot: index=113
2025-10-30T18:18:42.672Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found"
2025-10-30T18:18:42.672Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: aborting pipeline replication: peer="{Voter server-59d300b7-30d6-0976-d683-9115ee0b89ea 59d300b7-30d6-0976-d683-9115ee0b89ea}"
2025-10-30T18:18:42.672Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found"
2025-10-30T18:18:42.672Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: aborting pipeline replication: peer="{Voter server-d2c34d70-1753-4cbe-33c3-edb13f282d02 d2c34d70-1753-4cbe-33c3-edb13f282d02}"
2025-10-30T18:18:42.681Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found"
2025-10-30T18:18:42.681Z [INFO]  creating new snapshot: path=/tmp/raft2653141277/snapshots/2-113-1761848322681.tmp
2025-10-30T18:18:42.681Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found"
2025-10-30T18:18:42.682Z [INFO]  creating new snapshot: path=/tmp/raft45115175/snapshots/2-113-1761848322682.tmp
2025-10-30T18:18:42.682Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:42.683Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:42.738Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: copied to local snapshot: bytes=71
2025-10-30T18:18:42.738Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: copied to local snapshot: bytes=71
2025-10-30T18:18:42.738Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: snapshot restore progress: id=2-113-1761848322682 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:42.738Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: snapshot restore progress: id=2-113-1761848322681 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:42.738Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: removing all old logs from log store
2025-10-30T18:18:42.738Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: compacting logs: from=1 to=2
2025-10-30T18:18:42.738Z [INFO]  server-d2c34d70-1753-4cbe-33c3-edb13f282d02: Installed remote snapshot
2025-10-30T18:18:42.738Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: removing all old logs from log store
2025-10-30T18:18:42.738Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: compacting logs: from=1 to=2
2025-10-30T18:18:42.738Z [INFO]  server-59d300b7-30d6-0976-d683-9115ee0b89ea: Installed remote snapshot
2025-10-30T18:18:42.738Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-d2c34d70-1753-4cbe-33c3-edb13f282d02 d2c34d70-1753-4cbe-33c3-edb13f282d02}"
2025-10-30T18:18:42.738Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:42.738Z [INFO]  server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-59d300b7-30d6-0976-d683-9115ee0b89ea 59d300b7-30d6-0976-d683-9115ee0b89ea}"
2025-10-30T18:18:43.743Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:42.738427127 +0000 UTC m=+69.017101033 (2 nodes), 662ns from start of poll, 1.676315262s from cluster start. Timeout at 2025-10-30 18:18:43.743448518 +0000 UTC m=+70.022122416, 1.005021383s after stability
2025-10-30T18:18:43.755Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: aborting pipeline replication: peer="{Voter server-1818bd9b-16af-9652-9038-d6af42c2c520 1818bd9b-16af-9652-9038-d6af42c2c520}"
2025-10-30T18:18:43.755Z [INFO]  server-76a58140-ece6-e957-356d-750a6badabda: aborting pipeline replication: peer="{Voter server-748ca023-7768-a08f-8cd7-cf6701cd2f1a 748ca023-7768-a08f-8cd7-cf6701cd2f1a}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-10-30T18:18:43.756Z [INFO]  server-81a0274e-5779-d62b-5744-e91b8b85ae8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a0274e-5779-d62b-5744-e91b8b85ae8a Address:81a0274e-5779-d62b-5744-e91b8b85ae8a} {Suffrage:Voter ID:server-842f6bb8-5908-4692-e352-be148691141a Address:842f6bb8-5908-4692-e352-be148691141a} {Suffrage:Voter ID:server-422d2ebc-0dfe-d501-9c4a-903214eb8a05 Address:422d2ebc-0dfe-d501-9c4a-903214eb8a05}]"
2025-10-30T18:18:43.756Z [INFO]  server-842f6bb8-5908-4692-e352-be148691141a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a0274e-5779-d62b-5744-e91b8b85ae8a Address:81a0274e-5779-d62b-5744-e91b8b85ae8a} {Suffrage:Voter ID:server-842f6bb8-5908-4692-e352-be148691141a Address:842f6bb8-5908-4692-e352-be148691141a} {Suffrage:Voter ID:server-422d2ebc-0dfe-d501-9c4a-903214eb8a05 Address:422d2ebc-0dfe-d501-9c4a-903214eb8a05}]"
2025-10-30T18:18:43.756Z [INFO]  server-81a0274e-5779-d62b-5744-e91b8b85ae8a: entering follower state: follower="Node at 81a0274e-5779-d62b-5744-e91b8b85ae8a [Follower]" leader-address= leader-id=
2025-10-30T18:18:43.756Z [INFO]  server-842f6bb8-5908-4692-e352-be148691141a: entering follower state: follower="Node at 842f6bb8-5908-4692-e352-be148691141a [Follower]" leader-address= leader-id=
2025-10-30T18:18:43.756Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a0274e-5779-d62b-5744-e91b8b85ae8a Address:81a0274e-5779-d62b-5744-e91b8b85ae8a} {Suffrage:Voter ID:server-842f6bb8-5908-4692-e352-be148691141a Address:842f6bb8-5908-4692-e352-be148691141a} {Suffrage:Voter ID:server-422d2ebc-0dfe-d501-9c4a-903214eb8a05 Address:422d2ebc-0dfe-d501-9c4a-903214eb8a05}]"
2025-10-30T18:18:43.756Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:43.756Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: entering follower state: follower="Node at 422d2ebc-0dfe-d501-9c4a-903214eb8a05 [Follower]" leader-address= leader-id=
2025-10-30T18:18:44.388Z [WARN]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:44.389Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: entering candidate state: node="Node at 422d2ebc-0dfe-d501-9c4a-903214eb8a05 [Candidate]" term=2
2025-10-30T18:18:44.389Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: election won: term=2 tally=2
2025-10-30T18:18:44.389Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: entering leader state: leader="Node at 422d2ebc-0dfe-d501-9c4a-903214eb8a05 [Leader]"
2025-10-30T18:18:44.389Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: added peer, starting replication: peer=server-81a0274e-5779-d62b-5744-e91b8b85ae8a
2025-10-30T18:18:44.389Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: added peer, starting replication: peer=server-842f6bb8-5908-4692-e352-be148691141a
2025-10-30T18:18:44.389Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}"
2025-10-30T18:18:44.389Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}"
2025-10-30T18:18:45.394Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:44.389177553 +0000 UTC m=+70.667851449 (1 nodes), 632.931373ms from start of poll, 633.199885ms from cluster start. Timeout at 2025-10-30 18:18:45.394181221 +0000 UTC m=+71.672855118, 1.005003669s after stability
2025-10-30T18:18:45.395Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: starting snapshot up to: index=12
2025-10-30T18:18:45.395Z [INFO]  creating new snapshot: path=/tmp/raft3146786988/snapshots/2-12-1761848325395.tmp
2025-10-30T18:18:45.520Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: snapshot complete up to: index=12
2025-10-30T18:18:45.521Z [INFO]  creating new snapshot: path=/tmp/raft3146786988/snapshots/2-1013-1761848325521.tmp
2025-10-30T18:18:45.521Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: copied to local snapshot: bytes=71
2025-10-30T18:18:45.521Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: snapshot restore progress: id=2-1013-1761848325521 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:45.521Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: restored user snapshot: index=1013
2025-10-30T18:18:45.521Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found"
2025-10-30T18:18:45.521Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}"
2025-10-30T18:18:45.521Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found"
2025-10-30T18:18:45.521Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}"
2025-10-30T18:18:45.528Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found"
2025-10-30T18:18:45.529Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found"
2025-10-30T18:18:45.654Z [INFO]  creating new snapshot: path=/tmp/raft3184662359/snapshots/2-1013-1761848325654.tmp
2025-10-30T18:18:45.654Z [INFO]  creating new snapshot: path=/tmp/raft4105804530/snapshots/2-1013-1761848325654.tmp
2025-10-30T18:18:45.679Z [INFO]  server-81a0274e-5779-d62b-5744-e91b8b85ae8a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:45.679Z [INFO]  server-842f6bb8-5908-4692-e352-be148691141a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:45.679Z [INFO]  server-842f6bb8-5908-4692-e352-be148691141a: copied to local snapshot: bytes=71
2025-10-30T18:18:45.679Z [INFO]  server-81a0274e-5779-d62b-5744-e91b8b85ae8a: copied to local snapshot: bytes=71
2025-10-30T18:18:45.679Z [INFO]  server-842f6bb8-5908-4692-e352-be148691141a: snapshot restore progress: id=2-1013-1761848325654 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:45.679Z [INFO]  server-842f6bb8-5908-4692-e352-be148691141a: Installed remote snapshot
2025-10-30T18:18:45.679Z [INFO]  server-81a0274e-5779-d62b-5744-e91b8b85ae8a: snapshot restore progress: id=2-1013-1761848325654 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:45.679Z [INFO]  server-81a0274e-5779-d62b-5744-e91b8b85ae8a: Installed remote snapshot
2025-10-30T18:18:45.679Z [WARN]  server-842f6bb8-5908-4692-e352-be148691141a: skipping application of old log: index=22
2025-10-30T18:18:45.679Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}"
2025-10-30T18:18:45.679Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:45.679Z [WARN]  server-81a0274e-5779-d62b-5744-e91b8b85ae8a: skipping application of old log: index=1013
2025-10-30T18:18:45.679Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}"
2025-10-30T18:18:46.685Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:45.679825985 +0000 UTC m=+71.958499882 (2 nodes), 722ns from start of poll, 1.923848318s from cluster start. Timeout at 2025-10-30 18:18:46.684834769 +0000 UTC m=+72.963508684, 1.005008802s after stability
2025-10-30T18:18:46.696Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}"
2025-10-30T18:18:46.696Z [INFO]  server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-10-30T18:18:46.697Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf027ee5-214a-039a-38b7-d28ee558c1d4 Address:cf027ee5-214a-039a-38b7-d28ee558c1d4} {Suffrage:Voter ID:server-aa55c372-30da-8a30-8a67-1c2b879e8526 Address:aa55c372-30da-8a30-8a67-1c2b879e8526} {Suffrage:Voter ID:server-7b46b1e4-7fca-26ac-662c-cd4623daf019 Address:7b46b1e4-7fca-26ac-662c-cd4623daf019}]"
2025-10-30T18:18:46.697Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: entering follower state: follower="Node at cf027ee5-214a-039a-38b7-d28ee558c1d4 [Follower]" leader-address= leader-id=
2025-10-30T18:18:46.697Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf027ee5-214a-039a-38b7-d28ee558c1d4 Address:cf027ee5-214a-039a-38b7-d28ee558c1d4} {Suffrage:Voter ID:server-aa55c372-30da-8a30-8a67-1c2b879e8526 Address:aa55c372-30da-8a30-8a67-1c2b879e8526} {Suffrage:Voter ID:server-7b46b1e4-7fca-26ac-662c-cd4623daf019 Address:7b46b1e4-7fca-26ac-662c-cd4623daf019}]"
2025-10-30T18:18:46.697Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf027ee5-214a-039a-38b7-d28ee558c1d4 Address:cf027ee5-214a-039a-38b7-d28ee558c1d4} {Suffrage:Voter ID:server-aa55c372-30da-8a30-8a67-1c2b879e8526 Address:aa55c372-30da-8a30-8a67-1c2b879e8526} {Suffrage:Voter ID:server-7b46b1e4-7fca-26ac-662c-cd4623daf019 Address:7b46b1e4-7fca-26ac-662c-cd4623daf019}]"
2025-10-30T18:18:46.697Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: entering follower state: follower="Node at aa55c372-30da-8a30-8a67-1c2b879e8526 [Follower]" leader-address= leader-id=
2025-10-30T18:18:46.698Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:46.698Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: entering follower state: follower="Node at 7b46b1e4-7fca-26ac-662c-cd4623daf019 [Follower]" leader-address= leader-id=
2025-10-30T18:18:47.217Z [WARN]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:47.217Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: entering candidate state: node="Node at 7b46b1e4-7fca-26ac-662c-cd4623daf019 [Candidate]" term=2
2025-10-30T18:18:47.217Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: election won: term=2 tally=2
2025-10-30T18:18:47.217Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: entering leader state: leader="Node at 7b46b1e4-7fca-26ac-662c-cd4623daf019 [Leader]"
2025-10-30T18:18:47.217Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: added peer, starting replication: peer=server-cf027ee5-214a-039a-38b7-d28ee558c1d4
2025-10-30T18:18:47.217Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: added peer, starting replication: peer=server-aa55c372-30da-8a30-8a67-1c2b879e8526
2025-10-30T18:18:47.217Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}"
2025-10-30T18:18:47.217Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}"
2025-10-30T18:18:48.222Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:47.217179052 +0000 UTC m=+73.495852950 (1 nodes), 519.150914ms from start of poll, 519.368117ms from cluster start. Timeout at 2025-10-30 18:18:48.222181107 +0000 UTC m=+74.500855004, 1.005002054s after stability
2025-10-30T18:18:48.222Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: starting snapshot up to: index=12
2025-10-30T18:18:48.222Z [INFO]  creating new snapshot: path=/tmp/raft349724726/snapshots/2-12-1761848328222.tmp
2025-10-30T18:18:48.372Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: snapshot complete up to: index=12
2025-10-30T18:18:48.372Z [INFO]  creating new snapshot: path=/tmp/raft349724726/snapshots/2-1013-1761848328372.tmp
2025-10-30T18:18:48.482Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: copied to local snapshot: bytes=71
2025-10-30T18:18:48.482Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: snapshot restore progress: id=2-1013-1761848328372 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:48.482Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: removing all old logs from log store
2025-10-30T18:18:48.482Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: compacting logs: from=1 to=22
2025-10-30T18:18:48.482Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: restored user snapshot: index=1013
2025-10-30T18:18:48.482Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found"
2025-10-30T18:18:48.482Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}"
2025-10-30T18:18:48.482Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found"
2025-10-30T18:18:48.482Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}"
2025-10-30T18:18:48.488Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found"
2025-10-30T18:18:48.488Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found"
2025-10-30T18:18:48.489Z [INFO]  creating new snapshot: path=/tmp/raft2728368928/snapshots/2-1013-1761848328489.tmp
2025-10-30T18:18:48.609Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:48.609Z [INFO]  creating new snapshot: path=/tmp/raft1174517755/snapshots/2-1013-1761848328609.tmp
2025-10-30T18:18:48.609Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:48.609Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: copied to local snapshot: bytes=71
2025-10-30T18:18:48.609Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: snapshot restore progress: id=2-1013-1761848328489 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:48.609Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: removing all old logs from log store
2025-10-30T18:18:48.609Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: compacting logs: from=1 to=22
2025-10-30T18:18:48.609Z [INFO]  server-aa55c372-30da-8a30-8a67-1c2b879e8526: Installed remote snapshot
2025-10-30T18:18:48.609Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: copied to local snapshot: bytes=71
2025-10-30T18:18:48.609Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}"
2025-10-30T18:18:48.609Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:48.610Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: snapshot restore progress: id=2-1013-1761848328609 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:48.610Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: removing all old logs from log store
2025-10-30T18:18:48.610Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: compacting logs: from=1 to=22
2025-10-30T18:18:48.610Z [INFO]  server-cf027ee5-214a-039a-38b7-d28ee558c1d4: Installed remote snapshot
2025-10-30T18:18:48.610Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}"
2025-10-30T18:18:49.615Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:48.609947107 +0000 UTC m=+74.888621006 (2 nodes), 700ns from start of poll, 1.912136173s from cluster start. Timeout at 2025-10-30 18:18:49.614965579 +0000 UTC m=+75.893639476, 1.00501847s after stability
2025-10-30T18:18:49.626Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}"
2025-10-30T18:18:49.626Z [INFO]  server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-10-30T18:18:49.627Z [INFO]  server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 Address:01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8} {Suffrage:Voter ID:server-4dab8c44-22ef-72e7-6a5c-e907d894a332 Address:4dab8c44-22ef-72e7-6a5c-e907d894a332} {Suffrage:Voter ID:server-a4a5259c-dcb5-29a9-5212-37b471217af7 Address:a4a5259c-dcb5-29a9-5212-37b471217af7}]"
2025-10-30T18:18:49.627Z [INFO]  server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8: entering follower state: follower="Node at 01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 [Follower]" leader-address= leader-id=
2025-10-30T18:18:49.627Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 Address:01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8} {Suffrage:Voter ID:server-4dab8c44-22ef-72e7-6a5c-e907d894a332 Address:4dab8c44-22ef-72e7-6a5c-e907d894a332} {Suffrage:Voter ID:server-a4a5259c-dcb5-29a9-5212-37b471217af7 Address:a4a5259c-dcb5-29a9-5212-37b471217af7}]"
2025-10-30T18:18:49.628Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: entering follower state: follower="Node at 4dab8c44-22ef-72e7-6a5c-e907d894a332 [Follower]" leader-address= leader-id=
2025-10-30T18:18:49.628Z [INFO]  server-a4a5259c-dcb5-29a9-5212-37b471217af7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 Address:01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8} {Suffrage:Voter ID:server-4dab8c44-22ef-72e7-6a5c-e907d894a332 Address:4dab8c44-22ef-72e7-6a5c-e907d894a332} {Suffrage:Voter ID:server-a4a5259c-dcb5-29a9-5212-37b471217af7 Address:a4a5259c-dcb5-29a9-5212-37b471217af7}]"
2025-10-30T18:18:49.628Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:49.628Z [INFO]  server-a4a5259c-dcb5-29a9-5212-37b471217af7: entering follower state: follower="Node at a4a5259c-dcb5-29a9-5212-37b471217af7 [Follower]" leader-address= leader-id=
2025-10-30T18:18:50.314Z [WARN]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:50.314Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: entering candidate state: node="Node at 4dab8c44-22ef-72e7-6a5c-e907d894a332 [Candidate]" term=2
2025-10-30T18:18:50.314Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: election won: term=2 tally=2
2025-10-30T18:18:50.314Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: entering leader state: leader="Node at 4dab8c44-22ef-72e7-6a5c-e907d894a332 [Leader]"
2025-10-30T18:18:50.314Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: added peer, starting replication: peer=server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8
2025-10-30T18:18:50.314Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: added peer, starting replication: peer=server-a4a5259c-dcb5-29a9-5212-37b471217af7
2025-10-30T18:18:50.314Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: pipelining replication: peer="{Voter server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8}"
2025-10-30T18:18:50.314Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: pipelining replication: peer="{Voter server-a4a5259c-dcb5-29a9-5212-37b471217af7 a4a5259c-dcb5-29a9-5212-37b471217af7}"
2025-10-30T18:18:51.319Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:50.314833088 +0000 UTC m=+76.593507002 (1 nodes), 686.67718ms from start of poll, 686.98231ms from cluster start. Timeout at 2025-10-30 18:18:51.319836153 +0000 UTC m=+77.598510053, 1.005003051s after stability
2025-10-30T18:18:51.321Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: starting snapshot up to: index=12
2025-10-30T18:18:51.321Z [INFO]  creating new snapshot: path=/tmp/raft3852338432/snapshots/2-12-1761848331321.tmp
2025-10-30T18:18:51.322Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: snapshot complete up to: index=12
2025-10-30T18:18:51.322Z [INFO]  server-092a4917-d92e-1e4c-5b89-2061806a9306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-092a4917-d92e-1e4c-5b89-2061806a9306 Address:092a4917-d92e-1e4c-5b89-2061806a9306} {Suffrage:Voter ID:server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 Address:6a5b0949-b9a8-6d44-7acf-b636fe4e04b2} {Suffrage:Voter ID:server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 Address:874e1b19-2a1d-de7e-4f40-d3b88facc7e0}]"
2025-10-30T18:18:51.322Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-092a4917-d92e-1e4c-5b89-2061806a9306 Address:092a4917-d92e-1e4c-5b89-2061806a9306} {Suffrage:Voter ID:server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 Address:6a5b0949-b9a8-6d44-7acf-b636fe4e04b2} {Suffrage:Voter ID:server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 Address:874e1b19-2a1d-de7e-4f40-d3b88facc7e0}]"
2025-10-30T18:18:51.322Z [INFO]  server-092a4917-d92e-1e4c-5b89-2061806a9306: entering follower state: follower="Node at 092a4917-d92e-1e4c-5b89-2061806a9306 [Follower]" leader-address= leader-id=
2025-10-30T18:18:51.322Z [INFO]  server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-092a4917-d92e-1e4c-5b89-2061806a9306 Address:092a4917-d92e-1e4c-5b89-2061806a9306} {Suffrage:Voter ID:server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 Address:6a5b0949-b9a8-6d44-7acf-b636fe4e04b2} {Suffrage:Voter ID:server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 Address:874e1b19-2a1d-de7e-4f40-d3b88facc7e0}]"
2025-10-30T18:18:51.322Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:51.322Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: entering follower state: follower="Node at 6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 [Follower]" leader-address= leader-id=
2025-10-30T18:18:51.322Z [INFO]  server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: entering follower state: follower="Node at 874e1b19-2a1d-de7e-4f40-d3b88facc7e0 [Follower]" leader-address= leader-id=
2025-10-30T18:18:52.023Z [WARN]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:52.023Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: entering candidate state: node="Node at 6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 [Candidate]" term=2
2025-10-30T18:18:52.023Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: election won: term=2 tally=2
2025-10-30T18:18:52.023Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: entering leader state: leader="Node at 6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 [Leader]"
2025-10-30T18:18:52.023Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: added peer, starting replication: peer=server-092a4917-d92e-1e4c-5b89-2061806a9306
2025-10-30T18:18:52.023Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: added peer, starting replication: peer=server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0
2025-10-30T18:18:52.023Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 874e1b19-2a1d-de7e-4f40-d3b88facc7e0}"
2025-10-30T18:18:52.023Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-092a4917-d92e-1e4c-5b89-2061806a9306 092a4917-d92e-1e4c-5b89-2061806a9306}"
2025-10-30T18:18:53.029Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:52.023839024 +0000 UTC m=+78.302512921 (1 nodes), 700.86965ms from start of poll, 701.071395ms from cluster start. Timeout at 2025-10-30 18:18:53.028848052 +0000 UTC m=+79.307521949, 1.005009028s after stability
2025-10-30T18:18:53.029Z [INFO]  creating new snapshot: path=/tmp/raft1381586826/snapshots/2-1013-1761848333029.tmp
2025-10-30T18:18:53.029Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: copied to local snapshot: bytes=71
2025-10-30T18:18:53.029Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: snapshot restore progress: id=2-1013-1761848333029 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:53.029Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: restored user snapshot: index=1013
2025-10-30T18:18:53.029Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found"
2025-10-30T18:18:53.029Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: aborting pipeline replication: peer="{Voter server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 874e1b19-2a1d-de7e-4f40-d3b88facc7e0}"
2025-10-30T18:18:53.029Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found"
2025-10-30T18:18:53.029Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: aborting pipeline replication: peer="{Voter server-092a4917-d92e-1e4c-5b89-2061806a9306 092a4917-d92e-1e4c-5b89-2061806a9306}"
2025-10-30T18:18:53.036Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found"
2025-10-30T18:18:53.036Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found"
2025-10-30T18:18:53.036Z [INFO]  creating new snapshot: path=/tmp/raft3517113085/snapshots/2-1013-1761848333036.tmp
2025-10-30T18:18:53.036Z [INFO]  creating new snapshot: path=/tmp/raft1919493842/snapshots/2-1013-1761848333036.tmp
2025-10-30T18:18:53.036Z [INFO]  server-092a4917-d92e-1e4c-5b89-2061806a9306: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:53.036Z [INFO]  server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:53.036Z [INFO]  server-092a4917-d92e-1e4c-5b89-2061806a9306: copied to local snapshot: bytes=71
2025-10-30T18:18:53.036Z [INFO]  server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: copied to local snapshot: bytes=71
2025-10-30T18:18:53.037Z [INFO]  server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: snapshot restore progress: id=2-1013-1761848333036 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:53.037Z [INFO]  server-092a4917-d92e-1e4c-5b89-2061806a9306: snapshot restore progress: id=2-1013-1761848333036 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:53.037Z [INFO]  server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: Installed remote snapshot
2025-10-30T18:18:53.037Z [INFO]  server-092a4917-d92e-1e4c-5b89-2061806a9306: Installed remote snapshot
2025-10-30T18:18:53.037Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 874e1b19-2a1d-de7e-4f40-d3b88facc7e0}"
2025-10-30T18:18:53.037Z [INFO]  server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-092a4917-d92e-1e4c-5b89-2061806a9306 092a4917-d92e-1e4c-5b89-2061806a9306}"
2025-10-30T18:18:53.037Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:54.042Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:53.037298963 +0000 UTC m=+79.315972872 (2 nodes), 1.126µs from start of poll, 1.714531346s from cluster start. Timeout at 2025-10-30 18:18:54.042317552 +0000 UTC m=+80.320991449, 1.005018577s after stability
2025-10-30T18:18:54.054Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: aborting pipeline replication: peer="{Voter server-a4a5259c-dcb5-29a9-5212-37b471217af7 a4a5259c-dcb5-29a9-5212-37b471217af7}"
2025-10-30T18:18:54.054Z [INFO]  server-4dab8c44-22ef-72e7-6a5c-e907d894a332: aborting pipeline replication: peer="{Voter server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-10-30T18:18:54.057Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fee6882-dfe8-3952-a66f-c947a53ddfe5 Address:6fee6882-dfe8-3952-a66f-c947a53ddfe5} {Suffrage:Voter ID:server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc Address:b4aa64b9-6c20-40f0-17ef-35a28b344ffc} {Suffrage:Voter ID:server-4e16e616-bafa-1c9f-6408-e796c971cc8b Address:4e16e616-bafa-1c9f-6408-e796c971cc8b}]"
2025-10-30T18:18:54.057Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: entering follower state: follower="Node at 6fee6882-dfe8-3952-a66f-c947a53ddfe5 [Follower]" leader-address= leader-id=
2025-10-30T18:18:54.057Z [INFO]  server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fee6882-dfe8-3952-a66f-c947a53ddfe5 Address:6fee6882-dfe8-3952-a66f-c947a53ddfe5} {Suffrage:Voter ID:server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc Address:b4aa64b9-6c20-40f0-17ef-35a28b344ffc} {Suffrage:Voter ID:server-4e16e616-bafa-1c9f-6408-e796c971cc8b Address:4e16e616-bafa-1c9f-6408-e796c971cc8b}]"
2025-10-30T18:18:54.057Z [INFO]  server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc: entering follower state: follower="Node at b4aa64b9-6c20-40f0-17ef-35a28b344ffc [Follower]" leader-address= leader-id=
2025-10-30T18:18:54.057Z [INFO]  server-4e16e616-bafa-1c9f-6408-e796c971cc8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fee6882-dfe8-3952-a66f-c947a53ddfe5 Address:6fee6882-dfe8-3952-a66f-c947a53ddfe5} {Suffrage:Voter ID:server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc Address:b4aa64b9-6c20-40f0-17ef-35a28b344ffc} {Suffrage:Voter ID:server-4e16e616-bafa-1c9f-6408-e796c971cc8b Address:4e16e616-bafa-1c9f-6408-e796c971cc8b}]"
2025-10-30T18:18:54.057Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:54.057Z [INFO]  server-4e16e616-bafa-1c9f-6408-e796c971cc8b: entering follower state: follower="Node at 4e16e616-bafa-1c9f-6408-e796c971cc8b [Follower]" leader-address= leader-id=
2025-10-30T18:18:54.680Z [WARN]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:54.680Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: entering candidate state: node="Node at 6fee6882-dfe8-3952-a66f-c947a53ddfe5 [Candidate]" term=2
2025-10-30T18:18:54.681Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: election won: term=2 tally=2
2025-10-30T18:18:54.681Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: entering leader state: leader="Node at 6fee6882-dfe8-3952-a66f-c947a53ddfe5 [Leader]"
2025-10-30T18:18:54.681Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: added peer, starting replication: peer=server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc
2025-10-30T18:18:54.681Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: added peer, starting replication: peer=server-4e16e616-bafa-1c9f-6408-e796c971cc8b
2025-10-30T18:18:54.681Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: pipelining replication: peer="{Voter server-4e16e616-bafa-1c9f-6408-e796c971cc8b 4e16e616-bafa-1c9f-6408-e796c971cc8b}"
2025-10-30T18:18:54.681Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: pipelining replication: peer="{Voter server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc b4aa64b9-6c20-40f0-17ef-35a28b344ffc}"
2025-10-30T18:18:55.686Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:54.681172406 +0000 UTC m=+80.959846304 (1 nodes), 623.371633ms from start of poll, 623.701198ms from cluster start. Timeout at 2025-10-30 18:18:55.686178178 +0000 UTC m=+81.964852089, 1.005005785s after stability
2025-10-30T18:18:55.687Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: starting snapshot up to: index=12
2025-10-30T18:18:55.687Z [INFO]  creating new snapshot: path=/tmp/raft3309214673/snapshots/2-12-1761848335687.tmp
2025-10-30T18:18:55.687Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: snapshot complete up to: index=12
2025-10-30T18:18:55.688Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 Address:2cf29b51-4d7d-232a-cc8f-b7e4b5402734} {Suffrage:Voter ID:server-3759ae26-f796-c659-a13d-88d1d2088f31 Address:3759ae26-f796-c659-a13d-88d1d2088f31} {Suffrage:Voter ID:server-b24e3d03-bd41-bd39-f50e-410926f466a2 Address:b24e3d03-bd41-bd39-f50e-410926f466a2}]"
2025-10-30T18:18:55.689Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: entering follower state: follower="Node at 2cf29b51-4d7d-232a-cc8f-b7e4b5402734 [Follower]" leader-address= leader-id=
2025-10-30T18:18:55.689Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 Address:2cf29b51-4d7d-232a-cc8f-b7e4b5402734} {Suffrage:Voter ID:server-3759ae26-f796-c659-a13d-88d1d2088f31 Address:3759ae26-f796-c659-a13d-88d1d2088f31} {Suffrage:Voter ID:server-b24e3d03-bd41-bd39-f50e-410926f466a2 Address:b24e3d03-bd41-bd39-f50e-410926f466a2}]"
2025-10-30T18:18:55.689Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: entering follower state: follower="Node at 3759ae26-f796-c659-a13d-88d1d2088f31 [Follower]" leader-address= leader-id=
2025-10-30T18:18:55.689Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 Address:2cf29b51-4d7d-232a-cc8f-b7e4b5402734} {Suffrage:Voter ID:server-3759ae26-f796-c659-a13d-88d1d2088f31 Address:3759ae26-f796-c659-a13d-88d1d2088f31} {Suffrage:Voter ID:server-b24e3d03-bd41-bd39-f50e-410926f466a2 Address:b24e3d03-bd41-bd39-f50e-410926f466a2}]"
2025-10-30T18:18:55.689Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:55.689Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: entering follower state: follower="Node at b24e3d03-bd41-bd39-f50e-410926f466a2 [Follower]" leader-address= leader-id=
2025-10-30T18:18:56.502Z [WARN]  server-3759ae26-f796-c659-a13d-88d1d2088f31: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:56.502Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: entering candidate state: node="Node at 3759ae26-f796-c659-a13d-88d1d2088f31 [Candidate]" term=2
2025-10-30T18:18:56.502Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: election won: term=2 tally=2
2025-10-30T18:18:56.502Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: entering leader state: leader="Node at 3759ae26-f796-c659-a13d-88d1d2088f31 [Leader]"
2025-10-30T18:18:56.502Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: added peer, starting replication: peer=server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734
2025-10-30T18:18:56.502Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: added peer, starting replication: peer=server-b24e3d03-bd41-bd39-f50e-410926f466a2
2025-10-30T18:18:56.502Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 2cf29b51-4d7d-232a-cc8f-b7e4b5402734}"
2025-10-30T18:18:56.502Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-b24e3d03-bd41-bd39-f50e-410926f466a2 b24e3d03-bd41-bd39-f50e-410926f466a2}"
2025-10-30T18:18:57.508Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:56.502272751 +0000 UTC m=+82.780946648 (1 nodes), 813.090473ms from start of poll, 813.350554ms from cluster start. Timeout at 2025-10-30 18:18:57.507276137 +0000 UTC m=+83.785950036, 1.005003388s after stability
2025-10-30T18:18:57.508Z [INFO]  creating new snapshot: path=/tmp/raft3170197592/snapshots/2-1013-1761848337508.tmp
2025-10-30T18:18:57.573Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: copied to local snapshot: bytes=71
2025-10-30T18:18:57.573Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: snapshot restore progress: id=2-1013-1761848337508 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:57.573Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: removing all old logs from log store
2025-10-30T18:18:57.573Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: compacting logs: from=1 to=2
2025-10-30T18:18:57.573Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: restored user snapshot: index=1013
2025-10-30T18:18:57.573Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found"
2025-10-30T18:18:57.573Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: aborting pipeline replication: peer="{Voter server-b24e3d03-bd41-bd39-f50e-410926f466a2 b24e3d03-bd41-bd39-f50e-410926f466a2}"
2025-10-30T18:18:57.573Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found"
2025-10-30T18:18:57.573Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: aborting pipeline replication: peer="{Voter server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 2cf29b51-4d7d-232a-cc8f-b7e4b5402734}"
2025-10-30T18:18:57.581Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found"
2025-10-30T18:18:57.581Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found"
2025-10-30T18:18:57.581Z [INFO]  creating new snapshot: path=/tmp/raft1324536089/snapshots/2-1013-1761848337581.tmp
2025-10-30T18:18:57.581Z [INFO]  creating new snapshot: path=/tmp/raft2252923619/snapshots/2-1013-1761848337581.tmp
2025-10-30T18:18:57.681Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:57.681Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:57.681Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: copied to local snapshot: bytes=71
2025-10-30T18:18:57.681Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: copied to local snapshot: bytes=71
2025-10-30T18:18:57.682Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: snapshot restore progress: id=2-1013-1761848337581 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:57.682Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: removing all old logs from log store
2025-10-30T18:18:57.682Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: compacting logs: from=1 to=2
2025-10-30T18:18:57.682Z [INFO]  server-b24e3d03-bd41-bd39-f50e-410926f466a2: Installed remote snapshot
2025-10-30T18:18:57.682Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: snapshot restore progress: id=2-1013-1761848337581 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:18:57.682Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: removing all old logs from log store
2025-10-30T18:18:57.682Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: compacting logs: from=1 to=2
2025-10-30T18:18:57.682Z [INFO]  server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: Installed remote snapshot
2025-10-30T18:18:57.682Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-b24e3d03-bd41-bd39-f50e-410926f466a2 b24e3d03-bd41-bd39-f50e-410926f466a2}"
2025-10-30T18:18:57.682Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:18:57.682Z [INFO]  server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 2cf29b51-4d7d-232a-cc8f-b7e4b5402734}"
2025-10-30T18:18:58.687Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:18:57.682206485 +0000 UTC m=+83.960880389 (2 nodes), 899ns from start of poll, 1.993284295s from cluster start. Timeout at 2025-10-30 18:18:58.687209741 +0000 UTC m=+84.965883638, 1.005003249s after stability
2025-10-30T18:18:58.698Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: aborting pipeline replication: peer="{Voter server-4e16e616-bafa-1c9f-6408-e796c971cc8b 4e16e616-bafa-1c9f-6408-e796c971cc8b}"
2025-10-30T18:18:58.698Z [INFO]  server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: aborting pipeline replication: peer="{Voter server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc b4aa64b9-6c20-40f0-17ef-35a28b344ffc}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-10-30T18:18:58.699Z [INFO]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 Address:5467dcbf-11b5-38b4-ed79-f2eb3917af63} {Suffrage:Voter ID:server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b Address:56fb0e4e-5a40-4cd5-42ec-41e1050b123b} {Suffrage:Voter ID:server-ac35575a-b6e9-043a-3e9b-d67d217f335a Address:ac35575a-b6e9-043a-3e9b-d67d217f335a}]"
2025-10-30T18:18:58.699Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 Address:5467dcbf-11b5-38b4-ed79-f2eb3917af63} {Suffrage:Voter ID:server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b Address:56fb0e4e-5a40-4cd5-42ec-41e1050b123b} {Suffrage:Voter ID:server-ac35575a-b6e9-043a-3e9b-d67d217f335a Address:ac35575a-b6e9-043a-3e9b-d67d217f335a}]"
2025-10-30T18:18:58.699Z [INFO]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: entering follower state: follower="Node at 5467dcbf-11b5-38b4-ed79-f2eb3917af63 [Follower]" leader-address= leader-id=
2025-10-30T18:18:58.699Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: entering follower state: follower="Node at 56fb0e4e-5a40-4cd5-42ec-41e1050b123b [Follower]" leader-address= leader-id=
2025-10-30T18:18:58.699Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 Address:5467dcbf-11b5-38b4-ed79-f2eb3917af63} {Suffrage:Voter ID:server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b Address:56fb0e4e-5a40-4cd5-42ec-41e1050b123b} {Suffrage:Voter ID:server-ac35575a-b6e9-043a-3e9b-d67d217f335a Address:ac35575a-b6e9-043a-3e9b-d67d217f335a}]"
2025-10-30T18:18:58.699Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:18:58.699Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: entering follower state: follower="Node at ac35575a-b6e9-043a-3e9b-d67d217f335a [Follower]" leader-address= leader-id=
2025-10-30T18:18:59.269Z [WARN]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:59.269Z [WARN]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:18:59.269Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: entering candidate state: node="Node at ac35575a-b6e9-043a-3e9b-d67d217f335a [Candidate]" term=2
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: entering candidate state: node="Node at 56fb0e4e-5a40-4cd5-42ec-41e1050b123b [Candidate]" term=2
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: duplicate requestVote for same term: term=2
2025-10-30T18:18:59.269Z [INFO]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: duplicate requestVote for same term: term=2
2025-10-30T18:18:59.269Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: duplicate requestVote for same term: term=2
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: election won: term=2 tally=2
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: entering leader state: leader="Node at 56fb0e4e-5a40-4cd5-42ec-41e1050b123b [Leader]"
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: added peer, starting replication: peer=server-5467dcbf-11b5-38b4-ed79-f2eb3917af63
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: added peer, starting replication: peer=server-ac35575a-b6e9-043a-3e9b-d67d217f335a
2025-10-30T18:18:59.269Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: entering follower state: follower="Node at ac35575a-b6e9-043a-3e9b-d67d217f335a [Follower]" leader-address=56fb0e4e-5a40-4cd5-42ec-41e1050b123b leader-id=server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}"
2025-10-30T18:18:59.269Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}"
2025-10-30T18:19:00.275Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:18:59.269673478 +0000 UTC m=+85.548347388 (1 nodes), 569.679824ms from start of poll, 569.960286ms from cluster start. Timeout at 2025-10-30 18:19:00.274676734 +0000 UTC m=+86.553350631, 1.005003243s after stability
2025-10-30T18:19:00.276Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: starting snapshot up to: index=12
2025-10-30T18:19:00.276Z [INFO]  creating new snapshot: path=/tmp/raft846765429/snapshots/2-12-1761848340276.tmp
2025-10-30T18:19:00.410Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: snapshot complete up to: index=12
2025-10-30T18:19:00.410Z [INFO]  creating new snapshot: path=/tmp/raft846765429/snapshots/2-10013-1761848340410.tmp
2025-10-30T18:19:00.411Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: copied to local snapshot: bytes=71
2025-10-30T18:19:00.411Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: snapshot restore progress: id=2-10013-1761848340410 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:00.411Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: restored user snapshot: index=10013
2025-10-30T18:19:00.411Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found"
2025-10-30T18:19:00.411Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}"
2025-10-30T18:19:00.411Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found"
2025-10-30T18:19:00.411Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}"
2025-10-30T18:19:00.418Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found"
2025-10-30T18:19:00.418Z [INFO]  creating new snapshot: path=/tmp/raft1870409754/snapshots/2-10013-1761848340418.tmp
2025-10-30T18:19:00.418Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found"
2025-10-30T18:19:00.418Z [INFO]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:00.418Z [INFO]  creating new snapshot: path=/tmp/raft3121484863/snapshots/2-10013-1761848340418.tmp
2025-10-30T18:19:00.419Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:00.747Z [INFO]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: copied to local snapshot: bytes=71
2025-10-30T18:19:00.747Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: copied to local snapshot: bytes=71
2025-10-30T18:19:00.747Z [INFO]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: snapshot restore progress: id=2-10013-1761848340418 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:00.747Z [INFO]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: Installed remote snapshot
2025-10-30T18:19:00.747Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: snapshot restore progress: id=2-10013-1761848340418 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:00.747Z [INFO]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: Installed remote snapshot
2025-10-30T18:19:00.747Z [WARN]  server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: skipping application of old log: index=22
2025-10-30T18:19:00.747Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}"
2025-10-30T18:19:00.747Z [WARN]  server-ac35575a-b6e9-043a-3e9b-d67d217f335a: skipping application of old log: index=22
2025-10-30T18:19:00.747Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:00.747Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}"
2025-10-30T18:19:01.752Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:00.747541416 +0000 UTC m=+87.026215325 (2 nodes), 587ns from start of poll, 2.047828223s from cluster start. Timeout at 2025-10-30 18:19:01.752572413 +0000 UTC m=+88.031246310, 1.005030985s after stability
2025-10-30T18:19:01.764Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}"
2025-10-30T18:19:01.764Z [INFO]  server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-10-30T18:19:01.766Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 Address:6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3} {Suffrage:Voter ID:server-cdf886e8-c48b-e275-5aa2-6020318b4214 Address:cdf886e8-c48b-e275-5aa2-6020318b4214} {Suffrage:Voter ID:server-74fdb010-82bc-335b-07f0-7fbd3557c721 Address:74fdb010-82bc-335b-07f0-7fbd3557c721}]"
2025-10-30T18:19:01.766Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 Address:6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3} {Suffrage:Voter ID:server-cdf886e8-c48b-e275-5aa2-6020318b4214 Address:cdf886e8-c48b-e275-5aa2-6020318b4214} {Suffrage:Voter ID:server-74fdb010-82bc-335b-07f0-7fbd3557c721 Address:74fdb010-82bc-335b-07f0-7fbd3557c721}]"
2025-10-30T18:19:01.766Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: entering follower state: follower="Node at 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 [Follower]" leader-address= leader-id=
2025-10-30T18:19:01.766Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: entering follower state: follower="Node at cdf886e8-c48b-e275-5aa2-6020318b4214 [Follower]" leader-address= leader-id=
2025-10-30T18:19:01.766Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 Address:6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3} {Suffrage:Voter ID:server-cdf886e8-c48b-e275-5aa2-6020318b4214 Address:cdf886e8-c48b-e275-5aa2-6020318b4214} {Suffrage:Voter ID:server-74fdb010-82bc-335b-07f0-7fbd3557c721 Address:74fdb010-82bc-335b-07f0-7fbd3557c721}]"
2025-10-30T18:19:01.766Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:01.766Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: entering follower state: follower="Node at 74fdb010-82bc-335b-07f0-7fbd3557c721 [Follower]" leader-address= leader-id=
2025-10-30T18:19:02.442Z [WARN]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:02.442Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: entering candidate state: node="Node at 74fdb010-82bc-335b-07f0-7fbd3557c721 [Candidate]" term=2
2025-10-30T18:19:02.442Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: election won: term=2 tally=2
2025-10-30T18:19:02.442Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: entering leader state: leader="Node at 74fdb010-82bc-335b-07f0-7fbd3557c721 [Leader]"
2025-10-30T18:19:02.442Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: added peer, starting replication: peer=server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3
2025-10-30T18:19:02.442Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: added peer, starting replication: peer=server-cdf886e8-c48b-e275-5aa2-6020318b4214
2025-10-30T18:19:02.442Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}"
2025-10-30T18:19:02.442Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}"
2025-10-30T18:19:03.448Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:02.442878577 +0000 UTC m=+88.721552478 (1 nodes), 676.674323ms from start of poll, 676.971386ms from cluster start. Timeout at 2025-10-30 18:19:03.447881653 +0000 UTC m=+89.726555550, 1.005003072s after stability
2025-10-30T18:19:03.448Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: starting snapshot up to: index=12
2025-10-30T18:19:03.448Z [INFO]  creating new snapshot: path=/tmp/raft2791505475/snapshots/2-12-1761848343448.tmp
2025-10-30T18:19:03.502Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: snapshot complete up to: index=12
2025-10-30T18:19:03.503Z [INFO]  creating new snapshot: path=/tmp/raft2791505475/snapshots/2-10013-1761848343503.tmp
2025-10-30T18:19:03.503Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: copied to local snapshot: bytes=71
2025-10-30T18:19:03.503Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: snapshot restore progress: id=2-10013-1761848343503 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:03.504Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: removing all old logs from log store
2025-10-30T18:19:03.504Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: compacting logs: from=1 to=22
2025-10-30T18:19:03.504Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: restored user snapshot: index=10013
2025-10-30T18:19:03.504Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found"
2025-10-30T18:19:03.504Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}"
2025-10-30T18:19:03.504Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found"
2025-10-30T18:19:03.504Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}"
2025-10-30T18:19:03.509Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found"
2025-10-30T18:19:03.509Z [INFO]  creating new snapshot: path=/tmp/raft1624111273/snapshots/2-10013-1761848343509.tmp
2025-10-30T18:19:03.510Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:03.510Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: copied to local snapshot: bytes=71
2025-10-30T18:19:03.510Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: snapshot restore progress: id=2-10013-1761848343509 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:03.510Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: removing all old logs from log store
2025-10-30T18:19:03.510Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: compacting logs: from=1 to=22
2025-10-30T18:19:03.510Z [INFO]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: Installed remote snapshot
2025-10-30T18:19:03.510Z [WARN]  server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: skipping application of old log: index=22
2025-10-30T18:19:03.510Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}"
2025-10-30T18:19:03.510Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:03.511Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found"
2025-10-30T18:19:03.512Z [INFO]  creating new snapshot: path=/tmp/raft2596539982/snapshots/2-10013-1761848343512.tmp
2025-10-30T18:19:03.512Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:03.513Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: copied to local snapshot: bytes=71
2025-10-30T18:19:03.513Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: snapshot restore progress: id=2-10013-1761848343512 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:03.513Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: removing all old logs from log store
2025-10-30T18:19:03.513Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: compacting logs: from=1 to=22
2025-10-30T18:19:03.513Z [INFO]  server-cdf886e8-c48b-e275-5aa2-6020318b4214: Installed remote snapshot
2025-10-30T18:19:03.513Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}"
2025-10-30T18:19:04.516Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:03.510933327 +0000 UTC m=+89.789607224 (2 nodes), 661ns from start of poll, 1.745026132s from cluster start. Timeout at 2025-10-30 18:19:04.515963853 +0000 UTC m=+90.794637766, 1.005030542s after stability
2025-10-30T18:19:04.528Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}"
2025-10-30T18:19:04.528Z [INFO]  server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-10-30T18:19:04.529Z [INFO]  server-a12d35d8-4926-c326-572f-aec44863f599: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12d35d8-4926-c326-572f-aec44863f599 Address:a12d35d8-4926-c326-572f-aec44863f599} {Suffrage:Voter ID:server-38674a90-e3e6-97b6-9933-c36423fb6c83 Address:38674a90-e3e6-97b6-9933-c36423fb6c83} {Suffrage:Voter ID:server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 Address:0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3}]"
2025-10-30T18:19:04.529Z [INFO]  server-a12d35d8-4926-c326-572f-aec44863f599: entering follower state: follower="Node at a12d35d8-4926-c326-572f-aec44863f599 [Follower]" leader-address= leader-id=
2025-10-30T18:19:04.529Z [INFO]  server-38674a90-e3e6-97b6-9933-c36423fb6c83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12d35d8-4926-c326-572f-aec44863f599 Address:a12d35d8-4926-c326-572f-aec44863f599} {Suffrage:Voter ID:server-38674a90-e3e6-97b6-9933-c36423fb6c83 Address:38674a90-e3e6-97b6-9933-c36423fb6c83} {Suffrage:Voter ID:server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 Address:0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3}]"
2025-10-30T18:19:04.529Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12d35d8-4926-c326-572f-aec44863f599 Address:a12d35d8-4926-c326-572f-aec44863f599} {Suffrage:Voter ID:server-38674a90-e3e6-97b6-9933-c36423fb6c83 Address:38674a90-e3e6-97b6-9933-c36423fb6c83} {Suffrage:Voter ID:server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 Address:0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3}]"
2025-10-30T18:19:04.529Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:04.529Z [INFO]  server-38674a90-e3e6-97b6-9933-c36423fb6c83: entering follower state: follower="Node at 38674a90-e3e6-97b6-9933-c36423fb6c83 [Follower]" leader-address= leader-id=
2025-10-30T18:19:04.529Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: entering follower state: follower="Node at 0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 [Follower]" leader-address= leader-id=
2025-10-30T18:19:05.087Z [WARN]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:05.087Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: entering candidate state: node="Node at 0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 [Candidate]" term=2
2025-10-30T18:19:05.087Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: election won: term=2 tally=2
2025-10-30T18:19:05.087Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: entering leader state: leader="Node at 0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 [Leader]"
2025-10-30T18:19:05.087Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: added peer, starting replication: peer=server-a12d35d8-4926-c326-572f-aec44863f599
2025-10-30T18:19:05.087Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: added peer, starting replication: peer=server-38674a90-e3e6-97b6-9933-c36423fb6c83
2025-10-30T18:19:05.088Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: pipelining replication: peer="{Voter server-a12d35d8-4926-c326-572f-aec44863f599 a12d35d8-4926-c326-572f-aec44863f599}"
2025-10-30T18:19:05.088Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: pipelining replication: peer="{Voter server-38674a90-e3e6-97b6-9933-c36423fb6c83 38674a90-e3e6-97b6-9933-c36423fb6c83}"
2025-10-30T18:19:06.093Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:05.087966859 +0000 UTC m=+91.366640756 (1 nodes), 558.393062ms from start of poll, 558.616491ms from cluster start. Timeout at 2025-10-30 18:19:06.09296983 +0000 UTC m=+92.371643737, 1.005002981s after stability
2025-10-30T18:19:06.094Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: starting snapshot up to: index=12
2025-10-30T18:19:06.094Z [INFO]  creating new snapshot: path=/tmp/raft142849237/snapshots/2-12-1761848346094.tmp
2025-10-30T18:19:06.095Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: snapshot complete up to: index=12
2025-10-30T18:19:06.096Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5935a42d-564f-f451-1c45-3b1ead50c3f3 Address:5935a42d-564f-f451-1c45-3b1ead50c3f3} {Suffrage:Voter ID:server-54721090-29f0-cb5c-2bed-832f5d577bb8 Address:54721090-29f0-cb5c-2bed-832f5d577bb8} {Suffrage:Voter ID:server-3ea4006e-45c7-de28-d0a5-7e280e66c954 Address:3ea4006e-45c7-de28-d0a5-7e280e66c954}]"
2025-10-30T18:19:06.096Z [INFO]  server-54721090-29f0-cb5c-2bed-832f5d577bb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5935a42d-564f-f451-1c45-3b1ead50c3f3 Address:5935a42d-564f-f451-1c45-3b1ead50c3f3} {Suffrage:Voter ID:server-54721090-29f0-cb5c-2bed-832f5d577bb8 Address:54721090-29f0-cb5c-2bed-832f5d577bb8} {Suffrage:Voter ID:server-3ea4006e-45c7-de28-d0a5-7e280e66c954 Address:3ea4006e-45c7-de28-d0a5-7e280e66c954}]"
2025-10-30T18:19:06.096Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: entering follower state: follower="Node at 5935a42d-564f-f451-1c45-3b1ead50c3f3 [Follower]" leader-address= leader-id=
2025-10-30T18:19:06.096Z [INFO]  server-54721090-29f0-cb5c-2bed-832f5d577bb8: entering follower state: follower="Node at 54721090-29f0-cb5c-2bed-832f5d577bb8 [Follower]" leader-address= leader-id=
2025-10-30T18:19:06.096Z [INFO]  server-3ea4006e-45c7-de28-d0a5-7e280e66c954: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5935a42d-564f-f451-1c45-3b1ead50c3f3 Address:5935a42d-564f-f451-1c45-3b1ead50c3f3} {Suffrage:Voter ID:server-54721090-29f0-cb5c-2bed-832f5d577bb8 Address:54721090-29f0-cb5c-2bed-832f5d577bb8} {Suffrage:Voter ID:server-3ea4006e-45c7-de28-d0a5-7e280e66c954 Address:3ea4006e-45c7-de28-d0a5-7e280e66c954}]"
2025-10-30T18:19:06.096Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:06.096Z [INFO]  server-3ea4006e-45c7-de28-d0a5-7e280e66c954: entering follower state: follower="Node at 3ea4006e-45c7-de28-d0a5-7e280e66c954 [Follower]" leader-address= leader-id=
2025-10-30T18:19:06.624Z [WARN]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:06.624Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: entering candidate state: node="Node at 5935a42d-564f-f451-1c45-3b1ead50c3f3 [Candidate]" term=2
2025-10-30T18:19:06.624Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: election won: term=2 tally=2
2025-10-30T18:19:06.624Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: entering leader state: leader="Node at 5935a42d-564f-f451-1c45-3b1ead50c3f3 [Leader]"
2025-10-30T18:19:06.624Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: added peer, starting replication: peer=server-54721090-29f0-cb5c-2bed-832f5d577bb8
2025-10-30T18:19:06.624Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: added peer, starting replication: peer=server-3ea4006e-45c7-de28-d0a5-7e280e66c954
2025-10-30T18:19:06.624Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-3ea4006e-45c7-de28-d0a5-7e280e66c954 3ea4006e-45c7-de28-d0a5-7e280e66c954}"
2025-10-30T18:19:06.624Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-54721090-29f0-cb5c-2bed-832f5d577bb8 54721090-29f0-cb5c-2bed-832f5d577bb8}"
2025-10-30T18:19:07.630Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:06.624781466 +0000 UTC m=+92.903455370 (1 nodes), 528.547614ms from start of poll, 528.772642ms from cluster start. Timeout at 2025-10-30 18:19:07.629782667 +0000 UTC m=+93.908456565, 1.005001195s after stability
2025-10-30T18:19:07.630Z [INFO]  creating new snapshot: path=/tmp/raft914815851/snapshots/2-10013-1761848347630.tmp
2025-10-30T18:19:07.631Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: copied to local snapshot: bytes=71
2025-10-30T18:19:07.631Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: snapshot restore progress: id=2-10013-1761848347630 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:07.631Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: restored user snapshot: index=10013
2025-10-30T18:19:07.631Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found"
2025-10-30T18:19:07.631Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found"
2025-10-30T18:19:07.631Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: aborting pipeline replication: peer="{Voter server-54721090-29f0-cb5c-2bed-832f5d577bb8 54721090-29f0-cb5c-2bed-832f5d577bb8}"
2025-10-30T18:19:07.631Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: aborting pipeline replication: peer="{Voter server-3ea4006e-45c7-de28-d0a5-7e280e66c954 3ea4006e-45c7-de28-d0a5-7e280e66c954}"
2025-10-30T18:19:07.640Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found"
2025-10-30T18:19:07.640Z [INFO]  creating new snapshot: path=/tmp/raft3752347853/snapshots/2-10013-1761848347640.tmp
2025-10-30T18:19:07.641Z [INFO]  server-3ea4006e-45c7-de28-d0a5-7e280e66c954: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:07.641Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found"
2025-10-30T18:19:07.641Z [INFO]  creating new snapshot: path=/tmp/raft3719230373/snapshots/2-10013-1761848347641.tmp
2025-10-30T18:19:07.641Z [INFO]  server-3ea4006e-45c7-de28-d0a5-7e280e66c954: copied to local snapshot: bytes=71
2025-10-30T18:19:07.641Z [INFO]  server-3ea4006e-45c7-de28-d0a5-7e280e66c954: snapshot restore progress: id=2-10013-1761848347640 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:07.641Z [INFO]  server-3ea4006e-45c7-de28-d0a5-7e280e66c954: Installed remote snapshot
2025-10-30T18:19:07.641Z [INFO]  server-54721090-29f0-cb5c-2bed-832f5d577bb8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:07.641Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-3ea4006e-45c7-de28-d0a5-7e280e66c954 3ea4006e-45c7-de28-d0a5-7e280e66c954}"
2025-10-30T18:19:07.641Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:07.641Z [INFO]  server-54721090-29f0-cb5c-2bed-832f5d577bb8: copied to local snapshot: bytes=71
2025-10-30T18:19:07.642Z [INFO]  server-54721090-29f0-cb5c-2bed-832f5d577bb8: snapshot restore progress: id=2-10013-1761848347641 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:07.642Z [INFO]  server-54721090-29f0-cb5c-2bed-832f5d577bb8: Installed remote snapshot
2025-10-30T18:19:07.642Z [INFO]  server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-54721090-29f0-cb5c-2bed-832f5d577bb8 54721090-29f0-cb5c-2bed-832f5d577bb8}"
2025-10-30T18:19:08.647Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:07.641925303 +0000 UTC m=+93.920599202 (2 nodes), 979ns from start of poll, 1.545916474s from cluster start. Timeout at 2025-10-30 18:19:08.646949102 +0000 UTC m=+94.925622999, 1.005023797s after stability
2025-10-30T18:19:08.659Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: aborting pipeline replication: peer="{Voter server-a12d35d8-4926-c326-572f-aec44863f599 a12d35d8-4926-c326-572f-aec44863f599}"
2025-10-30T18:19:08.659Z [INFO]  server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: aborting pipeline replication: peer="{Voter server-38674a90-e3e6-97b6-9933-c36423fb6c83 38674a90-e3e6-97b6-9933-c36423fb6c83}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-10-30T18:19:08.660Z [INFO]  server-c275bf99-ff0d-a74b-3547-f4d8386c899d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c275bf99-ff0d-a74b-3547-f4d8386c899d Address:c275bf99-ff0d-a74b-3547-f4d8386c899d} {Suffrage:Voter ID:server-fd78e202-034d-df7b-0446-3c02197f42a1 Address:fd78e202-034d-df7b-0446-3c02197f42a1} {Suffrage:Voter ID:server-c495c0bd-a40e-b036-c40c-82fa0659a361 Address:c495c0bd-a40e-b036-c40c-82fa0659a361}]"
2025-10-30T18:19:08.660Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c275bf99-ff0d-a74b-3547-f4d8386c899d Address:c275bf99-ff0d-a74b-3547-f4d8386c899d} {Suffrage:Voter ID:server-fd78e202-034d-df7b-0446-3c02197f42a1 Address:fd78e202-034d-df7b-0446-3c02197f42a1} {Suffrage:Voter ID:server-c495c0bd-a40e-b036-c40c-82fa0659a361 Address:c495c0bd-a40e-b036-c40c-82fa0659a361}]"
2025-10-30T18:19:08.660Z [INFO]  server-c275bf99-ff0d-a74b-3547-f4d8386c899d: entering follower state: follower="Node at c275bf99-ff0d-a74b-3547-f4d8386c899d [Follower]" leader-address= leader-id=
2025-10-30T18:19:08.660Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: entering follower state: follower="Node at fd78e202-034d-df7b-0446-3c02197f42a1 [Follower]" leader-address= leader-id=
2025-10-30T18:19:08.660Z [INFO]  server-c495c0bd-a40e-b036-c40c-82fa0659a361: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c275bf99-ff0d-a74b-3547-f4d8386c899d Address:c275bf99-ff0d-a74b-3547-f4d8386c899d} {Suffrage:Voter ID:server-fd78e202-034d-df7b-0446-3c02197f42a1 Address:fd78e202-034d-df7b-0446-3c02197f42a1} {Suffrage:Voter ID:server-c495c0bd-a40e-b036-c40c-82fa0659a361 Address:c495c0bd-a40e-b036-c40c-82fa0659a361}]"
2025-10-30T18:19:08.660Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:08.660Z [INFO]  server-c495c0bd-a40e-b036-c40c-82fa0659a361: entering follower state: follower="Node at c495c0bd-a40e-b036-c40c-82fa0659a361 [Follower]" leader-address= leader-id=
2025-10-30T18:19:09.241Z [WARN]  server-fd78e202-034d-df7b-0446-3c02197f42a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:09.241Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: entering candidate state: node="Node at fd78e202-034d-df7b-0446-3c02197f42a1 [Candidate]" term=2
2025-10-30T18:19:09.241Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: election won: term=2 tally=2
2025-10-30T18:19:09.241Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: entering leader state: leader="Node at fd78e202-034d-df7b-0446-3c02197f42a1 [Leader]"
2025-10-30T18:19:09.241Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: added peer, starting replication: peer=server-c275bf99-ff0d-a74b-3547-f4d8386c899d
2025-10-30T18:19:09.241Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: added peer, starting replication: peer=server-c495c0bd-a40e-b036-c40c-82fa0659a361
2025-10-30T18:19:09.241Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: pipelining replication: peer="{Voter server-c495c0bd-a40e-b036-c40c-82fa0659a361 c495c0bd-a40e-b036-c40c-82fa0659a361}"
2025-10-30T18:19:09.242Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: pipelining replication: peer="{Voter server-c275bf99-ff0d-a74b-3547-f4d8386c899d c275bf99-ff0d-a74b-3547-f4d8386c899d}"
2025-10-30T18:19:10.247Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:09.241952763 +0000 UTC m=+95.520626660 (1 nodes), 581.076255ms from start of poll, 581.318129ms from cluster start. Timeout at 2025-10-30 18:19:10.246954926 +0000 UTC m=+96.525628823, 1.005002163s after stability
2025-10-30T18:19:10.248Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: starting snapshot up to: index=12
2025-10-30T18:19:10.248Z [INFO]  creating new snapshot: path=/tmp/raft265069148/snapshots/2-12-1761848350248.tmp
2025-10-30T18:19:10.283Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: snapshot complete up to: index=12
2025-10-30T18:19:10.284Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4487d22-db28-ad97-459e-bdc76e06a8fc Address:d4487d22-db28-ad97-459e-bdc76e06a8fc} {Suffrage:Voter ID:server-7961c63f-0644-b43d-47d3-1c6aed691c58 Address:7961c63f-0644-b43d-47d3-1c6aed691c58} {Suffrage:Voter ID:server-d97fd847-4517-e92e-2b4c-3b09900a54e0 Address:d97fd847-4517-e92e-2b4c-3b09900a54e0}]"
2025-10-30T18:19:10.284Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: entering follower state: follower="Node at d4487d22-db28-ad97-459e-bdc76e06a8fc [Follower]" leader-address= leader-id=
2025-10-30T18:19:10.284Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4487d22-db28-ad97-459e-bdc76e06a8fc Address:d4487d22-db28-ad97-459e-bdc76e06a8fc} {Suffrage:Voter ID:server-7961c63f-0644-b43d-47d3-1c6aed691c58 Address:7961c63f-0644-b43d-47d3-1c6aed691c58} {Suffrage:Voter ID:server-d97fd847-4517-e92e-2b4c-3b09900a54e0 Address:d97fd847-4517-e92e-2b4c-3b09900a54e0}]"
2025-10-30T18:19:10.284Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: entering follower state: follower="Node at 7961c63f-0644-b43d-47d3-1c6aed691c58 [Follower]" leader-address= leader-id=
2025-10-30T18:19:10.284Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4487d22-db28-ad97-459e-bdc76e06a8fc Address:d4487d22-db28-ad97-459e-bdc76e06a8fc} {Suffrage:Voter ID:server-7961c63f-0644-b43d-47d3-1c6aed691c58 Address:7961c63f-0644-b43d-47d3-1c6aed691c58} {Suffrage:Voter ID:server-d97fd847-4517-e92e-2b4c-3b09900a54e0 Address:d97fd847-4517-e92e-2b4c-3b09900a54e0}]"
2025-10-30T18:19:10.284Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:10.284Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: entering follower state: follower="Node at d97fd847-4517-e92e-2b4c-3b09900a54e0 [Follower]" leader-address= leader-id=
2025-10-30T18:19:10.846Z [WARN]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:10.846Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: entering candidate state: node="Node at d97fd847-4517-e92e-2b4c-3b09900a54e0 [Candidate]" term=2
2025-10-30T18:19:10.846Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: election won: term=2 tally=2
2025-10-30T18:19:10.846Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: entering leader state: leader="Node at d97fd847-4517-e92e-2b4c-3b09900a54e0 [Leader]"
2025-10-30T18:19:10.846Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: added peer, starting replication: peer=server-d4487d22-db28-ad97-459e-bdc76e06a8fc
2025-10-30T18:19:10.846Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: added peer, starting replication: peer=server-7961c63f-0644-b43d-47d3-1c6aed691c58
2025-10-30T18:19:10.846Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-7961c63f-0644-b43d-47d3-1c6aed691c58 7961c63f-0644-b43d-47d3-1c6aed691c58}"
2025-10-30T18:19:10.846Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-d4487d22-db28-ad97-459e-bdc76e06a8fc d4487d22-db28-ad97-459e-bdc76e06a8fc}"
2025-10-30T18:19:11.852Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:10.84659963 +0000 UTC m=+97.125273527 (1 nodes), 561.662825ms from start of poll, 561.989686ms from cluster start. Timeout at 2025-10-30 18:19:11.851601919 +0000 UTC m=+98.130275815, 1.005002288s after stability
2025-10-30T18:19:11.852Z [INFO]  creating new snapshot: path=/tmp/raft3705812841/snapshots/2-10013-1761848351852.tmp
2025-10-30T18:19:11.974Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: copied to local snapshot: bytes=71
2025-10-30T18:19:11.974Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: snapshot restore progress: id=2-10013-1761848351852 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:11.974Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: removing all old logs from log store
2025-10-30T18:19:11.974Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: compacting logs: from=1 to=2
2025-10-30T18:19:11.974Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: restored user snapshot: index=10013
2025-10-30T18:19:11.974Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found"
2025-10-30T18:19:11.974Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: aborting pipeline replication: peer="{Voter server-7961c63f-0644-b43d-47d3-1c6aed691c58 7961c63f-0644-b43d-47d3-1c6aed691c58}"
2025-10-30T18:19:11.974Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found"
2025-10-30T18:19:11.974Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: aborting pipeline replication: peer="{Voter server-d4487d22-db28-ad97-459e-bdc76e06a8fc d4487d22-db28-ad97-459e-bdc76e06a8fc}"
2025-10-30T18:19:11.983Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found"
2025-10-30T18:19:11.985Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found"
2025-10-30T18:19:11.985Z [INFO]  creating new snapshot: path=/tmp/raft3916884886/snapshots/2-10013-1761848351985.tmp
2025-10-30T18:19:12.021Z [INFO]  creating new snapshot: path=/tmp/raft2705774192/snapshots/2-10013-1761848352021.tmp
2025-10-30T18:19:12.025Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:12.025Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:12.025Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: copied to local snapshot: bytes=71
2025-10-30T18:19:12.025Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: copied to local snapshot: bytes=71
2025-10-30T18:19:12.025Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: snapshot restore progress: id=2-10013-1761848352021 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:12.025Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: removing all old logs from log store
2025-10-30T18:19:12.025Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: compacting logs: from=1 to=2
2025-10-30T18:19:12.025Z [INFO]  server-7961c63f-0644-b43d-47d3-1c6aed691c58: Installed remote snapshot
2025-10-30T18:19:12.025Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: snapshot restore progress: id=2-10013-1761848351985 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-10-30T18:19:12.025Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: removing all old logs from log store
2025-10-30T18:19:12.025Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: compacting logs: from=1 to=2
2025-10-30T18:19:12.025Z [INFO]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: Installed remote snapshot
2025-10-30T18:19:12.026Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-7961c63f-0644-b43d-47d3-1c6aed691c58 7961c63f-0644-b43d-47d3-1c6aed691c58}"
2025-10-30T18:19:12.026Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:12.026Z [WARN]  server-d4487d22-db28-ad97-459e-bdc76e06a8fc: skipping application of old log: index=10013
2025-10-30T18:19:12.026Z [INFO]  server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-d4487d22-db28-ad97-459e-bdc76e06a8fc d4487d22-db28-ad97-459e-bdc76e06a8fc}"
2025-10-30T18:19:13.031Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:12.026048671 +0000 UTC m=+98.304722569 (2 nodes), 639ns from start of poll, 1.741438728s from cluster start. Timeout at 2025-10-30 18:19:13.031085221 +0000 UTC m=+99.309759118, 1.005036549s after stability
2025-10-30T18:19:13.043Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: aborting pipeline replication: peer="{Voter server-c495c0bd-a40e-b036-c40c-82fa0659a361 c495c0bd-a40e-b036-c40c-82fa0659a361}"
2025-10-30T18:19:13.043Z [INFO]  server-fd78e202-034d-df7b-0446-3c02197f42a1: aborting pipeline replication: peer="{Voter server-c275bf99-ff0d-a74b-3547-f4d8386c899d c275bf99-ff0d-a74b-3547-f4d8386c899d}"
--- PASS: TestRaft_UserRestore (86.25s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.64s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.70s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.41s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.62s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.63s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.08s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.71s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.76s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.46s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.31s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.65s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.69s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.94s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.93s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.64s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (3.07s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.76s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.13s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.38s)
=== RUN   TestRaft_SendSnapshotFollower
2025-10-30T18:19:13.044Z [INFO]  server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 Address:7583d3a5-2c6c-1760-d3f1-4b0d55322785} {Suffrage:Voter ID:server-c7342f45-e086-d997-bac9-cfee8eb5839e Address:c7342f45-e086-d997-bac9-cfee8eb5839e} {Suffrage:Voter ID:server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 Address:8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89}]"
2025-10-30T18:19:13.044Z [INFO]  server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: entering follower state: follower="Node at 7583d3a5-2c6c-1760-d3f1-4b0d55322785 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.044Z [INFO]  server-c7342f45-e086-d997-bac9-cfee8eb5839e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 Address:7583d3a5-2c6c-1760-d3f1-4b0d55322785} {Suffrage:Voter ID:server-c7342f45-e086-d997-bac9-cfee8eb5839e Address:c7342f45-e086-d997-bac9-cfee8eb5839e} {Suffrage:Voter ID:server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 Address:8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89}]"
2025-10-30T18:19:13.044Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 Address:7583d3a5-2c6c-1760-d3f1-4b0d55322785} {Suffrage:Voter ID:server-c7342f45-e086-d997-bac9-cfee8eb5839e Address:c7342f45-e086-d997-bac9-cfee8eb5839e} {Suffrage:Voter ID:server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 Address:8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89}]"
2025-10-30T18:19:13.044Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:13.044Z [INFO]  server-c7342f45-e086-d997-bac9-cfee8eb5839e: entering follower state: follower="Node at c7342f45-e086-d997-bac9-cfee8eb5839e [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.044Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: entering follower state: follower="Node at 8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.111Z [WARN]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:13.111Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: entering candidate state: node="Node at 8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 [Candidate]" term=2
2025-10-30T18:19:13.111Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: election won: term=2 tally=2
2025-10-30T18:19:13.111Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: entering leader state: leader="Node at 8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 [Leader]"
2025-10-30T18:19:13.111Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: added peer, starting replication: peer=server-7583d3a5-2c6c-1760-d3f1-4b0d55322785
2025-10-30T18:19:13.111Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: added peer, starting replication: peer=server-c7342f45-e086-d997-bac9-cfee8eb5839e
2025-10-30T18:19:13.111Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: pipelining replication: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}"
2025-10-30T18:19:13.111Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: pipelining replication: peer="{Voter server-c7342f45-e086-d997-bac9-cfee8eb5839e c7342f45-e086-d997-bac9-cfee8eb5839e}"
2025-10-30T18:19:13.216Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:13.111256348 +0000 UTC m=+99.389930251 (2 nodes), 66.958009ms from start of poll, 67.187701ms from cluster start. Timeout at 2025-10-30 18:19:13.216259568 +0000 UTC m=+99.494933478, 105.003227ms after stability
2025-10-30T18:19:13.216Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:13.322Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:13.216767018 +0000 UTC m=+99.495440916 (1 nodes), 2.033µs from start of poll, 172.698366ms from cluster start. Timeout at 2025-10-30 18:19:13.321802796 +0000 UTC m=+99.600476695, 105.035779ms after stability
2025-10-30T18:19:13.322Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to pipeline appendEntries: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" error="append pipeline closed"
2025-10-30T18:19:13.322Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: aborting pipeline replication: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}"
2025-10-30T18:19:13.322Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to appendEntries to: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" error="failed to connect to peer: 7583d3a5-2c6c-1760-d3f1-4b0d55322785"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-10-30T18:19:13.326Z [ERROR] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: failed to take snapshot: error="nothing new to snapshot"
2025-10-30T18:19:13.326Z [INFO]  server-c7342f45-e086-d997-bac9-cfee8eb5839e: starting snapshot up to: index=55
2025-10-30T18:19:13.326Z [INFO]  creating new snapshot: path=/tmp/raft1374354312/snapshots/2-55-1761848353326.tmp
2025-10-30T18:19:13.326Z [INFO]  server-c7342f45-e086-d997-bac9-cfee8eb5839e: compacting logs: from=1 to=55
2025-10-30T18:19:13.326Z [INFO]  server-c7342f45-e086-d997-bac9-cfee8eb5839e: snapshot complete up to: index=55
2025-10-30T18:19:13.326Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: starting snapshot up to: index=102
2025-10-30T18:19:13.326Z [INFO]  creating new snapshot: path=/tmp/raft3709220428/snapshots/2-102-1761848353326.tmp
2025-10-30T18:19:13.330Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to heartbeat to: peer=7583d3a5-2c6c-1760-d3f1-4b0d55322785 backoff time=10ms error="failed to connect to peer: 7583d3a5-2c6c-1760-d3f1-4b0d55322785"
2025-10-30T18:19:13.333Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to appendEntries to: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" error="failed to connect to peer: 7583d3a5-2c6c-1760-d3f1-4b0d55322785"
2025-10-30T18:19:13.335Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: compacting logs: from=1 to=92
2025-10-30T18:19:13.335Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: snapshot complete up to: index=102
2025-10-30T18:19:13.344Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to get log: index=2 error="log not found"
2025-10-30T18:19:13.344Z [INFO]  creating new snapshot: path=/tmp/raft608591271/snapshots/2-102-1761848353344.tmp
2025-10-30T18:19:13.344Z [INFO]  server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-10-30T18:19:13.344Z [INFO]  server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: copied to local snapshot: bytes=693
2025-10-30T18:19:13.345Z [INFO]  server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: snapshot restore progress: id=2-102-1761848353344 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-10-30T18:19:13.345Z [INFO]  server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: Installed remote snapshot
2025-10-30T18:19:13.346Z [INFO]  server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: aborting pipeline replication: peer="{Voter server-c7342f45-e086-d997-bac9-cfee8eb5839e c7342f45-e086-d997-bac9-cfee8eb5839e}"
--- PASS: TestRaft_SendSnapshotFollower (0.31s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-10-30T18:19:13.353Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5dbca4c-3605-f49a-1631-4d44d5c97422 Address:f5dbca4c-3605-f49a-1631-4d44d5c97422} {Suffrage:Voter ID:server-a6a5222a-667c-6f68-0950-b006697a26c5 Address:a6a5222a-667c-6f68-0950-b006697a26c5} {Suffrage:Voter ID:server-f16cefb6-79a8-9b1c-ab72-06db40988785 Address:f16cefb6-79a8-9b1c-ab72-06db40988785}]"
2025-10-30T18:19:13.353Z [INFO]  server-a6a5222a-667c-6f68-0950-b006697a26c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5dbca4c-3605-f49a-1631-4d44d5c97422 Address:f5dbca4c-3605-f49a-1631-4d44d5c97422} {Suffrage:Voter ID:server-a6a5222a-667c-6f68-0950-b006697a26c5 Address:a6a5222a-667c-6f68-0950-b006697a26c5} {Suffrage:Voter ID:server-f16cefb6-79a8-9b1c-ab72-06db40988785 Address:f16cefb6-79a8-9b1c-ab72-06db40988785}]"
2025-10-30T18:19:13.353Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: entering follower state: follower="Node at f5dbca4c-3605-f49a-1631-4d44d5c97422 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.353Z [INFO]  server-a6a5222a-667c-6f68-0950-b006697a26c5: entering follower state: follower="Node at a6a5222a-667c-6f68-0950-b006697a26c5 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.353Z [INFO]  server-f16cefb6-79a8-9b1c-ab72-06db40988785: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5dbca4c-3605-f49a-1631-4d44d5c97422 Address:f5dbca4c-3605-f49a-1631-4d44d5c97422} {Suffrage:Voter ID:server-a6a5222a-667c-6f68-0950-b006697a26c5 Address:a6a5222a-667c-6f68-0950-b006697a26c5} {Suffrage:Voter ID:server-f16cefb6-79a8-9b1c-ab72-06db40988785 Address:f16cefb6-79a8-9b1c-ab72-06db40988785}]"
2025-10-30T18:19:13.353Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:13.353Z [INFO]  server-f16cefb6-79a8-9b1c-ab72-06db40988785: entering follower state: follower="Node at f16cefb6-79a8-9b1c-ab72-06db40988785 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.407Z [WARN]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:13.407Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: entering candidate state: node="Node at f5dbca4c-3605-f49a-1631-4d44d5c97422 [Candidate]" term=2
2025-10-30T18:19:13.407Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: election won: term=2 tally=2
2025-10-30T18:19:13.407Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: entering leader state: leader="Node at f5dbca4c-3605-f49a-1631-4d44d5c97422 [Leader]"
2025-10-30T18:19:13.407Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: added peer, starting replication: peer=server-a6a5222a-667c-6f68-0950-b006697a26c5
2025-10-30T18:19:13.407Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: added peer, starting replication: peer=server-f16cefb6-79a8-9b1c-ab72-06db40988785
2025-10-30T18:19:13.407Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: pipelining replication: peer="{Voter server-f16cefb6-79a8-9b1c-ab72-06db40988785 f16cefb6-79a8-9b1c-ab72-06db40988785}"
2025-10-30T18:19:13.407Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: pipelining replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}"
2025-10-30T18:19:13.513Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:13.407850738 +0000 UTC m=+99.686524650 (2 nodes), 54.285608ms from start of poll, 54.645364ms from cluster start. Timeout at 2025-10-30 18:19:13.512855387 +0000 UTC m=+99.791529299, 105.004649ms after stability
2025-10-30T18:19:13.513Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:13.619Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:13.513271522 +0000 UTC m=+99.791945424 (1 nodes), 1.476µs from start of poll, 160.066138ms from cluster start. Timeout at 2025-10-30 18:19:13.618288243 +0000 UTC m=+99.896962160, 105.016736ms after stability
2025-10-30T18:19:13.619Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to pipeline appendEntries: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" error="append pipeline closed"
2025-10-30T18:19:13.619Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: aborting pipeline replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}"
2025-10-30T18:19:13.619Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to appendEntries to: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" error="failed to connect to peer: a6a5222a-667c-6f68-0950-b006697a26c5"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-10-30T18:19:13.621Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: starting snapshot up to: index=102
2025-10-30T18:19:13.621Z [INFO]  creating new snapshot: path=/tmp/raft2299293677/snapshots/2-102-1761848353621.tmp
2025-10-30T18:19:13.623Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to heartbeat to: peer=a6a5222a-667c-6f68-0950-b006697a26c5 backoff time=10ms error="failed to connect to peer: a6a5222a-667c-6f68-0950-b006697a26c5"
2025-10-30T18:19:13.624Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: compacting logs: from=1 to=92
2025-10-30T18:19:13.624Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: snapshot complete up to: index=102
2025-10-30T18:19:13.624Z [ERROR] server-a6a5222a-667c-6f68-0950-b006697a26c5: failed to take snapshot: error="nothing new to snapshot"
2025-10-30T18:19:13.624Z [INFO]  server-f16cefb6-79a8-9b1c-ab72-06db40988785: starting snapshot up to: index=94
2025-10-30T18:19:13.624Z [INFO]  creating new snapshot: path=/tmp/raft1020817440/snapshots/2-94-1761848353624.tmp
2025-10-30T18:19:13.624Z [INFO]  server-f16cefb6-79a8-9b1c-ab72-06db40988785: compacting logs: from=1 to=92
2025-10-30T18:19:13.624Z [INFO]  server-f16cefb6-79a8-9b1c-ab72-06db40988785: snapshot complete up to: index=94
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-10-30T18:19:13.629Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to get log: index=2 error="log not found"
2025-10-30T18:19:13.630Z [INFO]  creating new snapshot: path=/tmp/raft3269466841/snapshots/2-102-1761848353630.tmp
2025-10-30T18:19:13.630Z [INFO]  server-a6a5222a-667c-6f68-0950-b006697a26c5: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-10-30T18:19:13.637Z [INFO]  server-a6a5222a-667c-6f68-0950-b006697a26c5: copied to local snapshot: bytes=693
2025-10-30T18:19:13.637Z [INFO]  server-a6a5222a-667c-6f68-0950-b006697a26c5: snapshot restore progress: id=2-102-1761848353630 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-10-30T18:19:13.637Z [INFO]  server-a6a5222a-667c-6f68-0950-b006697a26c5: Installed remote snapshot
2025-10-30T18:19:13.637Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: pipelining replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}"
2025-10-30T18:19:13.638Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: aborting pipeline replication: peer="{Voter server-f16cefb6-79a8-9b1c-ab72-06db40988785 f16cefb6-79a8-9b1c-ab72-06db40988785}"
2025-10-30T18:19:13.638Z [INFO]  server-f5dbca4c-3605-f49a-1631-4d44d5c97422: aborting pipeline replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s)
=== RUN   TestRaft_ReJoinFollower
2025-10-30T18:19:13.639Z [INFO]  server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959} {Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]"
2025-10-30T18:19:13.639Z [INFO]  server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959: entering follower state: follower="Node at a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.639Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959} {Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]"
2025-10-30T18:19:13.639Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: entering follower state: follower="Node at b33acf52-3b79-ca0d-3159-dcb3eef07e01 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.639Z [INFO]  server-ed4a13cc-5082-e111-3c27-598fb280b423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959} {Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]"
2025-10-30T18:19:13.639Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:13.639Z [INFO]  server-ed4a13cc-5082-e111-3c27-598fb280b423: entering follower state: follower="Node at ed4a13cc-5082-e111-3c27-598fb280b423 [Follower]" leader-address= leader-id=
2025-10-30T18:19:13.700Z [WARN]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:13.700Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: entering candidate state: node="Node at b33acf52-3b79-ca0d-3159-dcb3eef07e01 [Candidate]" term=2
2025-10-30T18:19:13.700Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: election won: term=2 tally=2
2025-10-30T18:19:13.700Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: entering leader state: leader="Node at b33acf52-3b79-ca0d-3159-dcb3eef07e01 [Leader]"
2025-10-30T18:19:13.700Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: added peer, starting replication: peer=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959
2025-10-30T18:19:13.700Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: added peer, starting replication: peer=server-ed4a13cc-5082-e111-3c27-598fb280b423
2025-10-30T18:19:13.700Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: pipelining replication: peer="{Voter server-ed4a13cc-5082-e111-3c27-598fb280b423 ed4a13cc-5082-e111-3c27-598fb280b423}"
2025-10-30T18:19:13.700Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: pipelining replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}"
2025-10-30T18:19:13.805Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:13.700589231 +0000 UTC m=+99.979263129 (1 nodes), 60.726454ms from start of poll, 60.971752ms from cluster start. Timeout at 2025-10-30 18:19:13.805593901 +0000 UTC m=+100.084267799, 105.00467ms after stability
2025-10-30T18:19:13.810Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:13.916Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:13.810947792 +0000 UTC m=+100.089621690 (2 nodes), 1.103µs from start of poll, 171.330313ms from cluster start. Timeout at 2025-10-30 18:19:13.915971615 +0000 UTC m=+100.194645512, 105.023822ms after stability
2025-10-30T18:19:13.916Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: updating configuration: command=RemoveServer server-id=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 server-addr= servers="[{Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]"
2025-10-30T18:19:13.916Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: removed peer, stopping replication: peer=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 last-index=3
2025-10-30T18:19:13.916Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: aborting pipeline replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}"
2025-10-30T18:19:14.021Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: updating configuration: command=AddVoter server-id=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 server-addr=a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 servers="[{Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423} {Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}]"
2025-10-30T18:19:14.021Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: added peer, starting replication: peer=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959
2025-10-30T18:19:14.021Z [WARN]  server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-10-30T18:19:14.021Z [WARN]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: appendEntries rejected, sending older logs: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}" next=4
2025-10-30T18:19:14.022Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: pipelining replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}"
2025-10-30T18:19:14.022Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:14.127Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:14.02207931 +0000 UTC m=+100.300753207 (1 nodes), 434ns from start of poll, 382.46183ms from cluster start. Timeout at 2025-10-30 18:19:14.127106464 +0000 UTC m=+100.405780360, 105.027153ms after stability
2025-10-30T18:19:14.127Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: aborting pipeline replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}"
2025-10-30T18:19:14.127Z [INFO]  server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: aborting pipeline replication: peer="{Voter server-ed4a13cc-5082-e111-3c27-598fb280b423 ed4a13cc-5082-e111-3c27-598fb280b423}"
--- PASS: TestRaft_ReJoinFollower (0.49s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-10-30T18:19:14.128Z [INFO]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75d17107-02cc-ceaa-2114-cb6425e8648f Address:75d17107-02cc-ceaa-2114-cb6425e8648f} {Suffrage:Voter ID:server-1e3da886-edcc-b52d-ed64-fa60185a48e1 Address:1e3da886-edcc-b52d-ed64-fa60185a48e1}]"
2025-10-30T18:19:14.128Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75d17107-02cc-ceaa-2114-cb6425e8648f Address:75d17107-02cc-ceaa-2114-cb6425e8648f} {Suffrage:Voter ID:server-1e3da886-edcc-b52d-ed64-fa60185a48e1 Address:1e3da886-edcc-b52d-ed64-fa60185a48e1}]"
2025-10-30T18:19:14.128Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:14.128Z [INFO]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering follower state: follower="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.128Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering follower state: follower="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.206Z [WARN]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:14.206Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=2
2025-10-30T18:19:14.206Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: election won: term=2 tally=2
2025-10-30T18:19:14.206Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering leader state: leader="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Leader]"
2025-10-30T18:19:14.206Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: added peer, starting replication: peer=server-75d17107-02cc-ceaa-2114-cb6425e8648f
2025-10-30T18:19:14.207Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: pipelining replication: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}"
2025-10-30T18:19:14.312Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:14.206855865 +0000 UTC m=+100.485529762 (1 nodes), 78.501942ms from start of poll, 78.673553ms from cluster start. Timeout at 2025-10-30 18:19:14.311856938 +0000 UTC m=+100.590530836, 105.001074ms after stability
2025-10-30T18:19:14.318Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:14.423Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:14.318548305 +0000 UTC m=+100.597222204 (1 nodes), 1.716µs from start of poll, 190.365995ms from cluster start. Timeout at 2025-10-30 18:19:14.423570529 +0000 UTC m=+100.702244426, 105.022222ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Follower]
2025-10-30T18:19:14.425Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to heartbeat to: peer=75d17107-02cc-ceaa-2114-cb6425e8648f backoff time=10ms error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f"
2025-10-30T18:19:14.426Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to pipeline appendEntries: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="append pipeline closed"
2025-10-30T18:19:14.426Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: aborting pipeline replication: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}"
2025-10-30T18:19:14.431Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to appendEntries to: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f"
2025-10-30T18:19:14.442Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to heartbeat to: peer=75d17107-02cc-ceaa-2114-cb6425e8648f backoff time=10ms error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f"
2025-10-30T18:19:14.451Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to appendEntries to: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f"
2025-10-30T18:19:14.459Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to heartbeat to: peer=75d17107-02cc-ceaa-2114-cb6425e8648f backoff time=10ms error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f"
2025-10-30T18:19:14.468Z [WARN]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to contact: server-id=server-75d17107-02cc-ceaa-2114-cb6425e8648f time=50.12124ms
2025-10-30T18:19:14.468Z [WARN]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to contact quorum of nodes, stepping down
2025-10-30T18:19:14.468Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering follower state: follower="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.468Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:14.468Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to appendEntries to: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f"
2025-10-30T18:19:14.473Z [WARN]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: heartbeat timeout reached, starting election: last-leader-addr=1e3da886-edcc-b52d-ed64-fa60185a48e1 last-leader-id=server-1e3da886-edcc-b52d-ed64-fa60185a48e1
2025-10-30T18:19:14.473Z [INFO]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=3
2025-10-30T18:19:14.473Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=3
2025-10-30T18:19:14.521Z [WARN]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:14.521Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=3
2025-10-30T18:19:14.521Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=3
2025-10-30T18:19:14.536Z [WARN]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election
2025-10-30T18:19:14.536Z [INFO]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=4
2025-10-30T18:19:14.536Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=4
2025-10-30T18:19:14.573Z [WARN]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: Election timeout reached, restarting election
2025-10-30T18:19:14.573Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=4
2025-10-30T18:19:14.573Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=4
2025-10-30T18:19:14.588Z [WARN]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election
2025-10-30T18:19:14.588Z [INFO]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=5
2025-10-30T18:19:14.588Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=5
2025-10-30T18:19:14.627Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:14.521909276 +0000 UTC m=+100.800583175 (0 nodes), 53.757186ms from start of poll, 393.726966ms from cluster start. Timeout at 2025-10-30 18:19:14.626911988 +0000 UTC m=+100.905585886, 105.002711ms after stability
2025-10-30T18:19:14.638Z [WARN]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: Election timeout reached, restarting election
2025-10-30T18:19:14.638Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=5
2025-10-30T18:19:14.639Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=5
2025-10-30T18:19:14.660Z [WARN]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election
2025-10-30T18:19:14.660Z [INFO]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=6
2025-10-30T18:19:14.660Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=6
2025-10-30T18:19:14.716Z [WARN]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election
2025-10-30T18:19:14.716Z [INFO]  server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=7
2025-10-30T18:19:14.716Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=7
2025-10-30T18:19:14.728Z [WARN]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: Election timeout reached, restarting election
2025-10-30T18:19:14.728Z [INFO]  server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=6
2025-10-30T18:19:14.728Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.61s)
=== RUN   TestRaft_Barrier
2025-10-30T18:19:14.734Z [INFO]  server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd Address:8cf18d3e-1e0d-d84e-d353-cae4f5c353dd} {Suffrage:Voter ID:server-711dc642-b30b-8ad0-6681-e3d4769045b2 Address:711dc642-b30b-8ad0-6681-e3d4769045b2} {Suffrage:Voter ID:server-88924579-3ef7-7d49-cbc3-d7054a5eba5f Address:88924579-3ef7-7d49-cbc3-d7054a5eba5f}]"
2025-10-30T18:19:14.734Z [INFO]  server-711dc642-b30b-8ad0-6681-e3d4769045b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd Address:8cf18d3e-1e0d-d84e-d353-cae4f5c353dd} {Suffrage:Voter ID:server-711dc642-b30b-8ad0-6681-e3d4769045b2 Address:711dc642-b30b-8ad0-6681-e3d4769045b2} {Suffrage:Voter ID:server-88924579-3ef7-7d49-cbc3-d7054a5eba5f Address:88924579-3ef7-7d49-cbc3-d7054a5eba5f}]"
2025-10-30T18:19:14.734Z [INFO]  server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd: entering follower state: follower="Node at 8cf18d3e-1e0d-d84e-d353-cae4f5c353dd [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.734Z [INFO]  server-711dc642-b30b-8ad0-6681-e3d4769045b2: entering follower state: follower="Node at 711dc642-b30b-8ad0-6681-e3d4769045b2 [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.734Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd Address:8cf18d3e-1e0d-d84e-d353-cae4f5c353dd} {Suffrage:Voter ID:server-711dc642-b30b-8ad0-6681-e3d4769045b2 Address:711dc642-b30b-8ad0-6681-e3d4769045b2} {Suffrage:Voter ID:server-88924579-3ef7-7d49-cbc3-d7054a5eba5f Address:88924579-3ef7-7d49-cbc3-d7054a5eba5f}]"
2025-10-30T18:19:14.734Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:14.734Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: entering follower state: follower="Node at 88924579-3ef7-7d49-cbc3-d7054a5eba5f [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.801Z [WARN]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:14.801Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: entering candidate state: node="Node at 88924579-3ef7-7d49-cbc3-d7054a5eba5f [Candidate]" term=2
2025-10-30T18:19:14.801Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: election won: term=2 tally=2
2025-10-30T18:19:14.801Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: entering leader state: leader="Node at 88924579-3ef7-7d49-cbc3-d7054a5eba5f [Leader]"
2025-10-30T18:19:14.801Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: added peer, starting replication: peer=server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd
2025-10-30T18:19:14.801Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: added peer, starting replication: peer=server-711dc642-b30b-8ad0-6681-e3d4769045b2
2025-10-30T18:19:14.801Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: pipelining replication: peer="{Voter server-711dc642-b30b-8ad0-6681-e3d4769045b2 711dc642-b30b-8ad0-6681-e3d4769045b2}"
2025-10-30T18:19:14.801Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: pipelining replication: peer="{Voter server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd 8cf18d3e-1e0d-d84e-d353-cae4f5c353dd}"
2025-10-30T18:19:14.906Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:14.801299484 +0000 UTC m=+101.079973381 (1 nodes), 67.046642ms from start of poll, 67.328163ms from cluster start. Timeout at 2025-10-30 18:19:14.906302998 +0000 UTC m=+101.184976895, 105.003514ms after stability
2025-10-30T18:19:14.914Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: aborting pipeline replication: peer="{Voter server-711dc642-b30b-8ad0-6681-e3d4769045b2 711dc642-b30b-8ad0-6681-e3d4769045b2}"
2025-10-30T18:19:14.914Z [INFO]  server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: aborting pipeline replication: peer="{Voter server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd 8cf18d3e-1e0d-d84e-d353-cae4f5c353dd}"
--- PASS: TestRaft_Barrier (0.18s)
=== RUN   TestRaft_VerifyLeader
2025-10-30T18:19:14.915Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4 Address:6afa2e74-f001-cc97-84b6-0f31d1ee37c4} {Suffrage:Voter ID:server-a809c745-30a2-6a02-9cc9-76d2a00fac93 Address:a809c745-30a2-6a02-9cc9-76d2a00fac93} {Suffrage:Voter ID:server-108d01e4-d13b-876c-5cf0-0ce4f78224cd Address:108d01e4-d13b-876c-5cf0-0ce4f78224cd}]"
2025-10-30T18:19:14.915Z [INFO]  server-a809c745-30a2-6a02-9cc9-76d2a00fac93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4 Address:6afa2e74-f001-cc97-84b6-0f31d1ee37c4} {Suffrage:Voter ID:server-a809c745-30a2-6a02-9cc9-76d2a00fac93 Address:a809c745-30a2-6a02-9cc9-76d2a00fac93} {Suffrage:Voter ID:server-108d01e4-d13b-876c-5cf0-0ce4f78224cd Address:108d01e4-d13b-876c-5cf0-0ce4f78224cd}]"
2025-10-30T18:19:14.915Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: entering follower state: follower="Node at 6afa2e74-f001-cc97-84b6-0f31d1ee37c4 [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.915Z [INFO]  server-a809c745-30a2-6a02-9cc9-76d2a00fac93: entering follower state: follower="Node at a809c745-30a2-6a02-9cc9-76d2a00fac93 [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.915Z [INFO]  server-108d01e4-d13b-876c-5cf0-0ce4f78224cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4 Address:6afa2e74-f001-cc97-84b6-0f31d1ee37c4} {Suffrage:Voter ID:server-a809c745-30a2-6a02-9cc9-76d2a00fac93 Address:a809c745-30a2-6a02-9cc9-76d2a00fac93} {Suffrage:Voter ID:server-108d01e4-d13b-876c-5cf0-0ce4f78224cd Address:108d01e4-d13b-876c-5cf0-0ce4f78224cd}]"
2025-10-30T18:19:14.915Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:14.915Z [INFO]  server-108d01e4-d13b-876c-5cf0-0ce4f78224cd: entering follower state: follower="Node at 108d01e4-d13b-876c-5cf0-0ce4f78224cd [Follower]" leader-address= leader-id=
2025-10-30T18:19:14.975Z [WARN]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:14.975Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: entering candidate state: node="Node at 6afa2e74-f001-cc97-84b6-0f31d1ee37c4 [Candidate]" term=2
2025-10-30T18:19:14.975Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: election won: term=2 tally=2
2025-10-30T18:19:14.975Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: entering leader state: leader="Node at 6afa2e74-f001-cc97-84b6-0f31d1ee37c4 [Leader]"
2025-10-30T18:19:14.975Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: added peer, starting replication: peer=server-a809c745-30a2-6a02-9cc9-76d2a00fac93
2025-10-30T18:19:14.975Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: added peer, starting replication: peer=server-108d01e4-d13b-876c-5cf0-0ce4f78224cd
2025-10-30T18:19:14.975Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: pipelining replication: peer="{Voter server-a809c745-30a2-6a02-9cc9-76d2a00fac93 a809c745-30a2-6a02-9cc9-76d2a00fac93}"
2025-10-30T18:19:14.975Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: pipelining replication: peer="{Voter server-108d01e4-d13b-876c-5cf0-0ce4f78224cd 108d01e4-d13b-876c-5cf0-0ce4f78224cd}"
2025-10-30T18:19:15.080Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:14.975169202 +0000 UTC m=+101.253843116 (1 nodes), 59.640239ms from start of poll, 59.905188ms from cluster start. Timeout at 2025-10-30 18:19:15.08018794 +0000 UTC m=+101.358861837, 105.018721ms after stability
2025-10-30T18:19:15.080Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: aborting pipeline replication: peer="{Voter server-108d01e4-d13b-876c-5cf0-0ce4f78224cd 108d01e4-d13b-876c-5cf0-0ce4f78224cd}"
2025-10-30T18:19:15.080Z [INFO]  server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: aborting pipeline replication: peer="{Voter server-a809c745-30a2-6a02-9cc9-76d2a00fac93 a809c745-30a2-6a02-9cc9-76d2a00fac93}"
--- PASS: TestRaft_VerifyLeader (0.17s)
=== RUN   TestRaft_VerifyLeader_Single
2025-10-30T18:19:15.081Z [INFO]  server-f88da151-a65b-93d4-84bb-81ea9c60df4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f88da151-a65b-93d4-84bb-81ea9c60df4c Address:f88da151-a65b-93d4-84bb-81ea9c60df4c}]"
2025-10-30T18:19:15.081Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:15.081Z [INFO]  server-f88da151-a65b-93d4-84bb-81ea9c60df4c: entering follower state: follower="Node at f88da151-a65b-93d4-84bb-81ea9c60df4c [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.160Z [WARN]  server-f88da151-a65b-93d4-84bb-81ea9c60df4c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:15.161Z [INFO]  server-f88da151-a65b-93d4-84bb-81ea9c60df4c: entering candidate state: node="Node at f88da151-a65b-93d4-84bb-81ea9c60df4c [Candidate]" term=2
2025-10-30T18:19:15.161Z [INFO]  server-f88da151-a65b-93d4-84bb-81ea9c60df4c: election won: term=2 tally=1
2025-10-30T18:19:15.161Z [INFO]  server-f88da151-a65b-93d4-84bb-81ea9c60df4c: entering leader state: leader="Node at f88da151-a65b-93d4-84bb-81ea9c60df4c [Leader]"
2025-10-30T18:19:15.266Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:15.16113857 +0000 UTC m=+101.439812468 (1 nodes), 79.942375ms from start of poll, 80.077021ms from cluster start. Timeout at 2025-10-30 18:19:15.266139208 +0000 UTC m=+101.544813106, 105.000638ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.19s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-10-30T18:19:15.267Z [INFO]  server-a330f9ea-7751-d1dc-7e24-68647d055d99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a330f9ea-7751-d1dc-7e24-68647d055d99 Address:a330f9ea-7751-d1dc-7e24-68647d055d99} {Suffrage:Voter ID:server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a Address:b96e07e7-ed22-c5bc-b36e-b6524d2eb36a}]"
2025-10-30T18:19:15.267Z [INFO]  server-a330f9ea-7751-d1dc-7e24-68647d055d99: entering follower state: follower="Node at a330f9ea-7751-d1dc-7e24-68647d055d99 [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.267Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a330f9ea-7751-d1dc-7e24-68647d055d99 Address:a330f9ea-7751-d1dc-7e24-68647d055d99} {Suffrage:Voter ID:server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a Address:b96e07e7-ed22-c5bc-b36e-b6524d2eb36a}]"
2025-10-30T18:19:15.267Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:15.267Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering follower state: follower="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.319Z [WARN]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:15.319Z [WARN]  server-a330f9ea-7751-d1dc-7e24-68647d055d99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:15.319Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering candidate state: node="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Candidate]" term=2
2025-10-30T18:19:15.319Z [INFO]  server-a330f9ea-7751-d1dc-7e24-68647d055d99: entering candidate state: node="Node at a330f9ea-7751-d1dc-7e24-68647d055d99 [Candidate]" term=2
2025-10-30T18:19:15.319Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: duplicate requestVote for same term: term=2
2025-10-30T18:19:15.319Z [INFO]  server-a330f9ea-7751-d1dc-7e24-68647d055d99: duplicate requestVote for same term: term=2
2025-10-30T18:19:15.388Z [WARN]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: Election timeout reached, restarting election
2025-10-30T18:19:15.388Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering candidate state: node="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Candidate]" term=3
2025-10-30T18:19:15.388Z [INFO]  server-a330f9ea-7751-d1dc-7e24-68647d055d99: entering follower state: follower="Node at a330f9ea-7751-d1dc-7e24-68647d055d99 [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.388Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: election won: term=3 tally=2
2025-10-30T18:19:15.388Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering leader state: leader="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Leader]"
2025-10-30T18:19:15.389Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: added peer, starting replication: peer=server-a330f9ea-7751-d1dc-7e24-68647d055d99
2025-10-30T18:19:15.389Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: pipelining replication: peer="{Voter server-a330f9ea-7751-d1dc-7e24-68647d055d99 a330f9ea-7751-d1dc-7e24-68647d055d99}"
2025-10-30T18:19:15.494Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:15.389029851 +0000 UTC m=+101.667703756 (1 nodes), 121.074135ms from start of poll, 121.274325ms from cluster start. Timeout at 2025-10-30 18:19:15.494041355 +0000 UTC m=+101.772715271, 105.011515ms after stability
2025-10-30T18:19:15.494Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:15.600Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:15.494985255 +0000 UTC m=+101.773659152 (1 nodes), 1.733µs from start of poll, 227.229721ms from cluster start. Timeout at 2025-10-30 18:19:15.600004403 +0000 UTC m=+101.878678318, 105.019166ms after stability
2025-10-30T18:19:15.605Z [ERROR] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: peer has newer term, stopping replication: peer="{Voter server-a330f9ea-7751-d1dc-7e24-68647d055d99 a330f9ea-7751-d1dc-7e24-68647d055d99}"
2025-10-30T18:19:15.605Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: aborting pipeline replication: peer="{Voter server-a330f9ea-7751-d1dc-7e24-68647d055d99 a330f9ea-7751-d1dc-7e24-68647d055d99}"
2025-10-30T18:19:15.605Z [INFO]  server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering follower state: follower="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.34s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-10-30T18:19:15.606Z [INFO]  server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 Address:c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6} {Suffrage:Voter ID:server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b Address:8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b} {Suffrage:Voter ID:server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 Address:a9c50915-fc5f-a683-a5bc-01d4469b8d33}]"
2025-10-30T18:19:15.606Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 Address:c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6} {Suffrage:Voter ID:server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b Address:8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b} {Suffrage:Voter ID:server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 Address:a9c50915-fc5f-a683-a5bc-01d4469b8d33}]"
2025-10-30T18:19:15.606Z [INFO]  server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6: entering follower state: follower="Node at c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.606Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: entering follower state: follower="Node at 8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.606Z [INFO]  server-a9c50915-fc5f-a683-a5bc-01d4469b8d33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 Address:c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6} {Suffrage:Voter ID:server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b Address:8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b} {Suffrage:Voter ID:server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 Address:a9c50915-fc5f-a683-a5bc-01d4469b8d33}]"
2025-10-30T18:19:15.606Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:15.606Z [INFO]  server-a9c50915-fc5f-a683-a5bc-01d4469b8d33: entering follower state: follower="Node at a9c50915-fc5f-a683-a5bc-01d4469b8d33 [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.670Z [WARN]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:15.670Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: entering candidate state: node="Node at 8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b [Candidate]" term=2
2025-10-30T18:19:15.670Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: election won: term=2 tally=2
2025-10-30T18:19:15.670Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: entering leader state: leader="Node at 8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b [Leader]"
2025-10-30T18:19:15.670Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: added peer, starting replication: peer=server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6
2025-10-30T18:19:15.670Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: added peer, starting replication: peer=server-a9c50915-fc5f-a683-a5bc-01d4469b8d33
2025-10-30T18:19:15.670Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: pipelining replication: peer="{Voter server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 a9c50915-fc5f-a683-a5bc-01d4469b8d33}"
2025-10-30T18:19:15.670Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: pipelining replication: peer="{Voter server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6}"
2025-10-30T18:19:15.776Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:15.670566719 +0000 UTC m=+101.949240626 (1 nodes), 64.190128ms from start of poll, 64.425618ms from cluster start. Timeout at 2025-10-30 18:19:15.775572524 +0000 UTC m=+102.054246422, 105.005796ms after stability
2025-10-30T18:19:15.781Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:15.887Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:15.781705463 +0000 UTC m=+102.060379360 (2 nodes), 839ns from start of poll, 175.564352ms from cluster start. Timeout at 2025-10-30 18:19:15.88672434 +0000 UTC m=+102.165398255, 105.018895ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 [Follower]
2025-10-30T18:19:15.887Z [ERROR] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: failed to heartbeat to: peer=c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 backoff time=10ms error="failed to connect to peer: c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6"
2025-10-30T18:19:15.887Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: aborting pipeline replication: peer="{Voter server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6}"
2025-10-30T18:19:15.887Z [INFO]  server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: aborting pipeline replication: peer="{Voter server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 a9c50915-fc5f-a683-a5bc-01d4469b8d33}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s)
=== RUN   TestRaft_NotifyCh
2025-10-30T18:19:15.888Z [INFO]  server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37369f83-96dd-21bf-2579-f8fab1b3fbbd Address:37369f83-96dd-21bf-2579-f8fab1b3fbbd}]"
2025-10-30T18:19:15.888Z [INFO]  server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: entering follower state: follower="Node at 37369f83-96dd-21bf-2579-f8fab1b3fbbd [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.958Z [WARN]  server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:15.958Z [INFO]  server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: entering candidate state: node="Node at 37369f83-96dd-21bf-2579-f8fab1b3fbbd [Candidate]" term=2
2025-10-30T18:19:15.958Z [INFO]  server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: election won: term=2 tally=1
2025-10-30T18:19:15.958Z [INFO]  server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: entering leader state: leader="Node at 37369f83-96dd-21bf-2579-f8fab1b3fbbd [Leader]"
--- PASS: TestRaft_NotifyCh (0.07s)
=== RUN   TestRaft_AppendEntry
2025-10-30T18:19:15.959Z [INFO]  server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd Address:b9a79db0-ed00-f7aa-d578-70cbe6cd62cd} {Suffrage:Voter ID:server-1d351432-1cc1-a366-b6ed-6fbb47728198 Address:1d351432-1cc1-a366-b6ed-6fbb47728198} {Suffrage:Voter ID:server-62da28e8-8262-542a-1a41-f01daa8d3931 Address:62da28e8-8262-542a-1a41-f01daa8d3931}]"
2025-10-30T18:19:15.959Z [INFO]  server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: entering follower state: follower="Node at b9a79db0-ed00-f7aa-d578-70cbe6cd62cd [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.959Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd Address:b9a79db0-ed00-f7aa-d578-70cbe6cd62cd} {Suffrage:Voter ID:server-1d351432-1cc1-a366-b6ed-6fbb47728198 Address:1d351432-1cc1-a366-b6ed-6fbb47728198} {Suffrage:Voter ID:server-62da28e8-8262-542a-1a41-f01daa8d3931 Address:62da28e8-8262-542a-1a41-f01daa8d3931}]"
2025-10-30T18:19:15.959Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: entering follower state: follower="Node at 1d351432-1cc1-a366-b6ed-6fbb47728198 [Follower]" leader-address= leader-id=
2025-10-30T18:19:15.959Z [INFO]  server-62da28e8-8262-542a-1a41-f01daa8d3931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd Address:b9a79db0-ed00-f7aa-d578-70cbe6cd62cd} {Suffrage:Voter ID:server-1d351432-1cc1-a366-b6ed-6fbb47728198 Address:1d351432-1cc1-a366-b6ed-6fbb47728198} {Suffrage:Voter ID:server-62da28e8-8262-542a-1a41-f01daa8d3931 Address:62da28e8-8262-542a-1a41-f01daa8d3931}]"
2025-10-30T18:19:15.959Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:15.959Z [INFO]  server-62da28e8-8262-542a-1a41-f01daa8d3931: entering follower state: follower="Node at 62da28e8-8262-542a-1a41-f01daa8d3931 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.033Z [WARN]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:16.033Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: entering candidate state: node="Node at 1d351432-1cc1-a366-b6ed-6fbb47728198 [Candidate]" term=2
2025-10-30T18:19:16.033Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: election won: term=2 tally=2
2025-10-30T18:19:16.033Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: entering leader state: leader="Node at 1d351432-1cc1-a366-b6ed-6fbb47728198 [Leader]"
2025-10-30T18:19:16.033Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: added peer, starting replication: peer=server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd
2025-10-30T18:19:16.033Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: added peer, starting replication: peer=server-62da28e8-8262-542a-1a41-f01daa8d3931
2025-10-30T18:19:16.033Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: pipelining replication: peer="{Voter server-62da28e8-8262-542a-1a41-f01daa8d3931 62da28e8-8262-542a-1a41-f01daa8d3931}"
2025-10-30T18:19:16.033Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: pipelining replication: peer="{Voter server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd b9a79db0-ed00-f7aa-d578-70cbe6cd62cd}"
2025-10-30T18:19:16.139Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:16.033455888 +0000 UTC m=+102.312129786 (2 nodes), 73.727249ms from start of poll, 73.9971ms from cluster start. Timeout at 2025-10-30 18:19:16.138457977 +0000 UTC m=+102.417131874, 105.002088ms after stability
2025-10-30T18:19:16.139Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:16.244Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:16.139410424 +0000 UTC m=+102.418084322 (1 nodes), 1.212µs from start of poll, 179.951636ms from cluster start. Timeout at 2025-10-30 18:19:16.244422031 +0000 UTC m=+102.523095927, 105.011605ms after stability
2025-10-30T18:19:16.244Z [WARN]  server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: clearing log suffix: from=1 to=2
2025-10-30T18:19:16.244Z [WARN]  server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: skipping application of old log: index=1
2025-10-30T18:19:16.244Z [WARN]  server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: skipping application of old log: index=1
2025-10-30T18:19:16.244Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: aborting pipeline replication: peer="{Voter server-62da28e8-8262-542a-1a41-f01daa8d3931 62da28e8-8262-542a-1a41-f01daa8d3931}"
2025-10-30T18:19:16.244Z [INFO]  server-1d351432-1cc1-a366-b6ed-6fbb47728198: aborting pipeline replication: peer="{Voter server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd b9a79db0-ed00-f7aa-d578-70cbe6cd62cd}"
--- PASS: TestRaft_AppendEntry (0.29s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-10-30T18:19:16.245Z [INFO]  server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba Address:4e41b38b-487e-8e96-d1c9-1403a9ab11ba} {Suffrage:Voter ID:server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 Address:b7e83b9e-b376-ca23-7e64-6163d1e3dab4} {Suffrage:Voter ID:server-e7b86f93-5674-2aad-7242-c0f0f55c0398 Address:e7b86f93-5674-2aad-7242-c0f0f55c0398}]"
2025-10-30T18:19:16.245Z [INFO]  server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba Address:4e41b38b-487e-8e96-d1c9-1403a9ab11ba} {Suffrage:Voter ID:server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 Address:b7e83b9e-b376-ca23-7e64-6163d1e3dab4} {Suffrage:Voter ID:server-e7b86f93-5674-2aad-7242-c0f0f55c0398 Address:e7b86f93-5674-2aad-7242-c0f0f55c0398}]"
2025-10-30T18:19:16.245Z [INFO]  server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba: entering follower state: follower="Node at 4e41b38b-487e-8e96-d1c9-1403a9ab11ba [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.245Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba Address:4e41b38b-487e-8e96-d1c9-1403a9ab11ba} {Suffrage:Voter ID:server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 Address:b7e83b9e-b376-ca23-7e64-6163d1e3dab4} {Suffrage:Voter ID:server-e7b86f93-5674-2aad-7242-c0f0f55c0398 Address:e7b86f93-5674-2aad-7242-c0f0f55c0398}]"
2025-10-30T18:19:16.245Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:16.245Z [INFO]  server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4: entering follower state: follower="Node at b7e83b9e-b376-ca23-7e64-6163d1e3dab4 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.245Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: entering follower state: follower="Node at e7b86f93-5674-2aad-7242-c0f0f55c0398 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.298Z [WARN]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:16.298Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: entering candidate state: node="Node at e7b86f93-5674-2aad-7242-c0f0f55c0398 [Candidate]" term=2
2025-10-30T18:19:16.298Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: election won: term=2 tally=2
2025-10-30T18:19:16.298Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: entering leader state: leader="Node at e7b86f93-5674-2aad-7242-c0f0f55c0398 [Leader]"
2025-10-30T18:19:16.298Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: added peer, starting replication: peer=server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba
2025-10-30T18:19:16.298Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: added peer, starting replication: peer=server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4
2025-10-30T18:19:16.298Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: pipelining replication: peer="{Voter server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 b7e83b9e-b376-ca23-7e64-6163d1e3dab4}"
2025-10-30T18:19:16.298Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: pipelining replication: peer="{Voter server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba 4e41b38b-487e-8e96-d1c9-1403a9ab11ba}"
2025-10-30T18:19:16.404Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:16.298494455 +0000 UTC m=+102.577168352 (2 nodes), 52.861663ms from start of poll, 53.027326ms from cluster start. Timeout at 2025-10-30 18:19:16.403497379 +0000 UTC m=+102.682171275, 105.002923ms after stability
2025-10-30T18:19:16.404Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:16.509Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:16.404064949 +0000 UTC m=+102.682738846 (1 nodes), 1.385µs from start of poll, 158.59782ms from cluster start. Timeout at 2025-10-30 18:19:16.509077072 +0000 UTC m=+102.787750986, 105.01214ms after stability
2025-10-30T18:19:16.509Z [WARN]  server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4: rejecting vote request since we have a leader: from=4e41b38b-487e-8e96-d1c9-1403a9ab11ba leader=e7b86f93-5674-2aad-7242-c0f0f55c0398 leader-id=server-e7b86f93-5674-2aad-7242-c0f0f55c0398
2025-10-30T18:19:16.509Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: aborting pipeline replication: peer="{Voter server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 b7e83b9e-b376-ca23-7e64-6163d1e3dab4}"
2025-10-30T18:19:16.509Z [INFO]  server-e7b86f93-5674-2aad-7242-c0f0f55c0398: aborting pipeline replication: peer="{Voter server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba 4e41b38b-487e-8e96-d1c9-1403a9ab11ba}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.26s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-10-30T18:19:16.510Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7030b20f-c40c-43d2-361d-5572c0e40e94 Address:7030b20f-c40c-43d2-361d-5572c0e40e94} {Suffrage:Voter ID:server-c6194370-bdf8-f920-7a3e-e0d629f35383 Address:c6194370-bdf8-f920-7a3e-e0d629f35383} {Suffrage:Voter ID:server-e43aa209-0d35-4525-2170-eb66f4e3cee6 Address:e43aa209-0d35-4525-2170-eb66f4e3cee6}]"
2025-10-30T18:19:16.510Z [INFO]  server-c6194370-bdf8-f920-7a3e-e0d629f35383: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7030b20f-c40c-43d2-361d-5572c0e40e94 Address:7030b20f-c40c-43d2-361d-5572c0e40e94} {Suffrage:Voter ID:server-c6194370-bdf8-f920-7a3e-e0d629f35383 Address:c6194370-bdf8-f920-7a3e-e0d629f35383} {Suffrage:Voter ID:server-e43aa209-0d35-4525-2170-eb66f4e3cee6 Address:e43aa209-0d35-4525-2170-eb66f4e3cee6}]"
2025-10-30T18:19:16.510Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: entering follower state: follower="Node at 7030b20f-c40c-43d2-361d-5572c0e40e94 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.510Z [INFO]  server-c6194370-bdf8-f920-7a3e-e0d629f35383: entering follower state: follower="Node at c6194370-bdf8-f920-7a3e-e0d629f35383 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.510Z [INFO]  server-e43aa209-0d35-4525-2170-eb66f4e3cee6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7030b20f-c40c-43d2-361d-5572c0e40e94 Address:7030b20f-c40c-43d2-361d-5572c0e40e94} {Suffrage:Voter ID:server-c6194370-bdf8-f920-7a3e-e0d629f35383 Address:c6194370-bdf8-f920-7a3e-e0d629f35383} {Suffrage:Voter ID:server-e43aa209-0d35-4525-2170-eb66f4e3cee6 Address:e43aa209-0d35-4525-2170-eb66f4e3cee6}]"
2025-10-30T18:19:16.510Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:16.510Z [INFO]  server-e43aa209-0d35-4525-2170-eb66f4e3cee6: entering follower state: follower="Node at e43aa209-0d35-4525-2170-eb66f4e3cee6 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.567Z [WARN]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:16.567Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: entering candidate state: node="Node at 7030b20f-c40c-43d2-361d-5572c0e40e94 [Candidate]" term=2
2025-10-30T18:19:16.567Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: election won: term=2 tally=2
2025-10-30T18:19:16.567Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: entering leader state: leader="Node at 7030b20f-c40c-43d2-361d-5572c0e40e94 [Leader]"
2025-10-30T18:19:16.567Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: added peer, starting replication: peer=server-c6194370-bdf8-f920-7a3e-e0d629f35383
2025-10-30T18:19:16.567Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: added peer, starting replication: peer=server-e43aa209-0d35-4525-2170-eb66f4e3cee6
2025-10-30T18:19:16.568Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: pipelining replication: peer="{Voter server-e43aa209-0d35-4525-2170-eb66f4e3cee6 e43aa209-0d35-4525-2170-eb66f4e3cee6}"
2025-10-30T18:19:16.568Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: pipelining replication: peer="{Voter server-c6194370-bdf8-f920-7a3e-e0d629f35383 c6194370-bdf8-f920-7a3e-e0d629f35383}"
2025-10-30T18:19:16.674Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:16.567998487 +0000 UTC m=+102.846672443 (2 nodes), 57.567362ms from start of poll, 57.754841ms from cluster start. Timeout at 2025-10-30 18:19:16.673007074 +0000 UTC m=+102.951681076, 105.008633ms after stability
2025-10-30T18:19:16.674Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:16.780Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:16.674373653 +0000 UTC m=+102.953047620 (1 nodes), 1.605µs from start of poll, 164.130018ms from cluster start. Timeout at 2025-10-30 18:19:16.779399217 +0000 UTC m=+103.058073115, 105.025495ms after stability
2025-10-30T18:19:16.780Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: aborting pipeline replication: peer="{Voter server-e43aa209-0d35-4525-2170-eb66f4e3cee6 e43aa209-0d35-4525-2170-eb66f4e3cee6}"
2025-10-30T18:19:16.780Z [INFO]  server-7030b20f-c40c-43d2-361d-5572c0e40e94: aborting pipeline replication: peer="{Voter server-c6194370-bdf8-f920-7a3e-e0d629f35383 c6194370-bdf8-f920-7a3e-e0d629f35383}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-10-30T18:19:16.780Z [INFO]  server-9198ff2a-70f9-06e1-00bb-0b84d2192c90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 Address:9198ff2a-70f9-06e1-00bb-0b84d2192c90} {Suffrage:Voter ID:server-ff9db0e7-abab-3741-a819-52041abca8f0 Address:ff9db0e7-abab-3741-a819-52041abca8f0} {Suffrage:Voter ID:server-329d1528-de09-adf4-689b-8190f06b39d8 Address:329d1528-de09-adf4-689b-8190f06b39d8}]"
2025-10-30T18:19:16.781Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 Address:9198ff2a-70f9-06e1-00bb-0b84d2192c90} {Suffrage:Voter ID:server-ff9db0e7-abab-3741-a819-52041abca8f0 Address:ff9db0e7-abab-3741-a819-52041abca8f0} {Suffrage:Voter ID:server-329d1528-de09-adf4-689b-8190f06b39d8 Address:329d1528-de09-adf4-689b-8190f06b39d8}]"
2025-10-30T18:19:16.781Z [INFO]  server-9198ff2a-70f9-06e1-00bb-0b84d2192c90: entering follower state: follower="Node at 9198ff2a-70f9-06e1-00bb-0b84d2192c90 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.781Z [INFO]  server-329d1528-de09-adf4-689b-8190f06b39d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 Address:9198ff2a-70f9-06e1-00bb-0b84d2192c90} {Suffrage:Voter ID:server-ff9db0e7-abab-3741-a819-52041abca8f0 Address:ff9db0e7-abab-3741-a819-52041abca8f0} {Suffrage:Voter ID:server-329d1528-de09-adf4-689b-8190f06b39d8 Address:329d1528-de09-adf4-689b-8190f06b39d8}]"
2025-10-30T18:19:16.781Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: entering follower state: follower="Node at ff9db0e7-abab-3741-a819-52041abca8f0 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.781Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:16.781Z [INFO]  server-329d1528-de09-adf4-689b-8190f06b39d8: entering follower state: follower="Node at 329d1528-de09-adf4-689b-8190f06b39d8 [Follower]" leader-address= leader-id=
2025-10-30T18:19:16.838Z [WARN]  server-ff9db0e7-abab-3741-a819-52041abca8f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:16.838Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: entering candidate state: node="Node at ff9db0e7-abab-3741-a819-52041abca8f0 [Candidate]" term=2
2025-10-30T18:19:16.838Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: election won: term=2 tally=2
2025-10-30T18:19:16.838Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: entering leader state: leader="Node at ff9db0e7-abab-3741-a819-52041abca8f0 [Leader]"
2025-10-30T18:19:16.838Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: added peer, starting replication: peer=server-9198ff2a-70f9-06e1-00bb-0b84d2192c90
2025-10-30T18:19:16.838Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: added peer, starting replication: peer=server-329d1528-de09-adf4-689b-8190f06b39d8
2025-10-30T18:19:16.838Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: pipelining replication: peer="{Voter server-329d1528-de09-adf4-689b-8190f06b39d8 329d1528-de09-adf4-689b-8190f06b39d8}"
2025-10-30T18:19:16.838Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: pipelining replication: peer="{Voter server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 9198ff2a-70f9-06e1-00bb-0b84d2192c90}"
2025-10-30T18:19:16.943Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:16.838186085 +0000 UTC m=+103.116859986 (1 nodes), 57.024257ms from start of poll, 57.282854ms from cluster start. Timeout at 2025-10-30 18:19:16.943190987 +0000 UTC m=+103.221864885, 105.004899ms after stability
2025-10-30T18:19:16.943Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.049Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:16.943970108 +0000 UTC m=+103.222644005 (1 nodes), 739ns from start of poll, 163.066873ms from cluster start. Timeout at 2025-10-30 18:19:17.048986705 +0000 UTC m=+103.327660604, 105.016599ms after stability
2025-10-30T18:19:17.049Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.154Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.049426365 +0000 UTC m=+103.328100264 (1 nodes), 564ns from start of poll, 268.523132ms from cluster start. Timeout at 2025-10-30 18:19:17.154436693 +0000 UTC m=+103.433110619, 105.010355ms after stability
2025-10-30T18:19:17.154Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.260Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.154889451 +0000 UTC m=+103.433563348 (1 nodes), 483ns from start of poll, 373.986216ms from cluster start. Timeout at 2025-10-30 18:19:17.259907096 +0000 UTC m=+103.538581011, 105.017663ms after stability
2025-10-30T18:19:17.260Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.366Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.260488994 +0000 UTC m=+103.539162891 (1 nodes), 596ns from start of poll, 479.585759ms from cluster start. Timeout at 2025-10-30 18:19:17.365500804 +0000 UTC m=+103.644174719, 105.011828ms after stability
2025-10-30T18:19:17.366Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.472Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.366508448 +0000 UTC m=+103.645182345 (1 nodes), 548ns from start of poll, 585.605213ms from cluster start. Timeout at 2025-10-30 18:19:17.471524973 +0000 UTC m=+103.750198888, 105.016543ms after stability
2025-10-30T18:19:17.578Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.684Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.578218462 +0000 UTC m=+103.856892360 (1 nodes), 1.057µs from start of poll, 797.315228ms from cluster start. Timeout at 2025-10-30 18:19:17.683238543 +0000 UTC m=+103.961912469, 105.020109ms after stability
2025-10-30T18:19:17.684Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: aborting pipeline replication: peer="{Voter server-329d1528-de09-adf4-689b-8190f06b39d8 329d1528-de09-adf4-689b-8190f06b39d8}"
2025-10-30T18:19:17.684Z [INFO]  server-ff9db0e7-abab-3741-a819-52041abca8f0: aborting pipeline replication: peer="{Voter server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 9198ff2a-70f9-06e1-00bb-0b84d2192c90}"
--- PASS: TestRaft_LeaderID_Propagated (0.90s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-10-30T18:19:17.685Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47071578-4369-2c98-ab20-01a9096717e9 Address:47071578-4369-2c98-ab20-01a9096717e9} {Suffrage:Voter ID:server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 Address:4866ae6b-dc4a-a1d1-4f2d-15976e391147} {Suffrage:Voter ID:server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff Address:9d9636fd-26a6-15e4-9ec2-c631de0a94ff}]"
2025-10-30T18:19:17.685Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: entering follower state: follower="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Follower]" leader-address= leader-id=
2025-10-30T18:19:17.685Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47071578-4369-2c98-ab20-01a9096717e9 Address:47071578-4369-2c98-ab20-01a9096717e9} {Suffrage:Voter ID:server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 Address:4866ae6b-dc4a-a1d1-4f2d-15976e391147} {Suffrage:Voter ID:server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff Address:9d9636fd-26a6-15e4-9ec2-c631de0a94ff}]"
2025-10-30T18:19:17.685Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: entering follower state: follower="Node at 4866ae6b-dc4a-a1d1-4f2d-15976e391147 [Follower]" leader-address= leader-id=
2025-10-30T18:19:17.685Z [INFO]  server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47071578-4369-2c98-ab20-01a9096717e9 Address:47071578-4369-2c98-ab20-01a9096717e9} {Suffrage:Voter ID:server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 Address:4866ae6b-dc4a-a1d1-4f2d-15976e391147} {Suffrage:Voter ID:server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff Address:9d9636fd-26a6-15e4-9ec2-c631de0a94ff}]"
2025-10-30T18:19:17.685Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.685Z [INFO]  server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff: entering follower state: follower="Node at 9d9636fd-26a6-15e4-9ec2-c631de0a94ff [Follower]" leader-address= leader-id=
2025-10-30T18:19:17.747Z [WARN]  server-47071578-4369-2c98-ab20-01a9096717e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:17.747Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: entering candidate state: node="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Candidate]" term=2
2025-10-30T18:19:17.748Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: election won: term=2 tally=2
2025-10-30T18:19:17.748Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: entering leader state: leader="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Leader]"
2025-10-30T18:19:17.748Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: added peer, starting replication: peer=server-4866ae6b-dc4a-a1d1-4f2d-15976e391147
2025-10-30T18:19:17.748Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: added peer, starting replication: peer=server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff
2025-10-30T18:19:17.748Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: pipelining replication: peer="{Voter server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 4866ae6b-dc4a-a1d1-4f2d-15976e391147}"
2025-10-30T18:19:17.748Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: pipelining replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}"
2025-10-30T18:19:17.853Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.748056234 +0000 UTC m=+104.026730131 (1 nodes), 62.466257ms from start of poll, 62.712596ms from cluster start. Timeout at 2025-10-30 18:19:17.853057959 +0000 UTC m=+104.131731856, 105.001725ms after stability
2025-10-30T18:19:17.853Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.959Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.853250683 +0000 UTC m=+104.131924584 (1 nodes), 1.47µs from start of poll, 167.907049ms from cluster start. Timeout at 2025-10-30 18:19:17.958268955 +0000 UTC m=+104.236942875, 105.018291ms after stability
2025-10-30T18:19:17.959Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: entering candidate state: node="Node at 4866ae6b-dc4a-a1d1-4f2d-15976e391147 [Candidate]" term=3
2025-10-30T18:19:17.959Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:17.959Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: entering follower state: follower="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Follower]" leader-address= leader-id=
2025-10-30T18:19:17.959Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: aborting pipeline replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}"
2025-10-30T18:19:17.959Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: election won: term=3 tally=2
2025-10-30T18:19:17.959Z [INFO]  server-47071578-4369-2c98-ab20-01a9096717e9: aborting pipeline replication: peer="{Voter server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 4866ae6b-dc4a-a1d1-4f2d-15976e391147}"
2025-10-30T18:19:17.959Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: entering leader state: leader="Node at 4866ae6b-dc4a-a1d1-4f2d-15976e391147 [Leader]"
2025-10-30T18:19:17.959Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: added peer, starting replication: peer=server-47071578-4369-2c98-ab20-01a9096717e9
2025-10-30T18:19:17.959Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: added peer, starting replication: peer=server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff
2025-10-30T18:19:17.959Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: pipelining replication: peer="{Voter server-47071578-4369-2c98-ab20-01a9096717e9 47071578-4369-2c98-ab20-01a9096717e9}"
2025-10-30T18:19:17.959Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: pipelining replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}"
2025-10-30T18:19:18.064Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:17.959483337 +0000 UTC m=+104.238157237 (1 nodes), 67.887µs from start of poll, 274.139702ms from cluster start. Timeout at 2025-10-30 18:19:18.064560628 +0000 UTC m=+104.343234524, 105.077287ms after stability
2025-10-30T18:19:18.064Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: aborting pipeline replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}"
2025-10-30T18:19:18.065Z [INFO]  server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: aborting pipeline replication: peer="{Voter server-47071578-4369-2c98-ab20-01a9096717e9 47071578-4369-2c98-ab20-01a9096717e9}"
2025-10-30T18:19:18.565Z [ERROR] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: failed to heartbeat to: peer=9d9636fd-26a6-15e4-9ec2-c631de0a94ff backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransfer (0.88s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-10-30T18:19:18.566Z [INFO]  server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ece6e09-724c-c5be-a0e3-1710b10f65c3 Address:4ece6e09-724c-c5be-a0e3-1710b10f65c3}]"
2025-10-30T18:19:18.566Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:18.566Z [INFO]  server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: entering follower state: follower="Node at 4ece6e09-724c-c5be-a0e3-1710b10f65c3 [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.635Z [WARN]  server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:18.635Z [INFO]  server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: entering candidate state: node="Node at 4ece6e09-724c-c5be-a0e3-1710b10f65c3 [Candidate]" term=2
2025-10-30T18:19:18.635Z [INFO]  server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: election won: term=2 tally=1
2025-10-30T18:19:18.635Z [INFO]  server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: entering leader state: leader="Node at 4ece6e09-724c-c5be-a0e3-1710b10f65c3 [Leader]"
2025-10-30T18:19:18.740Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:18.635131062 +0000 UTC m=+104.913804973 (1 nodes), 68.737698ms from start of poll, 68.859778ms from cluster start. Timeout at 2025-10-30 18:19:18.740133039 +0000 UTC m=+105.018806936, 105.001963ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-10-30T18:19:18.741Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]"
2025-10-30T18:19:18.741Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: entering follower state: follower="Node at f6350e70-f150-321b-0246-33eabaad8d7c [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.741Z [INFO]  server-ad273008-82c9-92fd-612b-af9ff515bcf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]"
2025-10-30T18:19:18.742Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]"
2025-10-30T18:19:18.742Z [INFO]  server-ad273008-82c9-92fd-612b-af9ff515bcf2: entering follower state: follower="Node at ad273008-82c9-92fd-612b-af9ff515bcf2 [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.742Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering follower state: follower="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.742Z [INFO]  server-8ba7c1e3-74e7-3184-0426-968ea78aa51d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]"
2025-10-30T18:19:18.742Z [INFO]  server-8ba7c1e3-74e7-3184-0426-968ea78aa51d: entering follower state: follower="Node at 8ba7c1e3-74e7-3184-0426-968ea78aa51d [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.742Z [INFO]  server-af119d47-ed92-4917-5b41-5baf3da5f549: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]"
2025-10-30T18:19:18.742Z [INFO]  server-af119d47-ed92-4917-5b41-5baf3da5f549: entering follower state: follower="Node at af119d47-ed92-4917-5b41-5baf3da5f549 [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.742Z [INFO]  server-627401af-c75a-b265-6083-abf09ffa2855: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]"
2025-10-30T18:19:18.743Z [INFO]  server-646fc452-30a0-b610-72a7-583056961d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]"
2025-10-30T18:19:18.743Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:18.743Z [INFO]  server-627401af-c75a-b265-6083-abf09ffa2855: entering follower state: follower="Node at 627401af-c75a-b265-6083-abf09ffa2855 [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.743Z [INFO]  server-646fc452-30a0-b610-72a7-583056961d5d: entering follower state: follower="Node at 646fc452-30a0-b610-72a7-583056961d5d [Follower]" leader-address= leader-id=
2025-10-30T18:19:18.803Z [WARN]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering candidate state: node="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Candidate]" term=2
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: election won: term=2 tally=4
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering leader state: leader="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Leader]"
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-f6350e70-f150-321b-0246-33eabaad8d7c
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-ad273008-82c9-92fd-612b-af9ff515bcf2
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-8ba7c1e3-74e7-3184-0426-968ea78aa51d
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-af119d47-ed92-4917-5b41-5baf3da5f549
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-627401af-c75a-b265-6083-abf09ffa2855
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-646fc452-30a0-b610-72a7-583056961d5d
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}"
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}"
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}"
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-f6350e70-f150-321b-0246-33eabaad8d7c f6350e70-f150-321b-0246-33eabaad8d7c}"
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}"
2025-10-30T18:19:18.803Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}"
2025-10-30T18:19:18.909Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:18.803553715 +0000 UTC m=+105.082227612 (1 nodes), 60.328763ms from start of poll, 61.734374ms from cluster start. Timeout at 2025-10-30 18:19:18.908555941 +0000 UTC m=+105.187229839, 105.002227ms after stability
2025-10-30T18:19:18.909Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:19.015Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:18.909620838 +0000 UTC m=+105.188294735 (6 nodes), 1.801µs from start of poll, 167.801497ms from cluster start. Timeout at 2025-10-30 18:19:19.014638026 +0000 UTC m=+105.293311922, 105.017187ms after stability
2025-10-30T18:19:19.015Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:19.121Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:19.015318083 +0000 UTC m=+105.293991992 (1 nodes), 2.028µs from start of poll, 273.498754ms from cluster start. Timeout at 2025-10-30 18:19:19.12033941 +0000 UTC m=+105.399013317, 105.021325ms after stability
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: entering candidate state: node="Node at f6350e70-f150-321b-0246-33eabaad8d7c [Candidate]" term=3
2025-10-30T18:19:19.121Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: election won: term=3 tally=4
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: entering leader state: leader="Node at f6350e70-f150-321b-0246-33eabaad8d7c [Leader]"
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-ad273008-82c9-92fd-612b-af9ff515bcf2
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-8ba7c1e3-74e7-3184-0426-968ea78aa51d
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-af119d47-ed92-4917-5b41-5baf3da5f549
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-627401af-c75a-b265-6083-abf09ffa2855
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-646fc452-30a0-b610-72a7-583056961d5d
2025-10-30T18:19:19.121Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering follower state: follower="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Follower]" leader-address= leader-id=
2025-10-30T18:19:19.121Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}"
2025-10-30T18:19:19.121Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}"
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}"
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}"
2025-10-30T18:19:19.121Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}"
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5}"
2025-10-30T18:19:19.121Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}"
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}"
2025-10-30T18:19:19.121Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}"
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}"
2025-10-30T18:19:19.121Z [INFO]  server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-f6350e70-f150-321b-0246-33eabaad8d7c f6350e70-f150-321b-0246-33eabaad8d7c}"
2025-10-30T18:19:19.121Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}"
2025-10-30T18:19:19.226Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:19.121376329 +0000 UTC m=+105.400050226 (1 nodes), 73.956µs from start of poll, 379.556988ms from cluster start. Timeout at 2025-10-30 18:19:19.226398439 +0000 UTC m=+105.505072354, 105.022128ms after stability
2025-10-30T18:19:19.227Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}"
2025-10-30T18:19:19.227Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}"
2025-10-30T18:19:19.227Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}"
2025-10-30T18:19:19.227Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5}"
2025-10-30T18:19:19.227Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}"
2025-10-30T18:19:19.227Z [INFO]  server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}"
2025-10-30T18:19:19.727Z [ERROR] server-f6350e70-f150-321b-0246-33eabaad8d7c: failed to heartbeat to: peer=ad273008-82c9-92fd-612b-af9ff515bcf2 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.99s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-10-30T18:19:19.728Z [INFO]  server-06d01780-d7b5-3c1b-6649-648a00bb9517: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06d01780-d7b5-3c1b-6649-648a00bb9517 Address:06d01780-d7b5-3c1b-6649-648a00bb9517} {Suffrage:Voter ID:server-c2316925-47e6-dcc4-9570-a76f6f244b07 Address:c2316925-47e6-dcc4-9570-a76f6f244b07} {Suffrage:Voter ID:server-97c82631-dd2e-1258-25b9-5cd8830525ce Address:97c82631-dd2e-1258-25b9-5cd8830525ce}]"
2025-10-30T18:19:19.728Z [INFO]  server-06d01780-d7b5-3c1b-6649-648a00bb9517: entering follower state: follower="Node at 06d01780-d7b5-3c1b-6649-648a00bb9517 [Follower]" leader-address= leader-id=
2025-10-30T18:19:19.728Z [INFO]  server-c2316925-47e6-dcc4-9570-a76f6f244b07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06d01780-d7b5-3c1b-6649-648a00bb9517 Address:06d01780-d7b5-3c1b-6649-648a00bb9517} {Suffrage:Voter ID:server-c2316925-47e6-dcc4-9570-a76f6f244b07 Address:c2316925-47e6-dcc4-9570-a76f6f244b07} {Suffrage:Voter ID:server-97c82631-dd2e-1258-25b9-5cd8830525ce Address:97c82631-dd2e-1258-25b9-5cd8830525ce}]"
2025-10-30T18:19:19.728Z [INFO]  server-c2316925-47e6-dcc4-9570-a76f6f244b07: entering follower state: follower="Node at c2316925-47e6-dcc4-9570-a76f6f244b07 [Follower]" leader-address= leader-id=
2025-10-30T18:19:19.729Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06d01780-d7b5-3c1b-6649-648a00bb9517 Address:06d01780-d7b5-3c1b-6649-648a00bb9517} {Suffrage:Voter ID:server-c2316925-47e6-dcc4-9570-a76f6f244b07 Address:c2316925-47e6-dcc4-9570-a76f6f244b07} {Suffrage:Voter ID:server-97c82631-dd2e-1258-25b9-5cd8830525ce Address:97c82631-dd2e-1258-25b9-5cd8830525ce}]"
2025-10-30T18:19:19.729Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:19.729Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: entering follower state: follower="Node at 97c82631-dd2e-1258-25b9-5cd8830525ce [Follower]" leader-address= leader-id=
2025-10-30T18:19:19.823Z [WARN]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:19.823Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: entering candidate state: node="Node at 97c82631-dd2e-1258-25b9-5cd8830525ce [Candidate]" term=2
2025-10-30T18:19:19.823Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: election won: term=2 tally=2
2025-10-30T18:19:19.823Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: entering leader state: leader="Node at 97c82631-dd2e-1258-25b9-5cd8830525ce [Leader]"
2025-10-30T18:19:19.823Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: added peer, starting replication: peer=server-06d01780-d7b5-3c1b-6649-648a00bb9517
2025-10-30T18:19:19.823Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: added peer, starting replication: peer=server-c2316925-47e6-dcc4-9570-a76f6f244b07
2025-10-30T18:19:19.823Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: pipelining replication: peer="{Voter server-c2316925-47e6-dcc4-9570-a76f6f244b07 c2316925-47e6-dcc4-9570-a76f6f244b07}"
2025-10-30T18:19:19.823Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: pipelining replication: peer="{Voter server-06d01780-d7b5-3c1b-6649-648a00bb9517 06d01780-d7b5-3c1b-6649-648a00bb9517}"
2025-10-30T18:19:19.929Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:19.82365916 +0000 UTC m=+106.102333058 (1 nodes), 94.577523ms from start of poll, 94.913226ms from cluster start. Timeout at 2025-10-30 18:19:19.928665145 +0000 UTC m=+106.207339043, 105.005985ms after stability
2025-10-30T18:19:19.929Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: aborting pipeline replication: peer="{Voter server-c2316925-47e6-dcc4-9570-a76f6f244b07 c2316925-47e6-dcc4-9570-a76f6f244b07}"
2025-10-30T18:19:19.929Z [INFO]  server-97c82631-dd2e-1258-25b9-5cd8830525ce: aborting pipeline replication: peer="{Voter server-06d01780-d7b5-3c1b-6649-648a00bb9517 06d01780-d7b5-3c1b-6649-648a00bb9517}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.20s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-10-30T18:19:19.930Z [INFO]  server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 Address:2da7005f-7e4e-e9cc-a9bc-26c7e70156d9} {Suffrage:Voter ID:server-a0d1989a-6eb7-1bed-ff62-8987cad84b42 Address:a0d1989a-6eb7-1bed-ff62-8987cad84b42} {Suffrage:Voter ID:server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 Address:6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}]"
2025-10-30T18:19:19.930Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 Address:2da7005f-7e4e-e9cc-a9bc-26c7e70156d9} {Suffrage:Voter ID:server-a0d1989a-6eb7-1bed-ff62-8987cad84b42 Address:a0d1989a-6eb7-1bed-ff62-8987cad84b42} {Suffrage:Voter ID:server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 Address:6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}]"
2025-10-30T18:19:19.930Z [INFO]  server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9: entering follower state: follower="Node at 2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 [Follower]" leader-address= leader-id=
2025-10-30T18:19:19.930Z [INFO]  server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 Address:2da7005f-7e4e-e9cc-a9bc-26c7e70156d9} {Suffrage:Voter ID:server-a0d1989a-6eb7-1bed-ff62-8987cad84b42 Address:a0d1989a-6eb7-1bed-ff62-8987cad84b42} {Suffrage:Voter ID:server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 Address:6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}]"
2025-10-30T18:19:19.930Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: entering follower state: follower="Node at a0d1989a-6eb7-1bed-ff62-8987cad84b42 [Follower]" leader-address= leader-id=
2025-10-30T18:19:19.930Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:19.930Z [INFO]  server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784: entering follower state: follower="Node at 6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 [Follower]" leader-address= leader-id=
2025-10-30T18:19:19.985Z [WARN]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:19.985Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: entering candidate state: node="Node at a0d1989a-6eb7-1bed-ff62-8987cad84b42 [Candidate]" term=2
2025-10-30T18:19:19.985Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: election won: term=2 tally=2
2025-10-30T18:19:19.985Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: entering leader state: leader="Node at a0d1989a-6eb7-1bed-ff62-8987cad84b42 [Leader]"
2025-10-30T18:19:19.985Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: added peer, starting replication: peer=server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9
2025-10-30T18:19:19.985Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: added peer, starting replication: peer=server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784
2025-10-30T18:19:19.985Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: pipelining replication: peer="{Voter server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}"
2025-10-30T18:19:19.985Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: pipelining replication: peer="{Voter server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 2da7005f-7e4e-e9cc-a9bc-26c7e70156d9}"
2025-10-30T18:19:20.091Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:19.985495834 +0000 UTC m=+106.264169730 (2 nodes), 55.26613ms from start of poll, 55.513154ms from cluster start. Timeout at 2025-10-30 18:19:20.09049638 +0000 UTC m=+106.369170277, 105.000547ms after stability
2025-10-30T18:19:20.091Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:20.196Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:20.091108403 +0000 UTC m=+106.369782312 (1 nodes), 1.829µs from start of poll, 161.125736ms from cluster start. Timeout at 2025-10-30 18:19:20.196131257 +0000 UTC m=+106.474805172, 105.02286ms after stability
2025-10-30T18:19:20.196Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: aborting pipeline replication: peer="{Voter server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}"
2025-10-30T18:19:20.196Z [INFO]  server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: aborting pipeline replication: peer="{Voter server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 2da7005f-7e4e-e9cc-a9bc-26c7e70156d9}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-10-30T18:19:20.197Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4aaee5-22de-b546-297d-acc9c20b31e2 Address:8e4aaee5-22de-b546-297d-acc9c20b31e2} {Suffrage:Voter ID:server-513b9a68-100d-87a9-3572-bedd306b52a7 Address:513b9a68-100d-87a9-3572-bedd306b52a7} {Suffrage:Voter ID:server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c Address:a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}]"
2025-10-30T18:19:20.197Z [INFO]  server-513b9a68-100d-87a9-3572-bedd306b52a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4aaee5-22de-b546-297d-acc9c20b31e2 Address:8e4aaee5-22de-b546-297d-acc9c20b31e2} {Suffrage:Voter ID:server-513b9a68-100d-87a9-3572-bedd306b52a7 Address:513b9a68-100d-87a9-3572-bedd306b52a7} {Suffrage:Voter ID:server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c Address:a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}]"
2025-10-30T18:19:20.197Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: entering follower state: follower="Node at 8e4aaee5-22de-b546-297d-acc9c20b31e2 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.197Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4aaee5-22de-b546-297d-acc9c20b31e2 Address:8e4aaee5-22de-b546-297d-acc9c20b31e2} {Suffrage:Voter ID:server-513b9a68-100d-87a9-3572-bedd306b52a7 Address:513b9a68-100d-87a9-3572-bedd306b52a7} {Suffrage:Voter ID:server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c Address:a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}]"
2025-10-30T18:19:20.197Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:20.197Z [INFO]  server-513b9a68-100d-87a9-3572-bedd306b52a7: entering follower state: follower="Node at 513b9a68-100d-87a9-3572-bedd306b52a7 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.197Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering follower state: follower="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.252Z [WARN]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:20.252Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering candidate state: node="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Candidate]" term=2
2025-10-30T18:19:20.252Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: election won: term=2 tally=2
2025-10-30T18:19:20.252Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering leader state: leader="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Leader]"
2025-10-30T18:19:20.253Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: added peer, starting replication: peer=server-8e4aaee5-22de-b546-297d-acc9c20b31e2
2025-10-30T18:19:20.253Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: added peer, starting replication: peer=server-513b9a68-100d-87a9-3572-bedd306b52a7
2025-10-30T18:19:20.253Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: pipelining replication: peer="{Voter server-8e4aaee5-22de-b546-297d-acc9c20b31e2 8e4aaee5-22de-b546-297d-acc9c20b31e2}"
2025-10-30T18:19:20.253Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: pipelining replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}"
2025-10-30T18:19:20.358Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:20.253046761 +0000 UTC m=+106.531720657 (1 nodes), 55.57619ms from start of poll, 55.752564ms from cluster start. Timeout at 2025-10-30 18:19:20.358048865 +0000 UTC m=+106.636722762, 105.002105ms after stability
2025-10-30T18:19:20.358Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:20.464Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:20.358195087 +0000 UTC m=+106.636868984 (2 nodes), 1.14µs from start of poll, 160.900891ms from cluster start. Timeout at 2025-10-30 18:19:20.463211463 +0000 UTC m=+106.741885377, 105.016393ms after stability
2025-10-30T18:19:20.487Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: entering candidate state: node="Node at 8e4aaee5-22de-b546-297d-acc9c20b31e2 [Candidate]" term=3
2025-10-30T18:19:20.487Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:20.487Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering follower state: follower="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.487Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: election won: term=3 tally=2
2025-10-30T18:19:20.487Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: aborting pipeline replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}"
2025-10-30T18:19:20.487Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: entering leader state: leader="Node at 8e4aaee5-22de-b546-297d-acc9c20b31e2 [Leader]"
2025-10-30T18:19:20.487Z [INFO]  server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: aborting pipeline replication: peer="{Voter server-8e4aaee5-22de-b546-297d-acc9c20b31e2 8e4aaee5-22de-b546-297d-acc9c20b31e2}"
2025-10-30T18:19:20.487Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: added peer, starting replication: peer=server-513b9a68-100d-87a9-3572-bedd306b52a7
2025-10-30T18:19:20.487Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: added peer, starting replication: peer=server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c
2025-10-30T18:19:20.487Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: pipelining replication: peer="{Voter server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}"
2025-10-30T18:19:20.487Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: pipelining replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}"
2025-10-30T18:19:20.593Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:20.487483266 +0000 UTC m=+106.766157166 (1 nodes), 43.874µs from start of poll, 290.189073ms from cluster start. Timeout at 2025-10-30 18:19:20.592516616 +0000 UTC m=+106.871190543, 105.033377ms after stability
2025-10-30T18:19:20.593Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: aborting pipeline replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}"
2025-10-30T18:19:20.593Z [INFO]  server-8e4aaee5-22de-b546-297d-acc9c20b31e2: aborting pipeline replication: peer="{Voter server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.40s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-10-30T18:19:20.593Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 Address:ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49} {Suffrage:Voter ID:server-09fec08a-0dbb-78e4-7618-a30d320038fb Address:09fec08a-0dbb-78e4-7618-a30d320038fb} {Suffrage:Voter ID:server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 Address:0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}]"
2025-10-30T18:19:20.593Z [INFO]  server-09fec08a-0dbb-78e4-7618-a30d320038fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 Address:ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49} {Suffrage:Voter ID:server-09fec08a-0dbb-78e4-7618-a30d320038fb Address:09fec08a-0dbb-78e4-7618-a30d320038fb} {Suffrage:Voter ID:server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 Address:0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}]"
2025-10-30T18:19:20.593Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: entering follower state: follower="Node at ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.593Z [INFO]  server-09fec08a-0dbb-78e4-7618-a30d320038fb: entering follower state: follower="Node at 09fec08a-0dbb-78e4-7618-a30d320038fb [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.593Z [INFO]  server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 Address:ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49} {Suffrage:Voter ID:server-09fec08a-0dbb-78e4-7618-a30d320038fb Address:09fec08a-0dbb-78e4-7618-a30d320038fb} {Suffrage:Voter ID:server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 Address:0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}]"
2025-10-30T18:19:20.593Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:20.593Z [INFO]  server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70: entering follower state: follower="Node at 0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.665Z [WARN]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:20.665Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: entering candidate state: node="Node at ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 [Candidate]" term=2
2025-10-30T18:19:20.665Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: election won: term=2 tally=2
2025-10-30T18:19:20.665Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: entering leader state: leader="Node at ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 [Leader]"
2025-10-30T18:19:20.665Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: added peer, starting replication: peer=server-09fec08a-0dbb-78e4-7618-a30d320038fb
2025-10-30T18:19:20.665Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: added peer, starting replication: peer=server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70
2025-10-30T18:19:20.665Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: pipelining replication: peer="{Voter server-09fec08a-0dbb-78e4-7618-a30d320038fb 09fec08a-0dbb-78e4-7618-a30d320038fb}"
2025-10-30T18:19:20.665Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: pipelining replication: peer="{Voter server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}"
2025-10-30T18:19:20.771Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:20.665193252 +0000 UTC m=+106.943867149 (1 nodes), 71.222273ms from start of poll, 71.413698ms from cluster start. Timeout at 2025-10-30 18:19:20.77019608 +0000 UTC m=+107.048869977, 105.002828ms after stability
2025-10-30T18:19:20.771Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: cannot transfer leadership to itself
2025-10-30T18:19:20.771Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: aborting pipeline replication: peer="{Voter server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}"
2025-10-30T18:19:20.771Z [INFO]  server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: aborting pipeline replication: peer="{Voter server-09fec08a-0dbb-78e4-7618-a30d320038fb 09fec08a-0dbb-78e4-7618-a30d320038fb}"
--- PASS: TestRaft_LeadershipTransferToItself (0.18s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-10-30T18:19:20.771Z [INFO]  server-60942810-3fdc-b2a7-f08f-e8d847f4ead5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 Address:60942810-3fdc-b2a7-f08f-e8d847f4ead5} {Suffrage:Voter ID:server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 Address:7e0cd94e-84ab-f3f0-066d-f89e7fffead9} {Suffrage:Voter ID:server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5 Address:abfd1a5d-9716-03d2-5ca1-834aa14da6a5}]"
2025-10-30T18:19:20.772Z [INFO]  server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 Address:60942810-3fdc-b2a7-f08f-e8d847f4ead5} {Suffrage:Voter ID:server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 Address:7e0cd94e-84ab-f3f0-066d-f89e7fffead9} {Suffrage:Voter ID:server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5 Address:abfd1a5d-9716-03d2-5ca1-834aa14da6a5}]"
2025-10-30T18:19:20.771Z [INFO]  server-60942810-3fdc-b2a7-f08f-e8d847f4ead5: entering follower state: follower="Node at 60942810-3fdc-b2a7-f08f-e8d847f4ead5 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.772Z [INFO]  server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9: entering follower state: follower="Node at 7e0cd94e-84ab-f3f0-066d-f89e7fffead9 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.772Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 Address:60942810-3fdc-b2a7-f08f-e8d847f4ead5} {Suffrage:Voter ID:server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 Address:7e0cd94e-84ab-f3f0-066d-f89e7fffead9} {Suffrage:Voter ID:server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5 Address:abfd1a5d-9716-03d2-5ca1-834aa14da6a5}]"
2025-10-30T18:19:20.772Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:20.772Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: entering follower state: follower="Node at abfd1a5d-9716-03d2-5ca1-834aa14da6a5 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.824Z [WARN]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:20.824Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: entering candidate state: node="Node at abfd1a5d-9716-03d2-5ca1-834aa14da6a5 [Candidate]" term=2
2025-10-30T18:19:20.824Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: election won: term=2 tally=2
2025-10-30T18:19:20.824Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: entering leader state: leader="Node at abfd1a5d-9716-03d2-5ca1-834aa14da6a5 [Leader]"
2025-10-30T18:19:20.824Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: added peer, starting replication: peer=server-60942810-3fdc-b2a7-f08f-e8d847f4ead5
2025-10-30T18:19:20.824Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: added peer, starting replication: peer=server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9
2025-10-30T18:19:20.824Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: pipelining replication: peer="{Voter server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 60942810-3fdc-b2a7-f08f-e8d847f4ead5}"
2025-10-30T18:19:20.824Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: pipelining replication: peer="{Voter server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 7e0cd94e-84ab-f3f0-066d-f89e7fffead9}"
2025-10-30T18:19:20.930Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:20.824164711 +0000 UTC m=+107.102838608 (1 nodes), 52.071538ms from start of poll, 52.289366ms from cluster start. Timeout at 2025-10-30 18:19:20.929165205 +0000 UTC m=+107.207839102, 105.000494ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-10-30T18:19:20.930Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: aborting pipeline replication: peer="{Voter server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 7e0cd94e-84ab-f3f0-066d-f89e7fffead9}"
2025-10-30T18:19:20.930Z [INFO]  server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: aborting pipeline replication: peer="{Voter server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 60942810-3fdc-b2a7-f08f-e8d847f4ead5}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-10-30T18:19:20.931Z [INFO]  server-fa35787c-946b-1b2b-d093-7511376ec0f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa35787c-946b-1b2b-d093-7511376ec0f9 Address:fa35787c-946b-1b2b-d093-7511376ec0f9} {Suffrage:Voter ID:server-2efc3f40-7050-ad1e-0635-0e2d7807522b Address:2efc3f40-7050-ad1e-0635-0e2d7807522b} {Suffrage:Voter ID:server-4094de85-65f8-2fca-23fd-1f775544793f Address:4094de85-65f8-2fca-23fd-1f775544793f}]"
2025-10-30T18:19:20.931Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa35787c-946b-1b2b-d093-7511376ec0f9 Address:fa35787c-946b-1b2b-d093-7511376ec0f9} {Suffrage:Voter ID:server-2efc3f40-7050-ad1e-0635-0e2d7807522b Address:2efc3f40-7050-ad1e-0635-0e2d7807522b} {Suffrage:Voter ID:server-4094de85-65f8-2fca-23fd-1f775544793f Address:4094de85-65f8-2fca-23fd-1f775544793f}]"
2025-10-30T18:19:20.931Z [INFO]  server-fa35787c-946b-1b2b-d093-7511376ec0f9: entering follower state: follower="Node at fa35787c-946b-1b2b-d093-7511376ec0f9 [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.931Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: entering follower state: follower="Node at 2efc3f40-7050-ad1e-0635-0e2d7807522b [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.931Z [INFO]  server-4094de85-65f8-2fca-23fd-1f775544793f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa35787c-946b-1b2b-d093-7511376ec0f9 Address:fa35787c-946b-1b2b-d093-7511376ec0f9} {Suffrage:Voter ID:server-2efc3f40-7050-ad1e-0635-0e2d7807522b Address:2efc3f40-7050-ad1e-0635-0e2d7807522b} {Suffrage:Voter ID:server-4094de85-65f8-2fca-23fd-1f775544793f Address:4094de85-65f8-2fca-23fd-1f775544793f}]"
2025-10-30T18:19:20.931Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:20.931Z [INFO]  server-4094de85-65f8-2fca-23fd-1f775544793f: entering follower state: follower="Node at 4094de85-65f8-2fca-23fd-1f775544793f [Follower]" leader-address= leader-id=
2025-10-30T18:19:20.983Z [WARN]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:20.983Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: entering candidate state: node="Node at 2efc3f40-7050-ad1e-0635-0e2d7807522b [Candidate]" term=2
2025-10-30T18:19:20.983Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: election won: term=2 tally=2
2025-10-30T18:19:20.983Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: entering leader state: leader="Node at 2efc3f40-7050-ad1e-0635-0e2d7807522b [Leader]"
2025-10-30T18:19:20.983Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: added peer, starting replication: peer=server-fa35787c-946b-1b2b-d093-7511376ec0f9
2025-10-30T18:19:20.983Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: added peer, starting replication: peer=server-4094de85-65f8-2fca-23fd-1f775544793f
2025-10-30T18:19:20.983Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: pipelining replication: peer="{Voter server-4094de85-65f8-2fca-23fd-1f775544793f 4094de85-65f8-2fca-23fd-1f775544793f}"
2025-10-30T18:19:20.983Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: pipelining replication: peer="{Voter server-fa35787c-946b-1b2b-d093-7511376ec0f9 fa35787c-946b-1b2b-d093-7511376ec0f9}"
2025-10-30T18:19:21.088Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:20.983649135 +0000 UTC m=+107.262323033 (1 nodes), 52.39573ms from start of poll, 52.615356ms from cluster start. Timeout at 2025-10-30 18:19:21.088652125 +0000 UTC m=+107.367326023, 105.00299ms after stability
2025-10-30T18:19:21.088Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:21.194Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:21.088805807 +0000 UTC m=+107.367479705 (2 nodes), 1.684µs from start of poll, 157.772028ms from cluster start. Timeout at 2025-10-30 18:19:21.193828965 +0000 UTC m=+107.472502862, 105.023157ms after stability
2025-10-30T18:19:21.393Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: aborting pipeline replication: peer="{Voter server-4094de85-65f8-2fca-23fd-1f775544793f 4094de85-65f8-2fca-23fd-1f775544793f}"
2025-10-30T18:19:21.393Z [INFO]  server-2efc3f40-7050-ad1e-0635-0e2d7807522b: aborting pipeline replication: peer="{Voter server-fa35787c-946b-1b2b-d093-7511376ec0f9 fa35787c-946b-1b2b-d093-7511376ec0f9}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.46s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-10-30T18:19:21.393Z [INFO]  server-b374f233-db7b-bf1a-4a12-ed6c4b52db93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 Address:b374f233-db7b-bf1a-4a12-ed6c4b52db93} {Suffrage:Voter ID:server-636418b4-0b2f-925b-a178-214aab4f6ac9 Address:636418b4-0b2f-925b-a178-214aab4f6ac9}]"
2025-10-30T18:19:21.393Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 Address:b374f233-db7b-bf1a-4a12-ed6c4b52db93} {Suffrage:Voter ID:server-636418b4-0b2f-925b-a178-214aab4f6ac9 Address:636418b4-0b2f-925b-a178-214aab4f6ac9}]"
2025-10-30T18:19:21.393Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:21.393Z [INFO]  server-b374f233-db7b-bf1a-4a12-ed6c4b52db93: entering follower state: follower="Node at b374f233-db7b-bf1a-4a12-ed6c4b52db93 [Follower]" leader-address= leader-id=
2025-10-30T18:19:21.393Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: entering follower state: follower="Node at 636418b4-0b2f-925b-a178-214aab4f6ac9 [Follower]" leader-address= leader-id=
2025-10-30T18:19:21.457Z [WARN]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:21.457Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: entering candidate state: node="Node at 636418b4-0b2f-925b-a178-214aab4f6ac9 [Candidate]" term=2
2025-10-30T18:19:21.457Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: election won: term=2 tally=2
2025-10-30T18:19:21.457Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: entering leader state: leader="Node at 636418b4-0b2f-925b-a178-214aab4f6ac9 [Leader]"
2025-10-30T18:19:21.457Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: added peer, starting replication: peer=server-b374f233-db7b-bf1a-4a12-ed6c4b52db93
2025-10-30T18:19:21.457Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: pipelining replication: peer="{Voter server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 b374f233-db7b-bf1a-4a12-ed6c4b52db93}"
2025-10-30T18:19:21.562Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:21.457291338 +0000 UTC m=+107.735965235 (1 nodes), 63.391326ms from start of poll, 63.523175ms from cluster start. Timeout at 2025-10-30 18:19:21.562291941 +0000 UTC m=+107.840965838, 105.000603ms after stability
2025-10-30T18:19:21.562Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:21.668Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:21.56259575 +0000 UTC m=+107.841269647 (1 nodes), 635ns from start of poll, 168.827587ms from cluster start. Timeout at 2025-10-30 18:19:21.667682502 +0000 UTC m=+107.946356442, 105.086795ms after stability
2025-10-30T18:19:21.668Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: updating configuration: command=DemoteVoter server-id=server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 server-addr= servers="[{Suffrage:Nonvoter ID:server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 Address:b374f233-db7b-bf1a-4a12-ed6c4b52db93} {Suffrage:Voter ID:server-636418b4-0b2f-925b-a178-214aab4f6ac9 Address:636418b4-0b2f-925b-a178-214aab4f6ac9}]"
2025-10-30T18:19:21.668Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:21.774Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:21.668494793 +0000 UTC m=+107.947168690 (1 nodes), 404ns from start of poll, 274.72663ms from cluster start. Timeout at 2025-10-30 18:19:21.773533911 +0000 UTC m=+108.052207807, 105.039117ms after stability
2025-10-30T18:19:21.774Z [INFO]  server-636418b4-0b2f-925b-a178-214aab4f6ac9: aborting pipeline replication: peer="{Voter server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 b374f233-db7b-bf1a-4a12-ed6c4b52db93}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.38s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-10-30T18:19:21.774Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6634f9-477d-72bc-ce02-f90dd8e69273 Address:9f6634f9-477d-72bc-ce02-f90dd8e69273} {Suffrage:Voter ID:server-28f891b1-7ab4-4221-09da-1925aed3535b Address:28f891b1-7ab4-4221-09da-1925aed3535b}]"
2025-10-30T18:19:21.774Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: entering follower state: follower="Node at 9f6634f9-477d-72bc-ce02-f90dd8e69273 [Follower]" leader-address= leader-id=
2025-10-30T18:19:21.774Z [INFO]  server-28f891b1-7ab4-4221-09da-1925aed3535b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6634f9-477d-72bc-ce02-f90dd8e69273 Address:9f6634f9-477d-72bc-ce02-f90dd8e69273} {Suffrage:Voter ID:server-28f891b1-7ab4-4221-09da-1925aed3535b Address:28f891b1-7ab4-4221-09da-1925aed3535b}]"
2025-10-30T18:19:21.774Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:21.774Z [INFO]  server-28f891b1-7ab4-4221-09da-1925aed3535b: entering follower state: follower="Node at 28f891b1-7ab4-4221-09da-1925aed3535b [Follower]" leader-address= leader-id=
2025-10-30T18:19:21.847Z [WARN]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:21.847Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: entering candidate state: node="Node at 9f6634f9-477d-72bc-ce02-f90dd8e69273 [Candidate]" term=2
2025-10-30T18:19:21.848Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: election won: term=2 tally=2
2025-10-30T18:19:21.848Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: entering leader state: leader="Node at 9f6634f9-477d-72bc-ce02-f90dd8e69273 [Leader]"
2025-10-30T18:19:21.848Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: added peer, starting replication: peer=server-28f891b1-7ab4-4221-09da-1925aed3535b
2025-10-30T18:19:21.848Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: pipelining replication: peer="{Voter server-28f891b1-7ab4-4221-09da-1925aed3535b 28f891b1-7ab4-4221-09da-1925aed3535b}"
2025-10-30T18:19:21.954Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:21.848111175 +0000 UTC m=+108.126785072 (1 nodes), 73.12359ms from start of poll, 73.277005ms from cluster start. Timeout at 2025-10-30 18:19:21.953116354 +0000 UTC m=+108.231790250, 105.005178ms after stability
2025-10-30T18:19:21.954Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:22.059Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:21.954226448 +0000 UTC m=+108.232900345 (1 nodes), 1.333µs from start of poll, 179.392278ms from cluster start. Timeout at 2025-10-30 18:19:22.059242874 +0000 UTC m=+108.337916785, 105.01644ms after stability
2025-10-30T18:19:22.070Z [INFO]  server-28f891b1-7ab4-4221-09da-1925aed3535b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6634f9-477d-72bc-ce02-f90dd8e69273 Address:9f6634f9-477d-72bc-ce02-f90dd8e69273} {Suffrage:Voter ID:server-28f891b1-7ab4-4221-09da-1925aed3535b Address:28f891b1-7ab4-4221-09da-1925aed3535b}]"
2025-10-30T18:19:22.070Z [INFO]  server-9f6634f9-477d-72bc-ce02-f90dd8e69273: aborting pipeline replication: peer="{Voter server-28f891b1-7ab4-4221-09da-1925aed3535b 28f891b1-7ab4-4221-09da-1925aed3535b}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-10-30T18:19:22.071Z [INFO]  server-5b298b0f-338b-a374-1fe3-5be47e63d82d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b298b0f-338b-a374-1fe3-5be47e63d82d Address:5b298b0f-338b-a374-1fe3-5be47e63d82d}]"
2025-10-30T18:19:22.071Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:22.071Z [INFO]  server-5b298b0f-338b-a374-1fe3-5be47e63d82d: entering follower state: follower="Node at 5b298b0f-338b-a374-1fe3-5be47e63d82d [Follower]" leader-address= leader-id=
2025-10-30T18:19:22.133Z [WARN]  server-5b298b0f-338b-a374-1fe3-5be47e63d82d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:22.133Z [INFO]  server-5b298b0f-338b-a374-1fe3-5be47e63d82d: entering candidate state: node="Node at 5b298b0f-338b-a374-1fe3-5be47e63d82d [Candidate]" term=2
2025-10-30T18:19:22.133Z [INFO]  server-5b298b0f-338b-a374-1fe3-5be47e63d82d: election won: term=2 tally=1
2025-10-30T18:19:22.133Z [INFO]  server-5b298b0f-338b-a374-1fe3-5be47e63d82d: entering leader state: leader="Node at 5b298b0f-338b-a374-1fe3-5be47e63d82d [Leader]"
2025-10-30T18:19:22.239Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:22.133375059 +0000 UTC m=+108.412048957 (1 nodes), 61.863089ms from start of poll, 61.948485ms from cluster start. Timeout at 2025-10-30 18:19:22.238375611 +0000 UTC m=+108.517049511, 105.000554ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.17s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-10-30T18:19:22.240Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1431383-6e28-4ce3-d3f4-c0afd09d061d Address:e1431383-6e28-4ce3-d3f4-c0afd09d061d} {Suffrage:Voter ID:server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 Address:5fad65b1-0e46-68b6-1f9c-ad508f5f4988}]"
2025-10-30T18:19:22.240Z [INFO]  server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1431383-6e28-4ce3-d3f4-c0afd09d061d Address:e1431383-6e28-4ce3-d3f4-c0afd09d061d} {Suffrage:Voter ID:server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 Address:5fad65b1-0e46-68b6-1f9c-ad508f5f4988}]"
2025-10-30T18:19:22.240Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:22.240Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: entering follower state: follower="Node at e1431383-6e28-4ce3-d3f4-c0afd09d061d [Follower]" leader-address= leader-id=
2025-10-30T18:19:22.240Z [INFO]  server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: entering follower state: follower="Node at 5fad65b1-0e46-68b6-1f9c-ad508f5f4988 [Follower]" leader-address= leader-id=
2025-10-30T18:19:22.317Z [WARN]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:22.317Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: entering candidate state: node="Node at e1431383-6e28-4ce3-d3f4-c0afd09d061d [Candidate]" term=2
2025-10-30T18:19:22.317Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: election won: term=2 tally=2
2025-10-30T18:19:22.317Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: entering leader state: leader="Node at e1431383-6e28-4ce3-d3f4-c0afd09d061d [Leader]"
2025-10-30T18:19:22.317Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: added peer, starting replication: peer=server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988
2025-10-30T18:19:22.317Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: pipelining replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}"
2025-10-30T18:19:22.422Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:22.317631434 +0000 UTC m=+108.596305331 (1 nodes), 77.378111ms from start of poll, 77.546331ms from cluster start. Timeout at 2025-10-30 18:19:22.422634678 +0000 UTC m=+108.701308579, 105.003248ms after stability
2025-10-30T18:19:22.422Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:22.528Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:22.422721257 +0000 UTC m=+108.701395159 (1 nodes), 745ns from start of poll, 182.636159ms from cluster start. Timeout at 2025-10-30 18:19:22.527741092 +0000 UTC m=+108.806414999, 105.01984ms after stability
2025-10-30T18:19:22.530Z [ERROR] server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-10-30T18:19:22.530Z [WARN]  server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-10-30T18:19:22.530Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: aborting pipeline replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}"
2025-10-30T18:19:22.539Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: pipelining replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}"
2025-10-30T18:19:22.539Z [INFO]  server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1431383-6e28-4ce3-d3f4-c0afd09d061d Address:e1431383-6e28-4ce3-d3f4-c0afd09d061d} {Suffrage:Voter ID:server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 Address:5fad65b1-0e46-68b6-1f9c-ad508f5f4988}]"
2025-10-30T18:19:22.540Z [INFO]  server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: aborting pipeline replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}"
--- PASS: TestRaft_CacheLogWithStoreError (0.30s)
=== RUN   TestRaft_ReloadConfig
2025-10-30T18:19:22.540Z [INFO]  server-3d212418-56d8-cddd-a2e6-5b0e7ef82d9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d212418-56d8-cddd-a2e6-5b0e7ef82d9b Address:3d212418-56d8-cddd-a2e6-5b0e7ef82d9b}]"
2025-10-30T18:19:22.540Z [INFO]  server-3d212418-56d8-cddd-a2e6-5b0e7ef82d9b: entering follower state: follower="Node at 3d212418-56d8-cddd-a2e6-5b0e7ef82d9b [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-10-30T18:19:22.540Z [INFO]  server-c060c281-996a-b3bf-90b3-e0998937da23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c060c281-996a-b3bf-90b3-e0998937da23 Address:c060c281-996a-b3bf-90b3-e0998937da23}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-10-30T18:19:22.541Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-10-30T18:19:22.541Z [INFO]  server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 Address:405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3} {Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]"
2025-10-30T18:19:22.541Z [INFO]  server-677caaff-e2d1-f687-42eb-19d578168f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 Address:405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3} {Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]"
2025-10-30T18:19:22.541Z [INFO]  server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3: entering follower state: follower="Node at 405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 [Follower]" leader-address= leader-id=
2025-10-30T18:19:22.541Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 Address:405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3} {Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]"
2025-10-30T18:19:22.541Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:22.541Z [INFO]  server-677caaff-e2d1-f687-42eb-19d578168f63: entering follower state: follower="Node at 677caaff-e2d1-f687-42eb-19d578168f63 [Follower]" leader-address= leader-id=
2025-10-30T18:19:22.541Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering follower state: follower="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Follower]" leader-address= leader-id=
2025-10-30T18:19:22.599Z [WARN]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:22.599Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering candidate state: node="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Candidate]" term=2
2025-10-30T18:19:22.599Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: election won: term=2 tally=2
2025-10-30T18:19:22.599Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering leader state: leader="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Leader]"
2025-10-30T18:19:22.599Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: added peer, starting replication: peer=server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3
2025-10-30T18:19:22.599Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: added peer, starting replication: peer=server-677caaff-e2d1-f687-42eb-19d578168f63
2025-10-30T18:19:22.599Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: pipelining replication: peer="{Voter server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3}"
2025-10-30T18:19:22.599Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: pipelining replication: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}"
2025-10-30T18:19:22.705Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:22.599502723 +0000 UTC m=+108.878176621 (1 nodes), 57.830932ms from start of poll, 57.982788ms from cluster start. Timeout at 2025-10-30 18:19:22.704505813 +0000 UTC m=+108.983179711, 105.00309ms after stability
2025-10-30T18:19:22.710Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:22.816Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:22.710758083 +0000 UTC m=+108.989431986 (2 nodes), 938ns from start of poll, 169.238153ms from cluster start. Timeout at 2025-10-30 18:19:22.815772816 +0000 UTC m=+109.094446732, 105.014746ms after stability
2025-10-30T18:19:22.816Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: updating configuration: command=RemoveServer server-id=server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 server-addr= servers="[{Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]"
2025-10-30T18:19:22.816Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: removed peer, stopping replication: peer=server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 last-index=3
2025-10-30T18:19:22.816Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: aborting pipeline replication: peer="{Voter server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3}"
2025-10-30T18:19:22.868Z [WARN]  server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-10-30T18:19:22.926Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to heartbeat to: peer=677caaff-e2d1-f687-42eb-19d578168f63 backoff time=10ms error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63"
2025-10-30T18:19:22.928Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to pipeline appendEntries: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="append pipeline closed"
2025-10-30T18:19:22.928Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: aborting pipeline replication: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}"
2025-10-30T18:19:22.936Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to appendEntries to: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63"
2025-10-30T18:19:22.946Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to heartbeat to: peer=677caaff-e2d1-f687-42eb-19d578168f63 backoff time=10ms error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63"
2025-10-30T18:19:22.957Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to appendEntries to: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63"
2025-10-30T18:19:22.966Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to heartbeat to: peer=677caaff-e2d1-f687-42eb-19d578168f63 backoff time=10ms error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63"
2025-10-30T18:19:22.970Z [WARN]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to contact: server-id=server-677caaff-e2d1-f687-42eb-19d578168f63 time=50.027089ms
2025-10-30T18:19:22.970Z [WARN]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to contact quorum of nodes, stepping down
2025-10-30T18:19:22.970Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering follower state: follower="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Follower]" leader-address= leader-id=
2025-10-30T18:19:22.974Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to appendEntries to: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63"
2025-10-30T18:19:23.045Z [WARN]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:23.046Z [INFO]  server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering candidate state: node="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Candidate]" term=3
2025-10-30T18:19:23.046Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to make requestVote RPC: target="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" term=3
2025-10-30T18:19:23.050Z [WARN]  server-677caaff-e2d1-f687-42eb-19d578168f63: heartbeat timeout reached, starting election: last-leader-addr=0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 last-leader-id=server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83
2025-10-30T18:19:23.050Z [INFO]  server-677caaff-e2d1-f687-42eb-19d578168f63: entering candidate state: node="Node at 677caaff-e2d1-f687-42eb-19d578168f63 [Candidate]" term=3
2025-10-30T18:19:23.050Z [WARN]  server-677caaff-e2d1-f687-42eb-19d578168f63: rejecting vote request since node is not in configuration: from=405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3
2025-10-30T18:19:23.050Z [ERROR] server-677caaff-e2d1-f687-42eb-19d578168f63: failed to make requestVote RPC: target="{Voter server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}" error="failed to connect to peer: 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83" term=3
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.51s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-10-30T18:19:23.051Z [INFO]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e} {Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]"
2025-10-30T18:19:23.051Z [INFO]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering follower state: follower="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Follower]" leader-address= leader-id=
2025-10-30T18:19:23.051Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e} {Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]"
2025-10-30T18:19:23.051Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering follower state: follower="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Follower]" leader-address= leader-id=
2025-10-30T18:19:23.051Z [INFO]  server-efa5b2fa-54b3-cb32-18da-8843719903cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e} {Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]"
2025-10-30T18:19:23.051Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:23.051Z [INFO]  server-efa5b2fa-54b3-cb32-18da-8843719903cd: entering follower state: follower="Node at efa5b2fa-54b3-cb32-18da-8843719903cd [Follower]" leader-address= leader-id=
2025-10-30T18:19:23.114Z [WARN]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:23.114Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering candidate state: node="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Candidate]" term=2
2025-10-30T18:19:23.114Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: election won: term=2 tally=2
2025-10-30T18:19:23.114Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering leader state: leader="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Leader]"
2025-10-30T18:19:23.114Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e
2025-10-30T18:19:23.114Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-efa5b2fa-54b3-cb32-18da-8843719903cd
2025-10-30T18:19:23.114Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}"
2025-10-30T18:19:23.114Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:23.219Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:23.114230494 +0000 UTC m=+109.392904391 (1 nodes), 62.722226ms from start of poll, 62.936244ms from cluster start. Timeout at 2025-10-30 18:19:23.219232755 +0000 UTC m=+109.497906652, 105.002261ms after stability
2025-10-30T18:19:23.225Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:23.331Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:23.225239878 +0000 UTC m=+109.503913775 (2 nodes), 1.549µs from start of poll, 173.945628ms from cluster start. Timeout at 2025-10-30 18:19:23.33025828 +0000 UTC m=+109.608932188, 105.018413ms after stability
2025-10-30T18:19:23.333Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=10ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.338Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to pipeline appendEntries: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="append pipeline closed"
2025-10-30T18:19:23.338Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:23.346Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.352Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=10ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.363Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.371Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=10ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.379Z [WARN]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to contact: server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e time=50.621426ms
2025-10-30T18:19:23.380Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.389Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=20ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.402Z [WARN]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: heartbeat timeout reached, starting election: last-leader-addr=c88d47c5-be75-34fa-2df0-109d3b6ae46a last-leader-id=server-c88d47c5-be75-34fa-2df0-109d3b6ae46a
2025-10-30T18:19:23.402Z [INFO]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering candidate state: node="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Candidate]" term=3
2025-10-30T18:19:23.402Z [ERROR] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to make requestVote RPC: target="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}" error="failed to connect to peer: efa5b2fa-54b3-cb32-18da-8843719903cd" term=3
2025-10-30T18:19:23.402Z [ERROR] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to make requestVote RPC: target="{Voter server-c88d47c5-be75-34fa-2df0-109d3b6ae46a c88d47c5-be75-34fa-2df0-109d3b6ae46a}" error="failed to connect to peer: c88d47c5-be75-34fa-2df0-109d3b6ae46a" term=3
2025-10-30T18:19:23.409Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.421Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=25ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.428Z [WARN]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to contact: server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e time=99.200295ms
2025-10-30T18:19:23.436Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=RemoveServer server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr= servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]"
2025-10-30T18:19:23.436Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: removed peer, stopping replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e last-index=3
2025-10-30T18:19:23.436Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=AddNonvoter server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd} {Suffrage:Nonvoter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}]"
2025-10-30T18:19:23.436Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e
2025-10-30T18:19:23.436Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e"
2025-10-30T18:19:23.453Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: peer has newer term, stopping replication: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:23.453Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering follower state: follower="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Follower]" leader-address= leader-id=
2025-10-30T18:19:23.453Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}"
2025-10-30T18:19:23.455Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: peer has newer term, stopping replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:23.461Z [WARN]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: Election timeout reached, restarting election
2025-10-30T18:19:23.461Z [INFO]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering candidate state: node="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Candidate]" term=23
2025-10-30T18:19:23.461Z [WARN]  server-efa5b2fa-54b3-cb32-18da-8843719903cd: rejecting vote request since we have a leader: from=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e leader=c88d47c5-be75-34fa-2df0-109d3b6ae46a leader-id=server-c88d47c5-be75-34fa-2df0-109d3b6ae46a
2025-10-30T18:19:23.461Z [WARN]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: rejecting vote request since node is not a voter: from=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e
2025-10-30T18:19:23.523Z [WARN]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:23.523Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering candidate state: node="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Candidate]" term=24
2025-10-30T18:19:23.523Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: election won: term=24 tally=2
2025-10-30T18:19:23.523Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering leader state: leader="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Leader]"
2025-10-30T18:19:23.523Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-efa5b2fa-54b3-cb32-18da-8843719903cd
2025-10-30T18:19:23.523Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e
2025-10-30T18:19:23.523Z [WARN]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-10-30T18:19:23.523Z [INFO]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering follower state: follower="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Follower]" leader-address=c88d47c5-be75-34fa-2df0-109d3b6ae46a leader-id=server-c88d47c5-be75-34fa-2df0-109d3b6ae46a
2025-10-30T18:19:23.523Z [WARN]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: appendEntries rejected, sending older logs: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" next=3
2025-10-30T18:19:23.524Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:23.524Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}"
2025-10-30T18:19:23.542Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:23.647Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:23.542460799 +0000 UTC m=+109.821134696 (1 nodes), 22.114µs from start of poll, 491.166549ms from cluster start. Timeout at 2025-10-30 18:19:23.647480981 +0000 UTC m=+109.926154878, 105.020182ms after stability
2025-10-30T18:19:23.666Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=RemoveServer server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr= servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]"
2025-10-30T18:19:23.666Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: removed peer, stopping replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e last-index=106
2025-10-30T18:19:23.666Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:23.666Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=AddVoter server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd} {Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}]"
2025-10-30T18:19:23.666Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e
2025-10-30T18:19:23.666Z [WARN]  server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2025-10-30T18:19:23.667Z [WARN]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: appendEntries rejected, sending older logs: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" next=107
2025-10-30T18:19:23.667Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:24.733Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:24.839Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:24.733793999 +0000 UTC m=+111.012467899 (1 nodes), 526ns from start of poll, 1.682499752s from cluster start. Timeout at 2025-10-30 18:19:24.838815442 +0000 UTC m=+111.117489339, 105.02144ms after stability
2025-10-30T18:19:24.839Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}"
2025-10-30T18:19:24.839Z [INFO]  server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.79s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-10-30T18:19:24.840Z [INFO]  server-ff614bb3-8186-fbc0-2495-2700e40934be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]"
2025-10-30T18:19:24.840Z [INFO]  server-ff614bb3-8186-fbc0-2495-2700e40934be: entering follower state: follower="Node at ff614bb3-8186-fbc0-2495-2700e40934be [Follower]" leader-address= leader-id=
2025-10-30T18:19:24.840Z [INFO]  server-5d90c322-23c1-7b8b-1254-88cc33c90377: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]"
2025-10-30T18:19:24.840Z [INFO]  server-5d90c322-23c1-7b8b-1254-88cc33c90377: entering follower state: follower="Node at 5d90c322-23c1-7b8b-1254-88cc33c90377 [Follower]" leader-address= leader-id=
2025-10-30T18:19:24.840Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]"
2025-10-30T18:19:24.840Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:24.840Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: entering follower state: follower="Node at f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 [Follower]" leader-address= leader-id=
2025-10-30T18:19:24.960Z [WARN]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:24.960Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: entering candidate state: node="Node at f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 [Candidate]" term=2
2025-10-30T18:19:24.960Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: election won: term=2 tally=2
2025-10-30T18:19:24.960Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: entering leader state: leader="Node at f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 [Leader]"
2025-10-30T18:19:24.960Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: added peer, starting replication: peer=server-ff614bb3-8186-fbc0-2495-2700e40934be
2025-10-30T18:19:24.960Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: added peer, starting replication: peer=server-5d90c322-23c1-7b8b-1254-88cc33c90377
2025-10-30T18:19:24.960Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: pipelining replication: peer="{Voter server-5d90c322-23c1-7b8b-1254-88cc33c90377 5d90c322-23c1-7b8b-1254-88cc33c90377}"
2025-10-30T18:19:24.960Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: pipelining replication: peer="{Voter server-ff614bb3-8186-fbc0-2495-2700e40934be ff614bb3-8186-fbc0-2495-2700e40934be}"
2025-10-30T18:19:25.166Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:24.960544116 +0000 UTC m=+111.239218023 (1 nodes), 120.095601ms from start of poll, 120.399813ms from cluster start. Timeout at 2025-10-30 18:19:25.165548143 +0000 UTC m=+111.444222039, 205.004016ms after stability
2025-10-30T18:19:25.166Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:25.371Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:25.166381543 +0000 UTC m=+111.445055454 (1 nodes), 1.429µs from start of poll, 326.237244ms from cluster start. Timeout at 2025-10-30 18:19:25.371398265 +0000 UTC m=+111.650072195, 205.016741ms after stability
2025-10-30T18:19:25.377Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:25.583Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:25.37754477 +0000 UTC m=+111.656218672 (2 nodes), 1.425µs from start of poll, 537.400462ms from cluster start. Timeout at 2025-10-30 18:19:25.582566071 +0000 UTC m=+111.861240009, 205.021337ms after stability
2025-10-30T18:19:25.583Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:25.789Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:25.58351426 +0000 UTC m=+111.862188158 (1 nodes), 1.327µs from start of poll, 743.369948ms from cluster start. Timeout at 2025-10-30 18:19:25.788536164 +0000 UTC m=+112.067210085, 205.021927ms after stability
    raft_test.go:2961: [INFO] restarting Node at ff614bb3-8186-fbc0-2495-2700e40934be [Follower]
2025-10-30T18:19:25.789Z [INFO]  server-ff614bb3-8186-fbc0-2495-2700e40934be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]"
2025-10-30T18:19:25.789Z [INFO]  server-ff614bb3-8186-fbc0-2495-2700e40934be: entering follower state: follower="Node at ff614bb3-8186-fbc0-2495-2700e40934be [Follower]" leader-address= leader-id=
2025-10-30T18:19:26.040Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: aborting pipeline replication: peer="{Voter server-5d90c322-23c1-7b8b-1254-88cc33c90377 5d90c322-23c1-7b8b-1254-88cc33c90377}"
2025-10-30T18:19:26.424Z [ERROR] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: failed to pipeline appendEntries: peer="{Voter server-ff614bb3-8186-fbc0-2495-2700e40934be ff614bb3-8186-fbc0-2495-2700e40934be}" error="command enqueue timeout"
2025-10-30T18:19:26.424Z [INFO]  server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: aborting pipeline replication: peer="{Voter server-ff614bb3-8186-fbc0-2495-2700e40934be ff614bb3-8186-fbc0-2495-2700e40934be}"
2025-10-30T18:19:26.553Z [ERROR] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: failed to heartbeat to: peer=ff614bb3-8186-fbc0-2495-2700e40934be backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.71s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-10-30T18:19:26.554Z [INFO]  server-8c96d448-e412-8301-3f0b-8caa6c1a4952: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c96d448-e412-8301-3f0b-8caa6c1a4952 Address:8c96d448-e412-8301-3f0b-8caa6c1a4952} {Suffrage:Voter ID:server-19c612f6-002f-1c88-e89a-66f8cdcca8ac Address:19c612f6-002f-1c88-e89a-66f8cdcca8ac} {Suffrage:Voter ID:server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c Address:103ff97f-365d-3ae8-7ea4-9a35ca72b24c}]"
2025-10-30T18:19:26.554Z [INFO]  server-8c96d448-e412-8301-3f0b-8caa6c1a4952: entering follower state: follower="Node at 8c96d448-e412-8301-3f0b-8caa6c1a4952 [Follower]" leader-address= leader-id=
2025-10-30T18:19:26.554Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c96d448-e412-8301-3f0b-8caa6c1a4952 Address:8c96d448-e412-8301-3f0b-8caa6c1a4952} {Suffrage:Voter ID:server-19c612f6-002f-1c88-e89a-66f8cdcca8ac Address:19c612f6-002f-1c88-e89a-66f8cdcca8ac} {Suffrage:Voter ID:server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c Address:103ff97f-365d-3ae8-7ea4-9a35ca72b24c}]"
2025-10-30T18:19:26.554Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering follower state: follower="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Follower]" leader-address= leader-id=
2025-10-30T18:19:26.554Z [INFO]  server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c96d448-e412-8301-3f0b-8caa6c1a4952 Address:8c96d448-e412-8301-3f0b-8caa6c1a4952} {Suffrage:Voter ID:server-19c612f6-002f-1c88-e89a-66f8cdcca8ac Address:19c612f6-002f-1c88-e89a-66f8cdcca8ac} {Suffrage:Voter ID:server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c Address:103ff97f-365d-3ae8-7ea4-9a35ca72b24c}]"
2025-10-30T18:19:26.554Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:26.554Z [INFO]  server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: entering follower state: follower="Node at 103ff97f-365d-3ae8-7ea4-9a35ca72b24c [Follower]" leader-address= leader-id=
2025-10-30T18:19:26.624Z [WARN]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:26.624Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering candidate state: node="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Candidate]" term=2
2025-10-30T18:19:26.624Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: election won: term=2 tally=2
2025-10-30T18:19:26.624Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering leader state: leader="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Leader]"
2025-10-30T18:19:26.624Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: added peer, starting replication: peer=server-8c96d448-e412-8301-3f0b-8caa6c1a4952
2025-10-30T18:19:26.624Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: added peer, starting replication: peer=server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c
2025-10-30T18:19:26.624Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: pipelining replication: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}"
2025-10-30T18:19:26.624Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: pipelining replication: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}"
2025-10-30T18:19:26.730Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:26.624394584 +0000 UTC m=+112.903068481 (1 nodes), 69.458553ms from start of poll, 69.729378ms from cluster start. Timeout at 2025-10-30 18:19:26.729395151 +0000 UTC m=+113.008069048, 105.000567ms after stability
2025-10-30T18:19:26.730Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-10-30T18:19:26.835Z [INFO]  cluster: stable state for Leader reached at 2025-10-30 18:19:26.730305679 +0000 UTC m=+113.008979576 (1 nodes), 774ns from start of poll, 175.640473ms from cluster start. Timeout at 2025-10-30 18:19:26.835319913 +0000 UTC m=+113.113993832, 105.014256ms after stability
2025-10-30T18:19:26.841Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-10-30T18:19:26.946Z [INFO]  cluster: stable state for Follower reached at 2025-10-30 18:19:26.841559278 +0000 UTC m=+113.120233176 (2 nodes), 2.104µs from start of poll, 286.894073ms from cluster start. Timeout at 2025-10-30 18:19:26.94656978 +0000 UTC m=+113.225243677, 105.010501ms after stability
2025-10-30T18:19:26.952Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=10ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:26.952Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to pipeline appendEntries: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="append pipeline closed"
2025-10-30T18:19:26.952Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: aborting pipeline replication: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}"
2025-10-30T18:19:26.953Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=103ff97f-365d-3ae8-7ea4-9a35ca72b24c backoff time=10ms error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c"
2025-10-30T18:19:26.954Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to pipeline appendEntries: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="append pipeline closed"
2025-10-30T18:19:26.954Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: aborting pipeline replication: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}"
2025-10-30T18:19:26.961Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:26.962Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c"
2025-10-30T18:19:26.972Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=10ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:26.973Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=103ff97f-365d-3ae8-7ea4-9a35ca72b24c backoff time=10ms error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c"
2025-10-30T18:19:26.977Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:26.982Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c"
2025-10-30T18:19:26.988Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=10ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:26.990Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=103ff97f-365d-3ae8-7ea4-9a35ca72b24c backoff time=10ms error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c"
2025-10-30T18:19:26.995Z [WARN]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact: server-id=server-8c96d448-e412-8301-3f0b-8caa6c1a4952 time=50.859628ms
2025-10-30T18:19:26.995Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:27.002Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c"
2025-10-30T18:19:27.005Z [WARN]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact: server-id=server-8c96d448-e412-8301-3f0b-8caa6c1a4952 time=61.170251ms
2025-10-30T18:19:27.005Z [WARN]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact: server-id=server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c time=58.737576ms
2025-10-30T18:19:27.005Z [WARN]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact quorum of nodes, stepping down
2025-10-30T18:19:27.005Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering follower state: follower="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Follower]" leader-address= leader-id=
2025-10-30T18:19:27.005Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=20ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:27.024Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952"
2025-10-30T18:19:27.046Z [WARN]  server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: heartbeat timeout reached, starting election: last-leader-addr=19c612f6-002f-1c88-e89a-66f8cdcca8ac last-leader-id=server-19c612f6-002f-1c88-e89a-66f8cdcca8ac
2025-10-30T18:19:27.046Z [INFO]  server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: entering candidate state: node="Node at 103ff97f-365d-3ae8-7ea4-9a35ca72b24c [Candidate]" term=3
2025-10-30T18:19:27.046Z [ERROR] server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: failed to make requestVote RPC: target="{Voter server-19c612f6-002f-1c88-e89a-66f8cdcca8ac 19c612f6-002f-1c88-e89a-66f8cdcca8ac}" error="failed to connect to peer: 19c612f6-002f-1c88-e89a-66f8cdcca8ac" term=3
2025-10-30T18:19:27.046Z [WARN]  server-8c96d448-e412-8301-3f0b-8caa6c1a4952: rejecting vote request since we have a leader: from=103ff97f-365d-3ae8-7ea4-9a35ca72b24c leader=19c612f6-002f-1c88-e89a-66f8cdcca8ac leader-id=server-19c612f6-002f-1c88-e89a-66f8cdcca8ac
2025-10-30T18:19:27.071Z [WARN]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-10-30T18:19:27.071Z [INFO]  server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering candidate state: node="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Candidate]" term=3
2025-10-30T18:19:27.071Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to make requestVote RPC: target="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" term=3
2025-10-30T18:19:27.071Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to make requestVote RPC: target="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" term=3
2025-10-30T18:19:27.088Z [WARN]  server-8c96d448-e412-8301-3f0b-8caa6c1a4952: heartbeat timeout reached, starting election: last-leader-addr=19c612f6-002f-1c88-e89a-66f8cdcca8ac last-leader-id=server-19c612f6-002f-1c88-e89a-66f8cdcca8ac
2025-10-30T18:19:27.088Z [INFO]  server-8c96d448-e412-8301-3f0b-8caa6c1a4952: entering candidate state: node="Node at 8c96d448-e412-8301-3f0b-8caa6c1a4952 [Candidate]" term=3
2025-10-30T18:19:27.088Z [INFO]  server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: duplicate requestVote for same term: term=3
2025-10-30T18:19:27.088Z [ERROR] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: failed to make requestVote RPC: target="{Voter server-19c612f6-002f-1c88-e89a-66f8cdcca8ac 19c612f6-002f-1c88-e89a-66f8cdcca8ac}" error="failed to connect to peer: 19c612f6-002f-1c88-e89a-66f8cdcca8ac" term=3
2025-10-30T18:19:27.090Z [INFO]  server-8c96d448-e412-8301-3f0b-8caa6c1a4952: entering follower state: follower="Node at 8c96d448-e412-8301-3f0b-8caa6c1a4952 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.54s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-10-30T18:19:27.090Z [INFO]  127.0.0.1:38727 :: starting node: addr=127.0.0.1:38727
2025-10-30T18:19:27.090Z [INFO]  127.0.0.1:38727 :: initial configuration: index=0 servers=[]
2025-10-30T18:19:27.090Z [INFO]  127.0.0.1:38727 :: entering follower state: follower="Node at 127.0.0.1:38727 [Follower]" leader-address= leader-id=
2025-10-30T18:19:27.142Z [WARN]  127.0.0.1:38727 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-10-30T18:19:27.241Z [INFO]  127.0.0.1:37503 :: starting node: addr=127.0.0.1:37503
2025-10-30T18:19:27.241Z [INFO]  127.0.0.1:37503 :: initial configuration: index=0 servers=[]
2025-10-30T18:19:27.241Z [INFO]  127.0.0.1:37503 :: entering follower state: follower="Node at 127.0.0.1:37503 [Follower]" leader-address= leader-id=
2025-10-30T18:19:27.303Z [WARN]  127.0.0.1:37503 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-10-30T18:19:27.392Z [INFO]  127.0.0.1:40225 :: starting node: addr=127.0.0.1:40225
2025-10-30T18:19:27.392Z [INFO]  127.0.0.1:40225 :: initial configuration: index=0 servers=[]
2025-10-30T18:19:27.392Z [INFO]  127.0.0.1:40225 :: entering follower state: follower="Node at 127.0.0.1:40225 [Follower]" leader-address= leader-id=
2025-10-30T18:19:27.486Z [WARN]  127.0.0.1:40225 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-10-30T18:19:27.543Z [INFO]  127.0.0.1:38873 :: starting node: addr=127.0.0.1:38873
2025-10-30T18:19:27.543Z [INFO]  127.0.0.1:38873 :: initial configuration: index=0 servers=[]
2025-10-30T18:19:27.543Z [INFO]  127.0.0.1:38873 :: entering follower state: follower="Node at 127.0.0.1:38873 [Follower]" leader-address= leader-id=
2025-10-30T18:19:27.543Z [WARN]  127.0.0.1:38873 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	113.982s
?   	github.com/hashicorp/raft/bench	[no test files]
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/228153 and its subdirectories
I: Current time: Thu Oct 30 06:19:32 -12 2025
I: pbuilder-time-stamp: 1761848372
Thu Oct 30 18:19:33 UTC 2025  I: 1st build successful. Starting 2nd build on remote node ionos15-amd64.debian.net.
Thu Oct 30 18:19:33 UTC 2025  I: Preparing to do remote build '2' on ionos15-amd64.debian.net.
Thu Oct 30 18:23:33 UTC 2025  I: Deleting $TMPDIR on ionos15-amd64.debian.net.