I: pbuilder: network access will be disabled during build I: Current time: Mon Jul 28 12:54:22 -12 2025 I: pbuilder-time-stamp: 1753750462 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source gpgv: Signature made Thu Jul 27 15:15:46 2023 gpgv: using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29 gpgv: issuer "dswarbrick@debian.org" gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/2332697/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='arm64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 ' DISTRIBUTION='trixie' HOME='/root' HOST_ARCH='arm64' IFS=' ' INVOCATION_ID='7f3f05024de84a97b61ab841238d3780' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='2332697' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.uTgxbsNx/pbuilderrc_WUxO --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.uTgxbsNx/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='109' SUDO_UID='104' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://192.168.101.4:3128' I: uname -a Linux codethink03-arm64 6.1.0-21-cloud-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Jun 5 2024 /bin -> usr/bin I: user script /srv/workspace/pbuilder/2332697/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: arm64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19809 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.22-go{a} golang-1.22-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 102 newly installed, 0 to remove and 0 not upgraded. Need to get 87.0 MB of archives. After unpacking 627 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian trixie/main arm64 sensible-utils all 0.0.23 [24.7 kB] Get: 2 http://deb.debian.org/debian trixie/main arm64 libmagic-mgc arm64 1:5.45-3 [314 kB] Get: 3 http://deb.debian.org/debian trixie/main arm64 libmagic1t64 arm64 1:5.45-3 [100 kB] Get: 4 http://deb.debian.org/debian trixie/main arm64 file arm64 1:5.45-3 [43.0 kB] Get: 5 http://deb.debian.org/debian trixie/main arm64 gettext-base arm64 0.21-15 [159 kB] Get: 6 http://deb.debian.org/debian trixie/main arm64 libuchardet0 arm64 0.0.8-1+b1 [69.0 kB] Get: 7 http://deb.debian.org/debian trixie/main arm64 groff-base arm64 1.23.0-4 [1130 kB] Get: 8 http://deb.debian.org/debian trixie/main arm64 bsdextrautils arm64 2.40.1-8.1 [95.1 kB] Get: 9 http://deb.debian.org/debian trixie/main arm64 libpipeline1 arm64 1.5.7-2 [36.5 kB] Get: 10 http://deb.debian.org/debian trixie/main arm64 man-db arm64 2.12.1-2 [1394 kB] Get: 11 http://deb.debian.org/debian trixie/main arm64 m4 arm64 1.4.19-4 [277 kB] Get: 12 http://deb.debian.org/debian trixie/main arm64 autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian trixie/main arm64 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian trixie/main arm64 automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian trixie/main arm64 autopoint all 0.21-15 [683 kB] Get: 16 http://deb.debian.org/debian trixie/main arm64 libdebhelper-perl all 13.16 [88.6 kB] Get: 17 http://deb.debian.org/debian trixie/main arm64 libtool all 2.4.7-7 [517 kB] Get: 18 http://deb.debian.org/debian trixie/main arm64 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian trixie/main arm64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian trixie/main arm64 libfile-stripnondeterminism-perl all 1.14.0-1 [19.5 kB] Get: 21 http://deb.debian.org/debian trixie/main arm64 dh-strip-nondeterminism all 1.14.0-1 [8448 B] Get: 22 http://deb.debian.org/debian trixie/main arm64 libelf1t64 arm64 0.191-1+b1 [187 kB] Get: 23 http://deb.debian.org/debian trixie/main arm64 dwz arm64 0.15-1+b1 [102 kB] Get: 24 http://deb.debian.org/debian trixie/main arm64 libicu72 arm64 72.1-4+b1 [9224 kB] Get: 25 http://deb.debian.org/debian trixie/main arm64 libxml2 arm64 2.9.14+dfsg-1.3+b3 [624 kB] Get: 26 http://deb.debian.org/debian trixie/main arm64 gettext arm64 0.21-15 [1249 kB] Get: 27 http://deb.debian.org/debian trixie/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 28 http://deb.debian.org/debian trixie/main arm64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 29 http://deb.debian.org/debian trixie/main arm64 debhelper all 13.16 [891 kB] Get: 30 http://deb.debian.org/debian trixie/main arm64 dh-golang all 1.62 [24.7 kB] Get: 31 http://deb.debian.org/debian trixie/main arm64 golang-1.22-src all 1.22.4-1 [18.8 MB] Get: 32 http://deb.debian.org/debian trixie/main arm64 golang-1.22-go arm64 1.22.4-1 [21.3 MB] Get: 33 http://deb.debian.org/debian trixie/main arm64 golang-src all 2:1.22~3 [5020 B] Get: 34 http://deb.debian.org/debian trixie/main arm64 golang-go arm64 2:1.22~3 [44.2 kB] Get: 35 http://deb.debian.org/debian trixie/main arm64 golang-any arm64 2:1.22~3 [5104 B] Get: 36 http://deb.debian.org/debian trixie/main arm64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 37 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 38 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 39 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sys-dev all 0.20.0-1 [430 kB] Get: 40 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 41 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 42 http://deb.debian.org/debian trixie/main arm64 golang-github-fatih-color-dev all 1.16.0-1 [13.8 kB] Get: 43 http://deb.debian.org/debian trixie/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 44 http://deb.debian.org/debian trixie/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 45 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 46 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 47 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB] Get: 48 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 49 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 50 http://deb.debian.org/debian trixie/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 51 http://deb.debian.org/debian trixie/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 52 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 53 http://deb.debian.org/debian trixie/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 54 http://deb.debian.org/debian trixie/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 55 http://deb.debian.org/debian trixie/main arm64 golang-google-protobuf-dev all 1.33.0-1 [625 kB] Get: 56 http://deb.debian.org/debian trixie/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 57 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 58 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 60 http://deb.debian.org/debian trixie/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 61 http://deb.debian.org/debian trixie/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 62 http://deb.debian.org/debian trixie/main arm64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 63 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 65 http://deb.debian.org/debian trixie/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 66 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 67 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 68 http://deb.debian.org/debian trixie/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 69 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-term-dev all 0.20.0-1 [14.8 kB] Get: 70 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-text-dev all 0.15.0-1 [4006 kB] Get: 71 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-crypto-dev all 1:0.23.0-1 [1689 kB] Get: 72 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-net-dev all 1:0.25.0+dfsg-1 [895 kB] Get: 73 http://deb.debian.org/debian trixie/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 74 http://deb.debian.org/debian trixie/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 75 http://deb.debian.org/debian trixie/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 76 http://deb.debian.org/debian trixie/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 77 http://deb.debian.org/debian trixie/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 78 http://deb.debian.org/debian trixie/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 79 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB] Get: 80 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 81 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-common-dev all 0.53.0-1 [114 kB] Get: 82 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 83 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB] Get: 84 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-golang-dev all 1.19.0-1 [179 kB] Get: 85 http://deb.debian.org/debian trixie/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 86 http://deb.debian.org/debian trixie/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 87 http://deb.debian.org/debian trixie/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 88 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 89 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 90 http://deb.debian.org/debian trixie/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-6 [20.1 kB] Get: 91 http://deb.debian.org/debian trixie/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-6 [55.3 kB] Get: 92 http://deb.debian.org/debian trixie/main arm64 libssl-dev arm64 3.2.1-3 [3089 kB] Get: 93 http://deb.debian.org/debian trixie/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-6 [226 kB] Get: 94 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 95 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-mod-dev all 0.17.0-1 [103 kB] Get: 96 http://deb.debian.org/debian trixie/main arm64 golang-github-yuin-goldmark-dev all 1.7.1-1 [113 kB] Get: 97 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-tools-dev all 1:0.20.0+ds-1 [1571 kB] Get: 98 http://deb.debian.org/debian trixie/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 99 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 100 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 101 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 102 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 87.0 MB in 1s (167 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19809 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.23_all.deb ... Unpacking sensible-utils (0.0.23) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3_arm64.deb ... Unpacking libmagic-mgc (1:5.45-3) ... Selecting previously unselected package libmagic1t64:arm64. Preparing to unpack .../002-libmagic1t64_1%3a5.45-3_arm64.deb ... Unpacking libmagic1t64:arm64 (1:5.45-3) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.45-3_arm64.deb ... Unpacking file (1:5.45-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.21-15_arm64.deb ... Unpacking gettext-base (0.21-15) ... Selecting previously unselected package libuchardet0:arm64. Preparing to unpack .../005-libuchardet0_0.0.8-1+b1_arm64.deb ... Unpacking libuchardet0:arm64 (0.0.8-1+b1) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-4_arm64.deb ... Unpacking groff-base (1.23.0-4) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.40.1-8.1_arm64.deb ... Unpacking bsdextrautils (2.40.1-8.1) ... Selecting previously unselected package libpipeline1:arm64. Preparing to unpack .../008-libpipeline1_1.5.7-2_arm64.deb ... Unpacking libpipeline1:arm64 (1.5.7-2) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.12.1-2_arm64.deb ... Unpacking man-db (2.12.1-2) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-4_arm64.deb ... Unpacking m4 (1.4.19-4) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.21-15_all.deb ... Unpacking autopoint (0.21-15) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../015-libdebhelper-perl_13.16_all.deb ... Unpacking libdebhelper-perl (13.16) ... Selecting previously unselected package libtool. Preparing to unpack .../016-libtool_2.4.7-7_all.deb ... Unpacking libtool (2.4.7-7) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../017-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../020-dh-strip-nondeterminism_1.14.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.14.0-1) ... Selecting previously unselected package libelf1t64:arm64. Preparing to unpack .../021-libelf1t64_0.191-1+b1_arm64.deb ... Unpacking libelf1t64:arm64 (0.191-1+b1) ... Selecting previously unselected package dwz. Preparing to unpack .../022-dwz_0.15-1+b1_arm64.deb ... Unpacking dwz (0.15-1+b1) ... Selecting previously unselected package libicu72:arm64. Preparing to unpack .../023-libicu72_72.1-4+b1_arm64.deb ... Unpacking libicu72:arm64 (72.1-4+b1) ... Selecting previously unselected package libxml2:arm64. Preparing to unpack .../024-libxml2_2.9.14+dfsg-1.3+b3_arm64.deb ... Unpacking libxml2:arm64 (2.9.14+dfsg-1.3+b3) ... Selecting previously unselected package gettext. Preparing to unpack .../025-gettext_0.21-15_arm64.deb ... Unpacking gettext (0.21-15) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../028-debhelper_13.16_all.deb ... Unpacking debhelper (13.16) ... Selecting previously unselected package dh-golang. Preparing to unpack .../029-dh-golang_1.62_all.deb ... Unpacking dh-golang (1.62) ... Selecting previously unselected package golang-1.22-src. Preparing to unpack .../030-golang-1.22-src_1.22.4-1_all.deb ... Unpacking golang-1.22-src (1.22.4-1) ... Selecting previously unselected package golang-1.22-go. Preparing to unpack .../031-golang-1.22-go_1.22.4-1_arm64.deb ... Unpacking golang-1.22-go (1.22.4-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../032-golang-src_2%3a1.22~3_all.deb ... Unpacking golang-src (2:1.22~3) ... Selecting previously unselected package golang-go:arm64. Preparing to unpack .../033-golang-go_2%3a1.22~3_arm64.deb ... Unpacking golang-go:arm64 (2:1.22~3) ... Selecting previously unselected package golang-any:arm64. Preparing to unpack .../034-golang-any_2%3a1.22~3_arm64.deb ... Unpacking golang-any:arm64 (2:1.22~3) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../038-golang-golang-x-sys-dev_0.20.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.20.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../041-golang-github-fatih-color-dev_1.16.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.16.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../054-golang-google-protobuf-dev_1.33.0-1_all.deb ... Unpacking golang-google-protobuf-dev (1.33.0-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../065-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../066-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../067-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../068-golang-golang-x-term-dev_0.20.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.20.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../069-golang-golang-x-text-dev_0.15.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.15.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../070-golang-golang-x-crypto-dev_1%3a0.23.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.23.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../071-golang-golang-x-net-dev_1%3a0.25.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.25.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../072-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../073-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../074-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../075-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../076-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../078-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../079-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../080-golang-github-prometheus-common-dev_0.53.0-1_all.deb ... Unpacking golang-github-prometheus-common-dev (0.53.0-1) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../081-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../082-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../083-golang-github-prometheus-client-golang-dev_1.19.0-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.19.0-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../084-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../085-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../086-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../087-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../088-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:arm64. Preparing to unpack .../089-libsasl2-modules-db_2.1.28+dfsg1-6_arm64.deb ... Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-6) ... Selecting previously unselected package libsasl2-2:arm64. Preparing to unpack .../090-libsasl2-2_2.1.28+dfsg1-6_arm64.deb ... Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-6) ... Selecting previously unselected package libssl-dev:arm64. Preparing to unpack .../091-libssl-dev_3.2.1-3_arm64.deb ... Unpacking libssl-dev:arm64 (3.2.1-3) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../092-libsasl2-dev_2.1.28+dfsg1-6_arm64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-6) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../093-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../094-golang-golang-x-mod-dev_0.17.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.17.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../095-golang-github-yuin-goldmark-dev_1.7.1-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.1-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../096-golang-golang-x-tools-dev_1%3a0.20.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.20.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../097-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../098-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../099-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../100-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../101-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:arm64 (1.5.7-2) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up libicu72:arm64 (72.1-4+b1) ... Setting up bsdextrautils (2.40.1-8.1) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.45-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.16) ... Setting up golang-golang-x-sys-dev (0.20.0-1) ... Setting up libmagic1t64:arm64 (1:5.45-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-15) ... Setting up m4 (1.4.19-4) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-golang-x-term-dev (0.20.0-1) ... Setting up file (1:5.45-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up libelf1t64:arm64 (0.191-1+b1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-6) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.33.0-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.9.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.1-1) ... Setting up golang-1.22-src (1.22.4-1) ... Setting up libssl-dev:arm64 (3.2.1-3) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.17.0-1) ... Setting up autopoint (0.21-15) ... Setting up libsasl2-2:arm64 (2.1.28+dfsg1-6) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b1) ... Setting up sensible-utils (0.0.23) ... Setting up golang-golang-x-text-dev (0.15.0-1) ... Setting up libuchardet0:arm64 (0.0.8-1+b1) ... Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.23.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.15.0-1) ... Setting up golang-src (2:1.22~3) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:arm64 (2.9.14+dfsg-1.3+b3) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.0-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.21-15) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-7) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-1.22-go (1.22.4-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-6) ... Setting up golang-golang-x-net-dev (1:0.25.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.14.0-1) ... Setting up groff-base (1.23.0-4) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.16.0-1) ... Setting up golang-go:arm64 (2:1.22~3) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.20.0+ds-1) ... Setting up man-db (2.12.1-2) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-any:arm64 (2:1.22~3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.16) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.62) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.53.0-1) ... Setting up golang-github-prometheus-client-golang-dev (1.19.0-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.38-13) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture arm64 debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader internal/race internal/goarch internal/goos unicode runtime/internal/math internal/godebugs internal/coverage/rtcov internal/goexperiment internal/chacha8rand unicode/utf8 sync/atomic internal/abi runtime/internal/syscall runtime/internal/sys internal/cpu container/list internal/itoa math/bits crypto/internal/alias crypto/subtle crypto/internal/boring/sig cmp encoding unicode/utf16 log/internal slices math internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 runtime/internal/atomic internal/bytealg vendor/golang.org/x/crypto/internal/alias runtime internal/reflectlite sync internal/testlog internal/bisect internal/singleflight runtime/cgo errors sort internal/oserror internal/godebug io strconv vendor/golang.org/x/net/dns/dnsmessage internal/safefilepath github.com/hashicorp/golang-lru/simplelru syscall crypto/internal/nistec/fiat path math/rand internal/intern bytes strings hash crypto/internal/randutil internal/saferio hash/crc32 hash/crc64 crypto reflect net/netip vendor/golang.org/x/text/transform crypto/rc4 net/http/internal/ascii github.com/hashicorp/go-immutable-radix bufio regexp/syntax html regexp internal/syscall/execenv time internal/syscall/unix context io/fs internal/poll embed encoding/binary internal/fmtsort os encoding/base64 vendor/golang.org/x/crypto/internal/poly1305 crypto/cipher crypto/internal/edwards25519/field crypto/md5 crypto/internal/nistec golang.org/x/sys/unix encoding/pem crypto/internal/edwards25519 crypto/internal/boring crypto/des vendor/golang.org/x/crypto/chacha20 crypto/aes crypto/sha512 crypto/hmac crypto/sha1 crypto/sha256 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/crypto/chacha20poly1305 path/filepath fmt os/signal io/ioutil internal/sysinfo crypto/ecdh net net/url log encoding/hex math/big mime compress/flate encoding/json vendor/golang.org/x/net/http2/hpack mime/quotedprintable vendor/golang.org/x/text/unicode/norm net/http/internal encoding/gob vendor/golang.org/x/text/unicode/bidi go/token text/template/parse flag runtime/debug runtime/trace compress/gzip vendor/golang.org/x/text/secure/bidirule testing vendor/golang.org/x/net/idna text/template github.com/mattn/go-isatty github.com/mattn/go-colorable crypto/rand github.com/fatih/color crypto/elliptic crypto/internal/bigmod crypto/internal/boring/bbig encoding/asn1 crypto/dsa crypto/ed25519 github.com/hashicorp/go-hclog crypto/rsa vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix html/template crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http github.com/armon/go-metrics net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench ? github.com/hashicorp/raft/bench [no test files] === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2025-07-29T00:55:20.317Z [INFO] creating new snapshot: path=/tmp/raft2588977083/raft121062515/snapshots/3-10-1753750520317.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot 2025-07-29T00:55:20.317Z [INFO] creating new snapshot: path=/tmp/raft669465574/snapshots/3-10-1753750520317.tmp 2025-07-29T00:55:20.318Z [WARN] found temporary snapshot: name=3-10-1753750520317.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot 2025-07-29T00:55:20.318Z [INFO] creating new snapshot: path=/tmp/raft1397277739/snapshots/3-10-1753750520318.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention 2025-07-29T00:55:20.319Z [INFO] creating new snapshot: path=/tmp/raft1095325253/snapshots/3-10-1753750520319.tmp 2025-07-29T00:55:20.319Z [INFO] creating new snapshot: path=/tmp/raft1095325253/snapshots/3-11-1753750520319.tmp 2025-07-29T00:55:20.319Z [INFO] creating new snapshot: path=/tmp/raft1095325253/snapshots/3-12-1753750520319.tmp 2025-07-29T00:55:20.319Z [INFO] reaping snapshot: path=/tmp/raft1095325253/snapshots/3-10-1753750520319 2025-07-29T00:55:20.319Z [INFO] creating new snapshot: path=/tmp/raft1095325253/snapshots/3-13-1753750520319.tmp 2025-07-29T00:55:20.320Z [INFO] reaping snapshot: path=/tmp/raft1095325253/snapshots/3-11-1753750520319 2025-07-29T00:55:20.320Z [INFO] creating new snapshot: path=/tmp/raft1095325253/snapshots/3-14-1753750520320.tmp 2025-07-29T00:55:20.320Z [INFO] reaping snapshot: path=/tmp/raft1095325253/snapshots/3-12-1753750520319 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2025-07-29T00:55:20.321Z [INFO] creating new snapshot: path=/tmp/raft2595385178/snapshots/5-130350-1753750520321.tmp 2025-07-29T00:55:20.322Z [INFO] creating new snapshot: path=/tmp/raft2595385178/snapshots/36-204917-1753750520322.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 2025-07-29T00:55:20.424Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 2025-07-29T00:55:20.476Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) 2025-07-29T00:55:20.528Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2025-07-29T00:55:20.535Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:20.540Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:20.551Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:20.572Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:20.613Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:20.694Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:20.854Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:21.175Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:21.816Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:22.818Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-07-29T00:55:23.820Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2025-07-29T00:55:24.537Z [INFO] server-a5cd3d66-8187-e654-268a-019d60c564dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5cd3d66-8187-e654-268a-019d60c564dd Address:a5cd3d66-8187-e654-268a-019d60c564dd}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2025-07-29T00:55:24.538Z [INFO] server-9d1feb83-39d6-7bf5-4d23-4bdac4932cfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d1feb83-39d6-7bf5-4d23-4bdac4932cfa Address:9d1feb83-39d6-7bf5-4d23-4bdac4932cfa}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2025-07-29T00:55:24.538Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: initial configuration: index=0 servers=[] 2025-07-29T00:55:24.539Z [INFO] server-94b6d54f-d9bc-5b59-de87-c90469908d33: initial configuration: index=0 servers=[] 2025-07-29T00:55:24.539Z [INFO] server-189046f2-3b6c-ef25-74ff-9710324c0f10: initial configuration: index=0 servers=[] 2025-07-29T00:55:24.539Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: entering follower state: follower="Node at 1d7336e1-aa31-4eab-404f-4fce24a31d23 [Follower]" leader-address= leader-id= 2025-07-29T00:55:24.539Z [INFO] server-94b6d54f-d9bc-5b59-de87-c90469908d33: entering follower state: follower="Node at 94b6d54f-d9bc-5b59-de87-c90469908d33 [Follower]" leader-address= leader-id= 2025-07-29T00:55:24.539Z [INFO] server-189046f2-3b6c-ef25-74ff-9710324c0f10: entering follower state: follower="Node at 189046f2-3b6c-ef25-74ff-9710324c0f10 [Follower]" leader-address= leader-id= 2025-07-29T00:55:24.539Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:24.626Z [WARN] server-94b6d54f-d9bc-5b59-de87-c90469908d33: no known peers, aborting election 2025-07-29T00:55:24.630Z [WARN] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:24.630Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: entering candidate state: node="Node at 1d7336e1-aa31-4eab-404f-4fce24a31d23 [Candidate]" term=2 2025-07-29T00:55:24.630Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: election won: term=2 tally=2 2025-07-29T00:55:24.630Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: entering leader state: leader="Node at 1d7336e1-aa31-4eab-404f-4fce24a31d23 [Leader]" 2025-07-29T00:55:24.630Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: added peer, starting replication: peer=server-94b6d54f-d9bc-5b59-de87-c90469908d33 2025-07-29T00:55:24.630Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: added peer, starting replication: peer=server-189046f2-3b6c-ef25-74ff-9710324c0f10 2025-07-29T00:55:24.630Z [WARN] server-189046f2-3b6c-ef25-74ff-9710324c0f10: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2025-07-29T00:55:24.630Z [WARN] server-94b6d54f-d9bc-5b59-de87-c90469908d33: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2025-07-29T00:55:24.630Z [WARN] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: appendEntries rejected, sending older logs: peer="{Voter server-189046f2-3b6c-ef25-74ff-9710324c0f10 189046f2-3b6c-ef25-74ff-9710324c0f10}" next=1 2025-07-29T00:55:24.630Z [WARN] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: appendEntries rejected, sending older logs: peer="{Voter server-94b6d54f-d9bc-5b59-de87-c90469908d33 94b6d54f-d9bc-5b59-de87-c90469908d33}" next=1 2025-07-29T00:55:24.630Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: pipelining replication: peer="{Voter server-189046f2-3b6c-ef25-74ff-9710324c0f10 189046f2-3b6c-ef25-74ff-9710324c0f10}" 2025-07-29T00:55:24.630Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: pipelining replication: peer="{Voter server-94b6d54f-d9bc-5b59-de87-c90469908d33 94b6d54f-d9bc-5b59-de87-c90469908d33}" 2025-07-29T00:55:24.735Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:24.630282297 +0000 UTC m=+4.319663641 (2 nodes), 90.94476ms from start of poll, 91.41024ms from cluster start. Timeout at 2025-07-29 00:55:24.735546017 +0000 UTC m=+4.424927401, 105.26376ms after stability 2025-07-29T00:55:24.735Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:24.841Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:24.736009177 +0000 UTC m=+4.425390521 (1 nodes), 840ns from start of poll, 197.13712ms from cluster start. Timeout at 2025-07-29 00:55:24.841518897 +0000 UTC m=+4.530900281, 105.50976ms after stability 2025-07-29T00:55:24.852Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: aborting pipeline replication: peer="{Voter server-189046f2-3b6c-ef25-74ff-9710324c0f10 189046f2-3b6c-ef25-74ff-9710324c0f10}" 2025-07-29T00:55:24.852Z [INFO] server-1d7336e1-aa31-4eab-404f-4fce24a31d23: aborting pipeline replication: peer="{Voter server-94b6d54f-d9bc-5b59-de87-c90469908d33 94b6d54f-d9bc-5b59-de87-c90469908d33}" --- PASS: TestRaft_LiveBootstrap (0.31s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2025-07-29T00:55:24.853Z [INFO] server-9665e233-b05d-7788-b298-e4a26c5eedfa: initial configuration: index=0 servers=[] 2025-07-29T00:55:24.853Z [INFO] server-331a34bb-44f6-d8e5-fae9-2af754a1f5ab: initial configuration: index=0 servers=[] 2025-07-29T00:55:24.853Z [INFO] server-9665e233-b05d-7788-b298-e4a26c5eedfa: entering follower state: follower="Node at 9665e233-b05d-7788-b298-e4a26c5eedfa [Follower]" leader-address= leader-id= 2025-07-29T00:55:24.853Z [INFO] server-331a34bb-44f6-d8e5-fae9-2af754a1f5ab: entering follower state: follower="Node at 331a34bb-44f6-d8e5-fae9-2af754a1f5ab [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2025-07-29T00:55:24.854Z [INFO] server-033e2c74-5b95-e67b-368c-8fd8a9bd92cd: initial configuration: index=0 servers=[] 2025-07-29T00:55:24.854Z [INFO] server-033e2c74-5b95-e67b-368c-8fd8a9bd92cd: entering follower state: follower="Node at 033e2c74-5b95-e67b-368c-8fd8a9bd92cd [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2025-07-29T00:55:24.855Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a81043f-08e5-6c02-21f6-301da90f75f8 Address:6a81043f-08e5-6c02-21f6-301da90f75f8} {Suffrage:Voter ID:server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 Address:f0b5b545-0b45-19e0-5d63-cf054b8d8cc2} {Suffrage:Voter ID:server-38381713-7844-9f74-1130-aa008b160926 Address:38381713-7844-9f74-1130-aa008b160926}]" 2025-07-29T00:55:24.855Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: entering follower state: follower="Node at 6a81043f-08e5-6c02-21f6-301da90f75f8 [Follower]" leader-address= leader-id= 2025-07-29T00:55:24.855Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a81043f-08e5-6c02-21f6-301da90f75f8 Address:6a81043f-08e5-6c02-21f6-301da90f75f8} {Suffrage:Voter ID:server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 Address:f0b5b545-0b45-19e0-5d63-cf054b8d8cc2} {Suffrage:Voter ID:server-38381713-7844-9f74-1130-aa008b160926 Address:38381713-7844-9f74-1130-aa008b160926}]" 2025-07-29T00:55:24.855Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: entering follower state: follower="Node at f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 [Follower]" leader-address= leader-id= 2025-07-29T00:55:24.855Z [INFO] server-38381713-7844-9f74-1130-aa008b160926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a81043f-08e5-6c02-21f6-301da90f75f8 Address:6a81043f-08e5-6c02-21f6-301da90f75f8} {Suffrage:Voter ID:server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 Address:f0b5b545-0b45-19e0-5d63-cf054b8d8cc2} {Suffrage:Voter ID:server-38381713-7844-9f74-1130-aa008b160926 Address:38381713-7844-9f74-1130-aa008b160926}]" 2025-07-29T00:55:24.855Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:24.855Z [INFO] server-38381713-7844-9f74-1130-aa008b160926: entering follower state: follower="Node at 38381713-7844-9f74-1130-aa008b160926 [Follower]" leader-address= leader-id= 2025-07-29T00:55:24.931Z [WARN] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:24.931Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: entering candidate state: node="Node at f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 [Candidate]" term=2 2025-07-29T00:55:24.931Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: election won: term=2 tally=2 2025-07-29T00:55:24.931Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: entering leader state: leader="Node at f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 [Leader]" 2025-07-29T00:55:24.931Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: added peer, starting replication: peer=server-6a81043f-08e5-6c02-21f6-301da90f75f8 2025-07-29T00:55:24.931Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: added peer, starting replication: peer=server-38381713-7844-9f74-1130-aa008b160926 2025-07-29T00:55:24.931Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: pipelining replication: peer="{Voter server-38381713-7844-9f74-1130-aa008b160926 38381713-7844-9f74-1130-aa008b160926}" 2025-07-29T00:55:24.931Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: pipelining replication: peer="{Voter server-6a81043f-08e5-6c02-21f6-301da90f75f8 6a81043f-08e5-6c02-21f6-301da90f75f8}" 2025-07-29T00:55:25.037Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:24.931440617 +0000 UTC m=+4.620821961 (1 nodes), 75.90028ms from start of poll, 76.36248ms from cluster start. Timeout at 2025-07-29 00:55:25.037212377 +0000 UTC m=+4.726593721, 105.77176ms after stability 2025-07-29T00:55:25.037Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: aborting pipeline replication: peer="{Voter server-38381713-7844-9f74-1130-aa008b160926 38381713-7844-9f74-1130-aa008b160926}" 2025-07-29T00:55:25.037Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: aborting pipeline replication: peer="{Voter server-6a81043f-08e5-6c02-21f6-301da90f75f8 6a81043f-08e5-6c02-21f6-301da90f75f8}" 2025-07-29T00:55:25.037Z [INFO] creating new snapshot: path=/tmp/raft3798055033/snapshots/2-2-1753750525037.tmp 2025-07-29T00:55:25.038Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: starting restore from snapshot: id=2-2-1753750525037 last-index=2 last-term=2 size-in-bytes=1 2025-07-29T00:55:25.038Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: snapshot restore progress: id=2-2-1753750525037 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-07-29T00:55:25.038Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: restored from snapshot: id=2-2-1753750525037 last-index=2 last-term=2 size-in-bytes=1 2025-07-29T00:55:25.038Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a81043f-08e5-6c02-21f6-301da90f75f8 Address:6a81043f-08e5-6c02-21f6-301da90f75f8} {Suffrage:Voter ID:server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 Address:f0b5b545-0b45-19e0-5d63-cf054b8d8cc2} {Suffrage:Voter ID:server-38381713-7844-9f74-1130-aa008b160926 Address:38381713-7844-9f74-1130-aa008b160926}]" 2025-07-29T00:55:25.038Z [INFO] creating new snapshot: path=/tmp/raft1886719032/snapshots/2-2-1753750525038.tmp 2025-07-29T00:55:25.038Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: entering follower state: follower="Node at 6a81043f-08e5-6c02-21f6-301da90f75f8 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.039Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: starting restore from snapshot: id=2-2-1753750525038 last-index=2 last-term=2 size-in-bytes=1 2025-07-29T00:55:25.039Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: snapshot restore progress: id=2-2-1753750525038 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-07-29T00:55:25.039Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: restored from snapshot: id=2-2-1753750525038 last-index=2 last-term=2 size-in-bytes=1 2025-07-29T00:55:25.039Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a81043f-08e5-6c02-21f6-301da90f75f8 Address:6a81043f-08e5-6c02-21f6-301da90f75f8} {Suffrage:Voter ID:server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 Address:f0b5b545-0b45-19e0-5d63-cf054b8d8cc2} {Suffrage:Voter ID:server-38381713-7844-9f74-1130-aa008b160926 Address:38381713-7844-9f74-1130-aa008b160926}]" 2025-07-29T00:55:25.039Z [INFO] server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2: entering follower state: follower="Node at f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.039Z [INFO] creating new snapshot: path=/tmp/raft3801519420/snapshots/2-2-1753750525039.tmp 2025-07-29T00:55:25.039Z [INFO] server-38381713-7844-9f74-1130-aa008b160926: starting restore from snapshot: id=2-2-1753750525039 last-index=2 last-term=2 size-in-bytes=1 2025-07-29T00:55:25.039Z [INFO] server-38381713-7844-9f74-1130-aa008b160926: snapshot restore progress: id=2-2-1753750525039 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-07-29T00:55:25.040Z [INFO] server-38381713-7844-9f74-1130-aa008b160926: restored from snapshot: id=2-2-1753750525039 last-index=2 last-term=2 size-in-bytes=1 2025-07-29T00:55:25.040Z [INFO] server-38381713-7844-9f74-1130-aa008b160926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a81043f-08e5-6c02-21f6-301da90f75f8 Address:6a81043f-08e5-6c02-21f6-301da90f75f8} {Suffrage:Voter ID:server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 Address:f0b5b545-0b45-19e0-5d63-cf054b8d8cc2} {Suffrage:Voter ID:server-38381713-7844-9f74-1130-aa008b160926 Address:38381713-7844-9f74-1130-aa008b160926}]" 2025-07-29T00:55:25.040Z [INFO] server-38381713-7844-9f74-1130-aa008b160926: entering follower state: follower="Node at 38381713-7844-9f74-1130-aa008b160926 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.090Z [WARN] server-6a81043f-08e5-6c02-21f6-301da90f75f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:25.090Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: entering candidate state: node="Node at 6a81043f-08e5-6c02-21f6-301da90f75f8 [Candidate]" term=3 2025-07-29T00:55:25.090Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: election won: term=3 tally=2 2025-07-29T00:55:25.090Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: entering leader state: leader="Node at 6a81043f-08e5-6c02-21f6-301da90f75f8 [Leader]" 2025-07-29T00:55:25.090Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: added peer, starting replication: peer=server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 2025-07-29T00:55:25.090Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: added peer, starting replication: peer=server-38381713-7844-9f74-1130-aa008b160926 2025-07-29T00:55:25.090Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: pipelining replication: peer="{Voter server-38381713-7844-9f74-1130-aa008b160926 38381713-7844-9f74-1130-aa008b160926}" 2025-07-29T00:55:25.090Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: pipelining replication: peer="{Voter server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 f0b5b545-0b45-19e0-5d63-cf054b8d8cc2}" 2025-07-29T00:55:25.355Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:25.460Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:25.355478417 +0000 UTC m=+5.044859761 (1 nodes), 45µs from start of poll, 500.40028ms from cluster start. Timeout at 2025-07-29 00:55:25.460596457 +0000 UTC m=+5.149977841, 105.11808ms after stability 2025-07-29T00:55:25.460Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: aborting pipeline replication: peer="{Voter server-38381713-7844-9f74-1130-aa008b160926 38381713-7844-9f74-1130-aa008b160926}" 2025-07-29T00:55:25.460Z [INFO] server-6a81043f-08e5-6c02-21f6-301da90f75f8: aborting pipeline replication: peer="{Voter server-f0b5b545-0b45-19e0-5d63-cf054b8d8cc2 f0b5b545-0b45-19e0-5d63-cf054b8d8cc2}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2025-07-29T00:55:25.461Z [INFO] server-666ae2b0-d12d-3a65-e993-2891909c4f64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-666ae2b0-d12d-3a65-e993-2891909c4f64 Address:666ae2b0-d12d-3a65-e993-2891909c4f64} {Suffrage:Voter ID:server-13c1af87-0e21-992f-028e-a6534cf1bf59 Address:13c1af87-0e21-992f-028e-a6534cf1bf59} {Suffrage:Voter ID:server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 Address:c65d2874-6ac9-a637-eca9-9e3f1a8c0423}]" 2025-07-29T00:55:25.461Z [INFO] server-666ae2b0-d12d-3a65-e993-2891909c4f64: entering follower state: follower="Node at 666ae2b0-d12d-3a65-e993-2891909c4f64 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.461Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-666ae2b0-d12d-3a65-e993-2891909c4f64 Address:666ae2b0-d12d-3a65-e993-2891909c4f64} {Suffrage:Voter ID:server-13c1af87-0e21-992f-028e-a6534cf1bf59 Address:13c1af87-0e21-992f-028e-a6534cf1bf59} {Suffrage:Voter ID:server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 Address:c65d2874-6ac9-a637-eca9-9e3f1a8c0423}]" 2025-07-29T00:55:25.462Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: entering follower state: follower="Node at 13c1af87-0e21-992f-028e-a6534cf1bf59 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.462Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-666ae2b0-d12d-3a65-e993-2891909c4f64 Address:666ae2b0-d12d-3a65-e993-2891909c4f64} {Suffrage:Voter ID:server-13c1af87-0e21-992f-028e-a6534cf1bf59 Address:13c1af87-0e21-992f-028e-a6534cf1bf59} {Suffrage:Voter ID:server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 Address:c65d2874-6ac9-a637-eca9-9e3f1a8c0423}]" 2025-07-29T00:55:25.462Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:25.462Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: entering follower state: follower="Node at c65d2874-6ac9-a637-eca9-9e3f1a8c0423 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.538Z [WARN] server-13c1af87-0e21-992f-028e-a6534cf1bf59: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:25.538Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: entering candidate state: node="Node at 13c1af87-0e21-992f-028e-a6534cf1bf59 [Candidate]" term=2 2025-07-29T00:55:25.538Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: election won: term=2 tally=2 2025-07-29T00:55:25.538Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: entering leader state: leader="Node at 13c1af87-0e21-992f-028e-a6534cf1bf59 [Leader]" 2025-07-29T00:55:25.538Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: added peer, starting replication: peer=server-666ae2b0-d12d-3a65-e993-2891909c4f64 2025-07-29T00:55:25.538Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: added peer, starting replication: peer=server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 2025-07-29T00:55:25.538Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: pipelining replication: peer="{Voter server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 c65d2874-6ac9-a637-eca9-9e3f1a8c0423}" 2025-07-29T00:55:25.538Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: pipelining replication: peer="{Voter server-666ae2b0-d12d-3a65-e993-2891909c4f64 666ae2b0-d12d-3a65-e993-2891909c4f64}" 2025-07-29T00:55:25.644Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:25.538882297 +0000 UTC m=+5.228263641 (1 nodes), 76.64428ms from start of poll, 77.26ms from cluster start. Timeout at 2025-07-29 00:55:25.644705617 +0000 UTC m=+5.334087001, 105.82336ms after stability 2025-07-29T00:55:25.645Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: aborting pipeline replication: peer="{Voter server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 c65d2874-6ac9-a637-eca9-9e3f1a8c0423}" 2025-07-29T00:55:25.645Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: aborting pipeline replication: peer="{Voter server-666ae2b0-d12d-3a65-e993-2891909c4f64 666ae2b0-d12d-3a65-e993-2891909c4f64}" 2025-07-29T00:55:25.645Z [INFO] creating new snapshot: path=/tmp/raft2721450924/snapshots/2-6-1753750525645.tmp 2025-07-29T00:55:25.646Z [INFO] server-666ae2b0-d12d-3a65-e993-2891909c4f64: starting restore from snapshot: id=2-6-1753750525645 last-index=6 last-term=2 size-in-bytes=25 2025-07-29T00:55:25.646Z [INFO] server-666ae2b0-d12d-3a65-e993-2891909c4f64: snapshot restore progress: id=2-6-1753750525645 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-07-29T00:55:25.646Z [INFO] server-666ae2b0-d12d-3a65-e993-2891909c4f64: restored from snapshot: id=2-6-1753750525645 last-index=6 last-term=2 size-in-bytes=25 2025-07-29T00:55:25.646Z [INFO] server-666ae2b0-d12d-3a65-e993-2891909c4f64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-666ae2b0-d12d-3a65-e993-2891909c4f64 Address:666ae2b0-d12d-3a65-e993-2891909c4f64} {Suffrage:Voter ID:server-13c1af87-0e21-992f-028e-a6534cf1bf59 Address:13c1af87-0e21-992f-028e-a6534cf1bf59} {Suffrage:Voter ID:server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 Address:c65d2874-6ac9-a637-eca9-9e3f1a8c0423}]" 2025-07-29T00:55:25.647Z [INFO] creating new snapshot: path=/tmp/raft321142185/snapshots/2-6-1753750525647.tmp 2025-07-29T00:55:25.647Z [INFO] server-666ae2b0-d12d-3a65-e993-2891909c4f64: entering follower state: follower="Node at 666ae2b0-d12d-3a65-e993-2891909c4f64 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.648Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: starting restore from snapshot: id=2-6-1753750525647 last-index=6 last-term=2 size-in-bytes=25 2025-07-29T00:55:25.648Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: snapshot restore progress: id=2-6-1753750525647 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-07-29T00:55:25.648Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: restored from snapshot: id=2-6-1753750525647 last-index=6 last-term=2 size-in-bytes=25 2025-07-29T00:55:25.648Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-666ae2b0-d12d-3a65-e993-2891909c4f64 Address:666ae2b0-d12d-3a65-e993-2891909c4f64} {Suffrage:Voter ID:server-13c1af87-0e21-992f-028e-a6534cf1bf59 Address:13c1af87-0e21-992f-028e-a6534cf1bf59} {Suffrage:Voter ID:server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 Address:c65d2874-6ac9-a637-eca9-9e3f1a8c0423}]" 2025-07-29T00:55:25.648Z [INFO] creating new snapshot: path=/tmp/raft3314939383/snapshots/2-6-1753750525648.tmp 2025-07-29T00:55:25.648Z [INFO] server-13c1af87-0e21-992f-028e-a6534cf1bf59: entering follower state: follower="Node at 13c1af87-0e21-992f-028e-a6534cf1bf59 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.649Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: starting restore from snapshot: id=2-6-1753750525648 last-index=6 last-term=2 size-in-bytes=25 2025-07-29T00:55:25.649Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: snapshot restore progress: id=2-6-1753750525648 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-07-29T00:55:25.649Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: restored from snapshot: id=2-6-1753750525648 last-index=6 last-term=2 size-in-bytes=25 2025-07-29T00:55:25.649Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-666ae2b0-d12d-3a65-e993-2891909c4f64 Address:666ae2b0-d12d-3a65-e993-2891909c4f64} {Suffrage:Voter ID:server-13c1af87-0e21-992f-028e-a6534cf1bf59 Address:13c1af87-0e21-992f-028e-a6534cf1bf59} {Suffrage:Voter ID:server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423 Address:c65d2874-6ac9-a637-eca9-9e3f1a8c0423}]" 2025-07-29T00:55:25.649Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: entering follower state: follower="Node at c65d2874-6ac9-a637-eca9-9e3f1a8c0423 [Follower]" leader-address= leader-id= 2025-07-29T00:55:25.710Z [WARN] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:25.710Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: entering candidate state: node="Node at c65d2874-6ac9-a637-eca9-9e3f1a8c0423 [Candidate]" term=3 2025-07-29T00:55:25.710Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: election won: term=3 tally=2 2025-07-29T00:55:25.710Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: entering leader state: leader="Node at c65d2874-6ac9-a637-eca9-9e3f1a8c0423 [Leader]" 2025-07-29T00:55:25.710Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: added peer, starting replication: peer=server-666ae2b0-d12d-3a65-e993-2891909c4f64 2025-07-29T00:55:25.710Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: added peer, starting replication: peer=server-13c1af87-0e21-992f-028e-a6534cf1bf59 2025-07-29T00:55:25.710Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: pipelining replication: peer="{Voter server-13c1af87-0e21-992f-028e-a6534cf1bf59 13c1af87-0e21-992f-028e-a6534cf1bf59}" 2025-07-29T00:55:25.710Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: pipelining replication: peer="{Voter server-666ae2b0-d12d-3a65-e993-2891909c4f64 666ae2b0-d12d-3a65-e993-2891909c4f64}" 2025-07-29T00:55:25.964Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:26.070Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:25.964893537 +0000 UTC m=+5.654274921 (1 nodes), 43.76µs from start of poll, 503.27128ms from cluster start. Timeout at 2025-07-29 00:55:26.070787497 +0000 UTC m=+5.760168841, 105.89392ms after stability 2025-07-29T00:55:26.070Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: aborting pipeline replication: peer="{Voter server-666ae2b0-d12d-3a65-e993-2891909c4f64 666ae2b0-d12d-3a65-e993-2891909c4f64}" 2025-07-29T00:55:26.070Z [INFO] server-c65d2874-6ac9-a637-eca9-9e3f1a8c0423: aborting pipeline replication: peer="{Voter server-13c1af87-0e21-992f-028e-a6534cf1bf59 13c1af87-0e21-992f-028e-a6534cf1bf59}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2025-07-29T00:55:26.071Z [INFO] server-ddf159a3-74d0-b104-eb04-b4cb7e04525c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ddf159a3-74d0-b104-eb04-b4cb7e04525c Address:ddf159a3-74d0-b104-eb04-b4cb7e04525c} {Suffrage:Voter ID:server-f8d5196a-3e16-f67f-1d35-796e17f27069 Address:f8d5196a-3e16-f67f-1d35-796e17f27069} {Suffrage:Voter ID:server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 Address:49dde4d3-da12-8444-6c18-8b5ca1e8f807}]" 2025-07-29T00:55:26.072Z [INFO] server-ddf159a3-74d0-b104-eb04-b4cb7e04525c: entering follower state: follower="Node at ddf159a3-74d0-b104-eb04-b4cb7e04525c [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.072Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ddf159a3-74d0-b104-eb04-b4cb7e04525c Address:ddf159a3-74d0-b104-eb04-b4cb7e04525c} {Suffrage:Voter ID:server-f8d5196a-3e16-f67f-1d35-796e17f27069 Address:f8d5196a-3e16-f67f-1d35-796e17f27069} {Suffrage:Voter ID:server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 Address:49dde4d3-da12-8444-6c18-8b5ca1e8f807}]" 2025-07-29T00:55:26.072Z [INFO] server-49dde4d3-da12-8444-6c18-8b5ca1e8f807: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ddf159a3-74d0-b104-eb04-b4cb7e04525c Address:ddf159a3-74d0-b104-eb04-b4cb7e04525c} {Suffrage:Voter ID:server-f8d5196a-3e16-f67f-1d35-796e17f27069 Address:f8d5196a-3e16-f67f-1d35-796e17f27069} {Suffrage:Voter ID:server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 Address:49dde4d3-da12-8444-6c18-8b5ca1e8f807}]" 2025-07-29T00:55:26.072Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:26.072Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: entering follower state: follower="Node at f8d5196a-3e16-f67f-1d35-796e17f27069 [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.072Z [INFO] server-49dde4d3-da12-8444-6c18-8b5ca1e8f807: entering follower state: follower="Node at 49dde4d3-da12-8444-6c18-8b5ca1e8f807 [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.132Z [WARN] server-f8d5196a-3e16-f67f-1d35-796e17f27069: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:26.132Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: entering candidate state: node="Node at f8d5196a-3e16-f67f-1d35-796e17f27069 [Candidate]" term=2 2025-07-29T00:55:26.132Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: election won: term=2 tally=2 2025-07-29T00:55:26.132Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: entering leader state: leader="Node at f8d5196a-3e16-f67f-1d35-796e17f27069 [Leader]" 2025-07-29T00:55:26.132Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: added peer, starting replication: peer=server-ddf159a3-74d0-b104-eb04-b4cb7e04525c 2025-07-29T00:55:26.132Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: added peer, starting replication: peer=server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 2025-07-29T00:55:26.132Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: pipelining replication: peer="{Voter server-ddf159a3-74d0-b104-eb04-b4cb7e04525c ddf159a3-74d0-b104-eb04-b4cb7e04525c}" 2025-07-29T00:55:26.132Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: pipelining replication: peer="{Voter server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 49dde4d3-da12-8444-6c18-8b5ca1e8f807}" 2025-07-29T00:55:26.237Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:26.132599617 +0000 UTC m=+5.821980961 (1 nodes), 60.33936ms from start of poll, 60.78508ms from cluster start. Timeout at 2025-07-29 00:55:26.237899457 +0000 UTC m=+5.927280801, 105.29984ms after stability 2025-07-29T00:55:26.242Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: aborting pipeline replication: peer="{Voter server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 49dde4d3-da12-8444-6c18-8b5ca1e8f807}" 2025-07-29T00:55:26.242Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: aborting pipeline replication: peer="{Voter server-ddf159a3-74d0-b104-eb04-b4cb7e04525c ddf159a3-74d0-b104-eb04-b4cb7e04525c}" 2025-07-29T00:55:26.242Z [INFO] creating new snapshot: path=/tmp/raft1218647723/snapshots/2-27-1753750526242.tmp 2025-07-29T00:55:26.243Z [INFO] server-ddf159a3-74d0-b104-eb04-b4cb7e04525c: starting restore from snapshot: id=2-27-1753750526242 last-index=27 last-term=2 size-in-bytes=168 2025-07-29T00:55:26.243Z [INFO] server-ddf159a3-74d0-b104-eb04-b4cb7e04525c: snapshot restore progress: id=2-27-1753750526242 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-07-29T00:55:26.243Z [INFO] server-ddf159a3-74d0-b104-eb04-b4cb7e04525c: restored from snapshot: id=2-27-1753750526242 last-index=27 last-term=2 size-in-bytes=168 2025-07-29T00:55:26.243Z [INFO] server-ddf159a3-74d0-b104-eb04-b4cb7e04525c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ddf159a3-74d0-b104-eb04-b4cb7e04525c Address:ddf159a3-74d0-b104-eb04-b4cb7e04525c} {Suffrage:Voter ID:server-f8d5196a-3e16-f67f-1d35-796e17f27069 Address:f8d5196a-3e16-f67f-1d35-796e17f27069} {Suffrage:Voter ID:server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 Address:49dde4d3-da12-8444-6c18-8b5ca1e8f807}]" 2025-07-29T00:55:26.243Z [INFO] server-ddf159a3-74d0-b104-eb04-b4cb7e04525c: entering follower state: follower="Node at ddf159a3-74d0-b104-eb04-b4cb7e04525c [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.243Z [INFO] creating new snapshot: path=/tmp/raft4137759021/snapshots/2-27-1753750526243.tmp 2025-07-29T00:55:26.244Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: starting restore from snapshot: id=2-27-1753750526243 last-index=27 last-term=2 size-in-bytes=168 2025-07-29T00:55:26.244Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: snapshot restore progress: id=2-27-1753750526243 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-07-29T00:55:26.244Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: restored from snapshot: id=2-27-1753750526243 last-index=27 last-term=2 size-in-bytes=168 2025-07-29T00:55:26.244Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ddf159a3-74d0-b104-eb04-b4cb7e04525c Address:ddf159a3-74d0-b104-eb04-b4cb7e04525c} {Suffrage:Voter ID:server-f8d5196a-3e16-f67f-1d35-796e17f27069 Address:f8d5196a-3e16-f67f-1d35-796e17f27069} {Suffrage:Voter ID:server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 Address:49dde4d3-da12-8444-6c18-8b5ca1e8f807}]" 2025-07-29T00:55:26.244Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: entering follower state: follower="Node at f8d5196a-3e16-f67f-1d35-796e17f27069 [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.244Z [INFO] creating new snapshot: path=/tmp/raft3659874674/snapshots/2-27-1753750526244.tmp 2025-07-29T00:55:26.245Z [INFO] server-49dde4d3-da12-8444-6c18-8b5ca1e8f807: starting restore from snapshot: id=2-27-1753750526244 last-index=27 last-term=2 size-in-bytes=168 2025-07-29T00:55:26.245Z [INFO] server-49dde4d3-da12-8444-6c18-8b5ca1e8f807: snapshot restore progress: id=2-27-1753750526244 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-07-29T00:55:26.245Z [INFO] server-49dde4d3-da12-8444-6c18-8b5ca1e8f807: restored from snapshot: id=2-27-1753750526244 last-index=27 last-term=2 size-in-bytes=168 2025-07-29T00:55:26.245Z [INFO] server-49dde4d3-da12-8444-6c18-8b5ca1e8f807: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ddf159a3-74d0-b104-eb04-b4cb7e04525c Address:ddf159a3-74d0-b104-eb04-b4cb7e04525c} {Suffrage:Voter ID:server-f8d5196a-3e16-f67f-1d35-796e17f27069 Address:f8d5196a-3e16-f67f-1d35-796e17f27069} {Suffrage:Voter ID:server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 Address:49dde4d3-da12-8444-6c18-8b5ca1e8f807}]" 2025-07-29T00:55:26.245Z [INFO] server-49dde4d3-da12-8444-6c18-8b5ca1e8f807: entering follower state: follower="Node at 49dde4d3-da12-8444-6c18-8b5ca1e8f807 [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.313Z [WARN] server-f8d5196a-3e16-f67f-1d35-796e17f27069: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:26.313Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: entering candidate state: node="Node at f8d5196a-3e16-f67f-1d35-796e17f27069 [Candidate]" term=3 2025-07-29T00:55:26.313Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: election won: term=3 tally=2 2025-07-29T00:55:26.313Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: entering leader state: leader="Node at f8d5196a-3e16-f67f-1d35-796e17f27069 [Leader]" 2025-07-29T00:55:26.314Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: added peer, starting replication: peer=server-ddf159a3-74d0-b104-eb04-b4cb7e04525c 2025-07-29T00:55:26.314Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: added peer, starting replication: peer=server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 2025-07-29T00:55:26.314Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: pipelining replication: peer="{Voter server-ddf159a3-74d0-b104-eb04-b4cb7e04525c ddf159a3-74d0-b104-eb04-b4cb7e04525c}" 2025-07-29T00:55:26.314Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: pipelining replication: peer="{Voter server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 49dde4d3-da12-8444-6c18-8b5ca1e8f807}" 2025-07-29T00:55:26.560Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:26.666Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:26.560613257 +0000 UTC m=+6.249994641 (1 nodes), 46.84µs from start of poll, 488.79876ms from cluster start. Timeout at 2025-07-29 00:55:26.666365617 +0000 UTC m=+6.355746961, 105.75232ms after stability 2025-07-29T00:55:26.666Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: aborting pipeline replication: peer="{Voter server-49dde4d3-da12-8444-6c18-8b5ca1e8f807 49dde4d3-da12-8444-6c18-8b5ca1e8f807}" 2025-07-29T00:55:26.666Z [INFO] server-f8d5196a-3e16-f67f-1d35-796e17f27069: aborting pipeline replication: peer="{Voter server-ddf159a3-74d0-b104-eb04-b4cb7e04525c ddf159a3-74d0-b104-eb04-b4cb7e04525c}" --- PASS: TestRaft_RecoverCluster (1.81s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.61s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.61s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.60s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2025-07-29T00:55:26.667Z [INFO] server-0bfe950d-7d38-b94a-1497-f64a37add14d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bfe950d-7d38-b94a-1497-f64a37add14d Address:0bfe950d-7d38-b94a-1497-f64a37add14d}]" 2025-07-29T00:55:26.668Z [INFO] server-0bfe950d-7d38-b94a-1497-f64a37add14d: entering follower state: follower="Node at 0bfe950d-7d38-b94a-1497-f64a37add14d [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.718Z [WARN] server-0bfe950d-7d38-b94a-1497-f64a37add14d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:26.718Z [INFO] server-0bfe950d-7d38-b94a-1497-f64a37add14d: entering candidate state: node="Node at 0bfe950d-7d38-b94a-1497-f64a37add14d [Candidate]" term=2 2025-07-29T00:55:26.718Z [INFO] server-0bfe950d-7d38-b94a-1497-f64a37add14d: election won: term=2 tally=1 2025-07-29T00:55:26.718Z [INFO] server-0bfe950d-7d38-b94a-1497-f64a37add14d: entering leader state: leader="Node at 0bfe950d-7d38-b94a-1497-f64a37add14d [Leader]" --- PASS: TestRaft_SingleNode (0.05s) === RUN TestRaft_TripleNode 2025-07-29T00:55:26.719Z [INFO] server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab Address:e7c01ef2-4f59-51f0-7191-80af3c2a72ab} {Suffrage:Voter ID:server-1f6447a1-869b-edce-5c81-e2d956a8849b Address:1f6447a1-869b-edce-5c81-e2d956a8849b} {Suffrage:Voter ID:server-47c31a6b-0f61-abce-7922-9028358c5032 Address:47c31a6b-0f61-abce-7922-9028358c5032}]" 2025-07-29T00:55:26.719Z [INFO] server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab: entering follower state: follower="Node at e7c01ef2-4f59-51f0-7191-80af3c2a72ab [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.719Z [INFO] server-1f6447a1-869b-edce-5c81-e2d956a8849b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab Address:e7c01ef2-4f59-51f0-7191-80af3c2a72ab} {Suffrage:Voter ID:server-1f6447a1-869b-edce-5c81-e2d956a8849b Address:1f6447a1-869b-edce-5c81-e2d956a8849b} {Suffrage:Voter ID:server-47c31a6b-0f61-abce-7922-9028358c5032 Address:47c31a6b-0f61-abce-7922-9028358c5032}]" 2025-07-29T00:55:26.720Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab Address:e7c01ef2-4f59-51f0-7191-80af3c2a72ab} {Suffrage:Voter ID:server-1f6447a1-869b-edce-5c81-e2d956a8849b Address:1f6447a1-869b-edce-5c81-e2d956a8849b} {Suffrage:Voter ID:server-47c31a6b-0f61-abce-7922-9028358c5032 Address:47c31a6b-0f61-abce-7922-9028358c5032}]" 2025-07-29T00:55:26.720Z [INFO] server-1f6447a1-869b-edce-5c81-e2d956a8849b: entering follower state: follower="Node at 1f6447a1-869b-edce-5c81-e2d956a8849b [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.720Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:26.720Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: entering follower state: follower="Node at 47c31a6b-0f61-abce-7922-9028358c5032 [Follower]" leader-address= leader-id= 2025-07-29T00:55:26.781Z [WARN] server-47c31a6b-0f61-abce-7922-9028358c5032: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:26.781Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: entering candidate state: node="Node at 47c31a6b-0f61-abce-7922-9028358c5032 [Candidate]" term=2 2025-07-29T00:55:26.781Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: election won: term=2 tally=2 2025-07-29T00:55:26.781Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: entering leader state: leader="Node at 47c31a6b-0f61-abce-7922-9028358c5032 [Leader]" 2025-07-29T00:55:26.781Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: added peer, starting replication: peer=server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab 2025-07-29T00:55:26.781Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: added peer, starting replication: peer=server-1f6447a1-869b-edce-5c81-e2d956a8849b 2025-07-29T00:55:26.781Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: pipelining replication: peer="{Voter server-1f6447a1-869b-edce-5c81-e2d956a8849b 1f6447a1-869b-edce-5c81-e2d956a8849b}" 2025-07-29T00:55:26.781Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: pipelining replication: peer="{Voter server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab e7c01ef2-4f59-51f0-7191-80af3c2a72ab}" 2025-07-29T00:55:26.886Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:26.781506057 +0000 UTC m=+6.470887401 (2 nodes), 61.41208ms from start of poll, 61.8634ms from cluster start. Timeout at 2025-07-29 00:55:26.886726537 +0000 UTC m=+6.576107881, 105.22048ms after stability 2025-07-29T00:55:26.886Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:26.992Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:26.886849257 +0000 UTC m=+6.576230601 (1 nodes), 1.2µs from start of poll, 167.2066ms from cluster start. Timeout at 2025-07-29 00:55:26.992631337 +0000 UTC m=+6.682012681, 105.78208ms after stability 2025-07-29T00:55:27.004Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: aborting pipeline replication: peer="{Voter server-1f6447a1-869b-edce-5c81-e2d956a8849b 1f6447a1-869b-edce-5c81-e2d956a8849b}" 2025-07-29T00:55:27.004Z [INFO] server-47c31a6b-0f61-abce-7922-9028358c5032: aborting pipeline replication: peer="{Voter server-e7c01ef2-4f59-51f0-7191-80af3c2a72ab e7c01ef2-4f59-51f0-7191-80af3c2a72ab}" --- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail 2025-07-29T00:55:27.005Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e730526e-47ac-0d11-167e-34b7b6493f7b Address:e730526e-47ac-0d11-167e-34b7b6493f7b} {Suffrage:Voter ID:server-0db64d40-a449-a177-b737-ed9bd17d7660 Address:0db64d40-a449-a177-b737-ed9bd17d7660} {Suffrage:Voter ID:server-24d86333-653f-1075-ec98-8a727a260674 Address:24d86333-653f-1075-ec98-8a727a260674}]" 2025-07-29T00:55:27.005Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: entering follower state: follower="Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.005Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e730526e-47ac-0d11-167e-34b7b6493f7b Address:e730526e-47ac-0d11-167e-34b7b6493f7b} {Suffrage:Voter ID:server-0db64d40-a449-a177-b737-ed9bd17d7660 Address:0db64d40-a449-a177-b737-ed9bd17d7660} {Suffrage:Voter ID:server-24d86333-653f-1075-ec98-8a727a260674 Address:24d86333-653f-1075-ec98-8a727a260674}]" 2025-07-29T00:55:27.005Z [INFO] server-24d86333-653f-1075-ec98-8a727a260674: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e730526e-47ac-0d11-167e-34b7b6493f7b Address:e730526e-47ac-0d11-167e-34b7b6493f7b} {Suffrage:Voter ID:server-0db64d40-a449-a177-b737-ed9bd17d7660 Address:0db64d40-a449-a177-b737-ed9bd17d7660} {Suffrage:Voter ID:server-24d86333-653f-1075-ec98-8a727a260674 Address:24d86333-653f-1075-ec98-8a727a260674}]" 2025-07-29T00:55:27.005Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:27.005Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: entering follower state: follower="Node at 0db64d40-a449-a177-b737-ed9bd17d7660 [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.005Z [INFO] server-24d86333-653f-1075-ec98-8a727a260674: entering follower state: follower="Node at 24d86333-653f-1075-ec98-8a727a260674 [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.057Z [WARN] server-e730526e-47ac-0d11-167e-34b7b6493f7b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:27.057Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: entering candidate state: node="Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Candidate]" term=2 2025-07-29T00:55:27.057Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: election won: term=2 tally=2 2025-07-29T00:55:27.057Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: entering leader state: leader="Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Leader]" 2025-07-29T00:55:27.057Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: added peer, starting replication: peer=server-0db64d40-a449-a177-b737-ed9bd17d7660 2025-07-29T00:55:27.057Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: added peer, starting replication: peer=server-24d86333-653f-1075-ec98-8a727a260674 2025-07-29T00:55:27.058Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: pipelining replication: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" 2025-07-29T00:55:27.058Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: pipelining replication: peer="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" 2025-07-29T00:55:27.163Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:27.057960537 +0000 UTC m=+6.747341921 (2 nodes), 52.6738ms from start of poll, 53.09516ms from cluster start. Timeout at 2025-07-29 00:55:27.163066257 +0000 UTC m=+6.852447601, 105.10568ms after stability 2025-07-29T00:55:27.163Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:27.269Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:27.163205657 +0000 UTC m=+6.852587001 (1 nodes), 960ns from start of poll, 158.34024ms from cluster start. Timeout at 2025-07-29 00:55:27.269253577 +0000 UTC m=+6.958634921, 106.04792ms after stability raft_test.go:378: [INFO] Disconnecting Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Leader] 2025-07-29T00:55:27.282Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to heartbeat to: peer=24d86333-653f-1075-ec98-8a727a260674 backoff time=10ms error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.282Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:27.282Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to pipeline appendEntries: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" error="append pipeline closed" 2025-07-29T00:55:27.282Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: aborting pipeline replication: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" 2025-07-29T00:55:27.282Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to pipeline appendEntries: peer="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" error="append pipeline closed" 2025-07-29T00:55:27.282Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: aborting pipeline replication: peer="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" 2025-07-29T00:55:27.283Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to heartbeat to: peer=0db64d40-a449-a177-b737-ed9bd17d7660 backoff time=10ms error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" 2025-07-29T00:55:27.290Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to appendEntries to: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.291Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to appendEntries to: peer="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" 2025-07-29T00:55:27.298Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to heartbeat to: peer=24d86333-653f-1075-ec98-8a727a260674 backoff time=10ms error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.302Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to heartbeat to: peer=0db64d40-a449-a177-b737-ed9bd17d7660 backoff time=10ms error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" 2025-07-29T00:55:27.309Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to appendEntries to: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.309Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to appendEntries to: peer="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" 2025-07-29T00:55:27.316Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to heartbeat to: peer=24d86333-653f-1075-ec98-8a727a260674 backoff time=10ms error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.323Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to heartbeat to: peer=0db64d40-a449-a177-b737-ed9bd17d7660 backoff time=10ms error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" 2025-07-29T00:55:27.326Z [WARN] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to contact: server-id=server-24d86333-653f-1075-ec98-8a727a260674 time=50.30312ms 2025-07-29T00:55:27.327Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to appendEntries to: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.329Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to appendEntries to: peer="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" 2025-07-29T00:55:27.336Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to heartbeat to: peer=24d86333-653f-1075-ec98-8a727a260674 backoff time=20ms error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.337Z [WARN] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to contact: server-id=server-0db64d40-a449-a177-b737-ed9bd17d7660 time=60.05352ms 2025-07-29T00:55:27.337Z [WARN] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to contact: server-id=server-24d86333-653f-1075-ec98-8a727a260674 time=61.31456ms 2025-07-29T00:55:27.337Z [WARN] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to contact quorum of nodes, stepping down 2025-07-29T00:55:27.337Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: entering follower state: follower="Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.351Z [WARN] server-0db64d40-a449-a177-b737-ed9bd17d7660: heartbeat timeout reached, starting election: last-leader-addr=e730526e-47ac-0d11-167e-34b7b6493f7b last-leader-id=server-e730526e-47ac-0d11-167e-34b7b6493f7b 2025-07-29T00:55:27.351Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: entering candidate state: node="Node at 0db64d40-a449-a177-b737-ed9bd17d7660 [Candidate]" term=3 2025-07-29T00:55:27.351Z [WARN] server-24d86333-653f-1075-ec98-8a727a260674: rejecting vote request since we have a leader: from=0db64d40-a449-a177-b737-ed9bd17d7660 leader=e730526e-47ac-0d11-167e-34b7b6493f7b leader-id=server-e730526e-47ac-0d11-167e-34b7b6493f7b 2025-07-29T00:55:27.351Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to make requestVote RPC: target="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" term=3 2025-07-29T00:55:27.354Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to appendEntries to: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" 2025-07-29T00:55:27.390Z [WARN] server-24d86333-653f-1075-ec98-8a727a260674: heartbeat timeout reached, starting election: last-leader-addr=e730526e-47ac-0d11-167e-34b7b6493f7b last-leader-id=server-e730526e-47ac-0d11-167e-34b7b6493f7b 2025-07-29T00:55:27.390Z [INFO] server-24d86333-653f-1075-ec98-8a727a260674: entering candidate state: node="Node at 24d86333-653f-1075-ec98-8a727a260674 [Candidate]" term=3 2025-07-29T00:55:27.390Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: duplicate requestVote for same term: term=3 2025-07-29T00:55:27.390Z [ERROR] server-24d86333-653f-1075-ec98-8a727a260674: failed to make requestVote RPC: target="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" term=3 2025-07-29T00:55:27.428Z [WARN] server-0db64d40-a449-a177-b737-ed9bd17d7660: Election timeout reached, restarting election 2025-07-29T00:55:27.428Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: entering candidate state: node="Node at 0db64d40-a449-a177-b737-ed9bd17d7660 [Candidate]" term=4 2025-07-29T00:55:27.428Z [INFO] server-24d86333-653f-1075-ec98-8a727a260674: entering follower state: follower="Node at 24d86333-653f-1075-ec98-8a727a260674 [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.428Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to make requestVote RPC: target="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" term=4 2025-07-29T00:55:27.428Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: election won: term=4 tally=2 2025-07-29T00:55:27.428Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: entering leader state: leader="Node at 0db64d40-a449-a177-b737-ed9bd17d7660 [Leader]" 2025-07-29T00:55:27.428Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: added peer, starting replication: peer=server-e730526e-47ac-0d11-167e-34b7b6493f7b 2025-07-29T00:55:27.428Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: added peer, starting replication: peer=server-24d86333-653f-1075-ec98-8a727a260674 2025-07-29T00:55:27.428Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to appendEntries to: peer="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.428Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: pipelining replication: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" 2025-07-29T00:55:27.429Z [WARN] server-e730526e-47ac-0d11-167e-34b7b6493f7b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:27.429Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: entering candidate state: node="Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Candidate]" term=3 2025-07-29T00:55:27.429Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to make requestVote RPC: target="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" term=3 2025-07-29T00:55:27.429Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to make requestVote RPC: target="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" term=3 2025-07-29T00:55:27.435Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to heartbeat to: peer=e730526e-47ac-0d11-167e-34b7b6493f7b backoff time=10ms error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.448Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to appendEntries to: peer="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.451Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to heartbeat to: peer=e730526e-47ac-0d11-167e-34b7b6493f7b backoff time=10ms error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.468Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to appendEntries to: peer="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.469Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to heartbeat to: peer=e730526e-47ac-0d11-167e-34b7b6493f7b backoff time=10ms error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.479Z [WARN] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to contact: server-id=server-e730526e-47ac-0d11-167e-34b7b6493f7b time=50.39912ms 2025-07-29T00:55:27.489Z [WARN] server-e730526e-47ac-0d11-167e-34b7b6493f7b: Election timeout reached, restarting election 2025-07-29T00:55:27.489Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: entering candidate state: node="Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Candidate]" term=4 2025-07-29T00:55:27.489Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to heartbeat to: peer=e730526e-47ac-0d11-167e-34b7b6493f7b backoff time=20ms error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.489Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to make requestVote RPC: target="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" error="failed to connect to peer: 24d86333-653f-1075-ec98-8a727a260674" term=4 2025-07-29T00:55:27.489Z [ERROR] server-e730526e-47ac-0d11-167e-34b7b6493f7b: failed to make requestVote RPC: target="{Voter server-0db64d40-a449-a177-b737-ed9bd17d7660 0db64d40-a449-a177-b737-ed9bd17d7660}" error="failed to connect to peer: 0db64d40-a449-a177-b737-ed9bd17d7660" term=4 2025-07-29T00:55:27.496Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to appendEntries to: peer="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.515Z [ERROR] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to heartbeat to: peer=e730526e-47ac-0d11-167e-34b7b6493f7b backoff time=25ms error="failed to connect to peer: e730526e-47ac-0d11-167e-34b7b6493f7b" 2025-07-29T00:55:27.527Z [WARN] server-0db64d40-a449-a177-b737-ed9bd17d7660: failed to contact: server-id=server-e730526e-47ac-0d11-167e-34b7b6493f7b time=99.09476ms 2025-07-29T00:55:27.535Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:27.429893777 +0000 UTC m=+7.119275121 (1 nodes), 147.31196ms from start of poll, 425.02836ms from cluster start. Timeout at 2025-07-29 00:55:27.535528977 +0000 UTC m=+7.224910321, 105.6352ms after stability raft_test.go:413: [INFO] Reconnecting Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Candidate] 2025-07-29T00:55:27.547Z [INFO] server-e730526e-47ac-0d11-167e-34b7b6493f7b: entering follower state: follower="Node at e730526e-47ac-0d11-167e-34b7b6493f7b [Follower]" leader-address=0db64d40-a449-a177-b737-ed9bd17d7660 leader-id=server-0db64d40-a449-a177-b737-ed9bd17d7660 2025-07-29T00:55:27.547Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: pipelining replication: peer="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" 2025-07-29T00:55:27.551Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: aborting pipeline replication: peer="{Voter server-24d86333-653f-1075-ec98-8a727a260674 24d86333-653f-1075-ec98-8a727a260674}" 2025-07-29T00:55:27.551Z [INFO] server-0db64d40-a449-a177-b737-ed9bd17d7660: aborting pipeline replication: peer="{Voter server-e730526e-47ac-0d11-167e-34b7b6493f7b e730526e-47ac-0d11-167e-34b7b6493f7b}" --- PASS: TestRaft_LeaderFail (0.55s) === RUN TestRaft_BehindFollower 2025-07-29T00:55:27.551Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14dab7d3-fd63-0f88-2f13-2fa017b9233a Address:14dab7d3-fd63-0f88-2f13-2fa017b9233a} {Suffrage:Voter ID:server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 Address:3102f7d5-7a71-b371-4bcc-6ca4b56fdd92} {Suffrage:Voter ID:server-cb575ce6-ca21-89fb-1973-20549bfdf4ea Address:cb575ce6-ca21-89fb-1973-20549bfdf4ea}]" 2025-07-29T00:55:27.552Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: entering follower state: follower="Node at 14dab7d3-fd63-0f88-2f13-2fa017b9233a [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.552Z [INFO] server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14dab7d3-fd63-0f88-2f13-2fa017b9233a Address:14dab7d3-fd63-0f88-2f13-2fa017b9233a} {Suffrage:Voter ID:server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 Address:3102f7d5-7a71-b371-4bcc-6ca4b56fdd92} {Suffrage:Voter ID:server-cb575ce6-ca21-89fb-1973-20549bfdf4ea Address:cb575ce6-ca21-89fb-1973-20549bfdf4ea}]" 2025-07-29T00:55:27.552Z [INFO] server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92: entering follower state: follower="Node at 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.552Z [INFO] server-cb575ce6-ca21-89fb-1973-20549bfdf4ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14dab7d3-fd63-0f88-2f13-2fa017b9233a Address:14dab7d3-fd63-0f88-2f13-2fa017b9233a} {Suffrage:Voter ID:server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 Address:3102f7d5-7a71-b371-4bcc-6ca4b56fdd92} {Suffrage:Voter ID:server-cb575ce6-ca21-89fb-1973-20549bfdf4ea Address:cb575ce6-ca21-89fb-1973-20549bfdf4ea}]" 2025-07-29T00:55:27.552Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:27.552Z [INFO] server-cb575ce6-ca21-89fb-1973-20549bfdf4ea: entering follower state: follower="Node at cb575ce6-ca21-89fb-1973-20549bfdf4ea [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.633Z [WARN] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:27.633Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: entering candidate state: node="Node at 14dab7d3-fd63-0f88-2f13-2fa017b9233a [Candidate]" term=2 2025-07-29T00:55:27.634Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: election won: term=2 tally=2 2025-07-29T00:55:27.634Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: entering leader state: leader="Node at 14dab7d3-fd63-0f88-2f13-2fa017b9233a [Leader]" 2025-07-29T00:55:27.634Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: added peer, starting replication: peer=server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 2025-07-29T00:55:27.634Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: added peer, starting replication: peer=server-cb575ce6-ca21-89fb-1973-20549bfdf4ea 2025-07-29T00:55:27.634Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: pipelining replication: peer="{Voter server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92}" 2025-07-29T00:55:27.634Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: pipelining replication: peer="{Voter server-cb575ce6-ca21-89fb-1973-20549bfdf4ea cb575ce6-ca21-89fb-1973-20549bfdf4ea}" 2025-07-29T00:55:27.739Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:27.634156217 +0000 UTC m=+7.323537561 (1 nodes), 81.93272ms from start of poll, 82.38512ms from cluster start. Timeout at 2025-07-29 00:55:27.739566537 +0000 UTC m=+7.428947881, 105.41032ms after stability 2025-07-29T00:55:27.739Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:27.845Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:27.739657457 +0000 UTC m=+7.429038801 (2 nodes), 1.6µs from start of poll, 187.88636ms from cluster start. Timeout at 2025-07-29 00:55:27.845367177 +0000 UTC m=+7.534748521, 105.70972ms after stability 2025-07-29T00:55:27.845Z [ERROR] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: failed to pipeline appendEntries: peer="{Voter server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92}" error="append pipeline closed" 2025-07-29T00:55:27.845Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: aborting pipeline replication: peer="{Voter server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92}" 2025-07-29T00:55:27.845Z [ERROR] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: failed to appendEntries to: peer="{Voter server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92}" error="failed to connect to peer: 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92" 2025-07-29T00:55:27.849Z [ERROR] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: failed to heartbeat to: peer=3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 backoff time=10ms error="failed to connect to peer: 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92" raft_test.go:459: [INFO] Finished apply without behind follower 2025-07-29T00:55:27.856Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: pipelining replication: peer="{Voter server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92}" 2025-07-29T00:55:27.866Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:27.972Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:27.866577337 +0000 UTC m=+7.555958681 (1 nodes), 1.2µs from start of poll, 314.80624ms from cluster start. Timeout at 2025-07-29 00:55:27.972081897 +0000 UTC m=+7.661463241, 105.50456ms after stability 2025-07-29T00:55:27.972Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: aborting pipeline replication: peer="{Voter server-3102f7d5-7a71-b371-4bcc-6ca4b56fdd92 3102f7d5-7a71-b371-4bcc-6ca4b56fdd92}" 2025-07-29T00:55:27.972Z [INFO] server-14dab7d3-fd63-0f88-2f13-2fa017b9233a: aborting pipeline replication: peer="{Voter server-cb575ce6-ca21-89fb-1973-20549bfdf4ea cb575ce6-ca21-89fb-1973-20549bfdf4ea}" --- PASS: TestRaft_BehindFollower (0.42s) === RUN TestRaft_ApplyNonLeader 2025-07-29T00:55:27.973Z [INFO] server-128e93b1-5eb2-3f36-b506-4fb61cb5d106: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-128e93b1-5eb2-3f36-b506-4fb61cb5d106 Address:128e93b1-5eb2-3f36-b506-4fb61cb5d106} {Suffrage:Voter ID:server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d Address:ca19d9ca-55c9-2423-bddd-ebdc0de4415d} {Suffrage:Voter ID:server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc Address:a0e333ce-b6a8-5fd7-20c0-a57260fa2adc}]" 2025-07-29T00:55:27.973Z [INFO] server-128e93b1-5eb2-3f36-b506-4fb61cb5d106: entering follower state: follower="Node at 128e93b1-5eb2-3f36-b506-4fb61cb5d106 [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.973Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-128e93b1-5eb2-3f36-b506-4fb61cb5d106 Address:128e93b1-5eb2-3f36-b506-4fb61cb5d106} {Suffrage:Voter ID:server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d Address:ca19d9ca-55c9-2423-bddd-ebdc0de4415d} {Suffrage:Voter ID:server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc Address:a0e333ce-b6a8-5fd7-20c0-a57260fa2adc}]" 2025-07-29T00:55:27.973Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: entering follower state: follower="Node at ca19d9ca-55c9-2423-bddd-ebdc0de4415d [Follower]" leader-address= leader-id= 2025-07-29T00:55:27.973Z [INFO] server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-128e93b1-5eb2-3f36-b506-4fb61cb5d106 Address:128e93b1-5eb2-3f36-b506-4fb61cb5d106} {Suffrage:Voter ID:server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d Address:ca19d9ca-55c9-2423-bddd-ebdc0de4415d} {Suffrage:Voter ID:server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc Address:a0e333ce-b6a8-5fd7-20c0-a57260fa2adc}]" 2025-07-29T00:55:27.973Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:27.973Z [INFO] server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc: entering follower state: follower="Node at a0e333ce-b6a8-5fd7-20c0-a57260fa2adc [Follower]" leader-address= leader-id= 2025-07-29T00:55:28.054Z [WARN] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:28.054Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: entering candidate state: node="Node at ca19d9ca-55c9-2423-bddd-ebdc0de4415d [Candidate]" term=2 2025-07-29T00:55:28.054Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: election won: term=2 tally=2 2025-07-29T00:55:28.054Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: entering leader state: leader="Node at ca19d9ca-55c9-2423-bddd-ebdc0de4415d [Leader]" 2025-07-29T00:55:28.054Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: added peer, starting replication: peer=server-128e93b1-5eb2-3f36-b506-4fb61cb5d106 2025-07-29T00:55:28.054Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: added peer, starting replication: peer=server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc 2025-07-29T00:55:28.054Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: pipelining replication: peer="{Voter server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc a0e333ce-b6a8-5fd7-20c0-a57260fa2adc}" 2025-07-29T00:55:28.054Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: pipelining replication: peer="{Voter server-128e93b1-5eb2-3f36-b506-4fb61cb5d106 128e93b1-5eb2-3f36-b506-4fb61cb5d106}" 2025-07-29T00:55:28.159Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:28.054489497 +0000 UTC m=+7.743870841 (1 nodes), 81.05688ms from start of poll, 81.50504ms from cluster start. Timeout at 2025-07-29 00:55:28.159500697 +0000 UTC m=+7.848882041, 105.0112ms after stability 2025-07-29T00:55:28.159Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:28.265Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:28.159591897 +0000 UTC m=+7.848973241 (2 nodes), 2.36µs from start of poll, 186.60744ms from cluster start. Timeout at 2025-07-29 00:55:28.265284977 +0000 UTC m=+7.954666321, 105.69308ms after stability 2025-07-29T00:55:28.265Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: aborting pipeline replication: peer="{Voter server-128e93b1-5eb2-3f36-b506-4fb61cb5d106 128e93b1-5eb2-3f36-b506-4fb61cb5d106}" 2025-07-29T00:55:28.265Z [INFO] server-ca19d9ca-55c9-2423-bddd-ebdc0de4415d: aborting pipeline replication: peer="{Voter server-a0e333ce-b6a8-5fd7-20c0-a57260fa2adc a0e333ce-b6a8-5fd7-20c0-a57260fa2adc}" --- PASS: TestRaft_ApplyNonLeader (0.29s) === RUN TestRaft_ApplyConcurrent 2025-07-29T00:55:28.266Z [INFO] server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d Address:87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d} {Suffrage:Voter ID:server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03 Address:52ea698a-f2d3-ed4e-7163-a2e63fcd9b03} {Suffrage:Voter ID:server-589d5689-8ad3-2bc3-43e1-7628fd519b1b Address:589d5689-8ad3-2bc3-43e1-7628fd519b1b}]" 2025-07-29T00:55:28.266Z [INFO] server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d: entering follower state: follower="Node at 87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d [Follower]" leader-address= leader-id= 2025-07-29T00:55:28.266Z [INFO] server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d Address:87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d} {Suffrage:Voter ID:server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03 Address:52ea698a-f2d3-ed4e-7163-a2e63fcd9b03} {Suffrage:Voter ID:server-589d5689-8ad3-2bc3-43e1-7628fd519b1b Address:589d5689-8ad3-2bc3-43e1-7628fd519b1b}]" 2025-07-29T00:55:28.266Z [INFO] server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03: entering follower state: follower="Node at 52ea698a-f2d3-ed4e-7163-a2e63fcd9b03 [Follower]" leader-address= leader-id= 2025-07-29T00:55:28.266Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d Address:87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d} {Suffrage:Voter ID:server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03 Address:52ea698a-f2d3-ed4e-7163-a2e63fcd9b03} {Suffrage:Voter ID:server-589d5689-8ad3-2bc3-43e1-7628fd519b1b Address:589d5689-8ad3-2bc3-43e1-7628fd519b1b}]" 2025-07-29T00:55:28.266Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:28.266Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: entering follower state: follower="Node at 589d5689-8ad3-2bc3-43e1-7628fd519b1b [Follower]" leader-address= leader-id= 2025-07-29T00:55:28.449Z [WARN] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:28.449Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: entering candidate state: node="Node at 589d5689-8ad3-2bc3-43e1-7628fd519b1b [Candidate]" term=2 2025-07-29T00:55:28.449Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: election won: term=2 tally=2 2025-07-29T00:55:28.449Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: entering leader state: leader="Node at 589d5689-8ad3-2bc3-43e1-7628fd519b1b [Leader]" 2025-07-29T00:55:28.449Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: added peer, starting replication: peer=server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d 2025-07-29T00:55:28.449Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: added peer, starting replication: peer=server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03 2025-07-29T00:55:28.449Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: pipelining replication: peer="{Voter server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03 52ea698a-f2d3-ed4e-7163-a2e63fcd9b03}" 2025-07-29T00:55:28.450Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: pipelining replication: peer="{Voter server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d 87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d}" 2025-07-29T00:55:28.655Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:28.449836697 +0000 UTC m=+8.139218041 (1 nodes), 183.16072ms from start of poll, 183.65264ms from cluster start. Timeout at 2025-07-29 00:55:28.655796777 +0000 UTC m=+8.345178121, 205.96008ms after stability 2025-07-29T00:55:28.669Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: aborting pipeline replication: peer="{Voter server-52ea698a-f2d3-ed4e-7163-a2e63fcd9b03 52ea698a-f2d3-ed4e-7163-a2e63fcd9b03}" 2025-07-29T00:55:28.669Z [INFO] server-589d5689-8ad3-2bc3-43e1-7628fd519b1b: aborting pipeline replication: peer="{Voter server-87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d 87f97f7f-a02b-b85d-90a6-9d4ffbab9f6d}" --- PASS: TestRaft_ApplyConcurrent (0.40s) === RUN TestRaft_ApplyConcurrent_Timeout 2025-07-29T00:55:28.670Z [INFO] server-2d69b761-d7ef-d810-ce6d-e24fff630361: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d69b761-d7ef-d810-ce6d-e24fff630361 Address:2d69b761-d7ef-d810-ce6d-e24fff630361}]" 2025-07-29T00:55:28.670Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:28.670Z [INFO] server-2d69b761-d7ef-d810-ce6d-e24fff630361: entering follower state: follower="Node at 2d69b761-d7ef-d810-ce6d-e24fff630361 [Follower]" leader-address= leader-id= 2025-07-29T00:55:28.835Z [WARN] server-2d69b761-d7ef-d810-ce6d-e24fff630361: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:28.835Z [INFO] server-2d69b761-d7ef-d810-ce6d-e24fff630361: entering candidate state: node="Node at 2d69b761-d7ef-d810-ce6d-e24fff630361 [Candidate]" term=2 2025-07-29T00:55:28.835Z [INFO] server-2d69b761-d7ef-d810-ce6d-e24fff630361: election won: term=2 tally=1 2025-07-29T00:55:28.835Z [INFO] server-2d69b761-d7ef-d810-ce6d-e24fff630361: entering leader state: leader="Node at 2d69b761-d7ef-d810-ce6d-e24fff630361 [Leader]" 2025-07-29T00:55:29.037Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:28.835477417 +0000 UTC m=+8.524858761 (1 nodes), 164.80632ms from start of poll, 165.0016ms from cluster start. Timeout at 2025-07-29 00:55:29.037258137 +0000 UTC m=+8.726639481, 201.78072ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.37s) === RUN TestRaft_JoinNode 2025-07-29T00:55:29.038Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fab50951-920e-f0f8-7c00-649760d6cbc9 Address:fab50951-920e-f0f8-7c00-649760d6cbc9} {Suffrage:Voter ID:server-1dd20f41-189e-4923-1603-26578dcdcd97 Address:1dd20f41-189e-4923-1603-26578dcdcd97}]" 2025-07-29T00:55:29.038Z [INFO] server-1dd20f41-189e-4923-1603-26578dcdcd97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fab50951-920e-f0f8-7c00-649760d6cbc9 Address:fab50951-920e-f0f8-7c00-649760d6cbc9} {Suffrage:Voter ID:server-1dd20f41-189e-4923-1603-26578dcdcd97 Address:1dd20f41-189e-4923-1603-26578dcdcd97}]" 2025-07-29T00:55:29.038Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: entering follower state: follower="Node at fab50951-920e-f0f8-7c00-649760d6cbc9 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.038Z [INFO] server-1dd20f41-189e-4923-1603-26578dcdcd97: entering follower state: follower="Node at 1dd20f41-189e-4923-1603-26578dcdcd97 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.038Z [INFO] server-5ec900d8-e291-63b3-ccff-3d110dedf91f: initial configuration: index=0 servers=[] 2025-07-29T00:55:29.038Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:29.038Z [INFO] server-5ec900d8-e291-63b3-ccff-3d110dedf91f: entering follower state: follower="Node at 5ec900d8-e291-63b3-ccff-3d110dedf91f [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.090Z [WARN] server-fab50951-920e-f0f8-7c00-649760d6cbc9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:29.090Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: entering candidate state: node="Node at fab50951-920e-f0f8-7c00-649760d6cbc9 [Candidate]" term=2 2025-07-29T00:55:29.090Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: election won: term=2 tally=2 2025-07-29T00:55:29.090Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: entering leader state: leader="Node at fab50951-920e-f0f8-7c00-649760d6cbc9 [Leader]" 2025-07-29T00:55:29.090Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: added peer, starting replication: peer=server-1dd20f41-189e-4923-1603-26578dcdcd97 2025-07-29T00:55:29.090Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: pipelining replication: peer="{Voter server-1dd20f41-189e-4923-1603-26578dcdcd97 1dd20f41-189e-4923-1603-26578dcdcd97}" 2025-07-29T00:55:29.095Z [WARN] server-5ec900d8-e291-63b3-ccff-3d110dedf91f: no known peers, aborting election 2025-07-29T00:55:29.196Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:29.090636977 +0000 UTC m=+8.780018361 (1 nodes), 52.03864ms from start of poll, 52.54252ms from cluster start. Timeout at 2025-07-29 00:55:29.195994497 +0000 UTC m=+8.885375841, 105.35748ms after stability 2025-07-29T00:55:29.196Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: updating configuration: command=AddVoter server-id=server-5ec900d8-e291-63b3-ccff-3d110dedf91f server-addr=5ec900d8-e291-63b3-ccff-3d110dedf91f servers="[{Suffrage:Voter ID:server-fab50951-920e-f0f8-7c00-649760d6cbc9 Address:fab50951-920e-f0f8-7c00-649760d6cbc9} {Suffrage:Voter ID:server-1dd20f41-189e-4923-1603-26578dcdcd97 Address:1dd20f41-189e-4923-1603-26578dcdcd97} {Suffrage:Voter ID:server-5ec900d8-e291-63b3-ccff-3d110dedf91f Address:5ec900d8-e291-63b3-ccff-3d110dedf91f}]" 2025-07-29T00:55:29.196Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: added peer, starting replication: peer=server-5ec900d8-e291-63b3-ccff-3d110dedf91f 2025-07-29T00:55:29.196Z [WARN] server-5ec900d8-e291-63b3-ccff-3d110dedf91f: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2025-07-29T00:55:29.196Z [WARN] server-fab50951-920e-f0f8-7c00-649760d6cbc9: appendEntries rejected, sending older logs: peer="{Voter server-5ec900d8-e291-63b3-ccff-3d110dedf91f 5ec900d8-e291-63b3-ccff-3d110dedf91f}" next=1 2025-07-29T00:55:29.196Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: pipelining replication: peer="{Voter server-5ec900d8-e291-63b3-ccff-3d110dedf91f 5ec900d8-e291-63b3-ccff-3d110dedf91f}" 2025-07-29T00:55:29.196Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:29.302Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:29.196474497 +0000 UTC m=+8.885855841 (1 nodes), 840ns from start of poll, 158.38ms from cluster start. Timeout at 2025-07-29 00:55:29.302399017 +0000 UTC m=+8.991780361, 105.92452ms after stability 2025-07-29T00:55:29.302Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: aborting pipeline replication: peer="{Voter server-5ec900d8-e291-63b3-ccff-3d110dedf91f 5ec900d8-e291-63b3-ccff-3d110dedf91f}" 2025-07-29T00:55:29.302Z [INFO] server-fab50951-920e-f0f8-7c00-649760d6cbc9: aborting pipeline replication: peer="{Voter server-1dd20f41-189e-4923-1603-26578dcdcd97 1dd20f41-189e-4923-1603-26578dcdcd97}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore 2025-07-29T00:55:29.303Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77478171-4645-abd8-f3d5-dab048983d42 Address:77478171-4645-abd8-f3d5-dab048983d42}]" 2025-07-29T00:55:29.303Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: entering follower state: follower="Node at 77478171-4645-abd8-f3d5-dab048983d42 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.303Z [INFO] server-b5c627f0-ae94-159f-fc4d-405bb2e75971: initial configuration: index=0 servers=[] 2025-07-29T00:55:29.303Z [INFO] server-b5c627f0-ae94-159f-fc4d-405bb2e75971: entering follower state: follower="Node at b5c627f0-ae94-159f-fc4d-405bb2e75971 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.303Z [INFO] server-c970ec9d-fc47-78bc-18ea-611f67215512: initial configuration: index=0 servers=[] 2025-07-29T00:55:29.303Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:29.303Z [INFO] server-c970ec9d-fc47-78bc-18ea-611f67215512: entering follower state: follower="Node at c970ec9d-fc47-78bc-18ea-611f67215512 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.357Z [WARN] server-b5c627f0-ae94-159f-fc4d-405bb2e75971: no known peers, aborting election 2025-07-29T00:55:29.376Z [WARN] server-77478171-4645-abd8-f3d5-dab048983d42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:29.376Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: entering candidate state: node="Node at 77478171-4645-abd8-f3d5-dab048983d42 [Candidate]" term=2 2025-07-29T00:55:29.376Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: election won: term=2 tally=1 2025-07-29T00:55:29.376Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: entering leader state: leader="Node at 77478171-4645-abd8-f3d5-dab048983d42 [Leader]" 2025-07-29T00:55:29.386Z [WARN] server-c970ec9d-fc47-78bc-18ea-611f67215512: no known peers, aborting election 2025-07-29T00:55:29.482Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:29.376562297 +0000 UTC m=+9.065943681 (1 nodes), 72.8014ms from start of poll, 73.35764ms from cluster start. Timeout at 2025-07-29 00:55:29.482226097 +0000 UTC m=+9.171607481, 105.6638ms after stability 2025-07-29T00:55:29.482Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: updating configuration: command=AddVoter server-id=server-b5c627f0-ae94-159f-fc4d-405bb2e75971 server-addr=b5c627f0-ae94-159f-fc4d-405bb2e75971 servers="[{Suffrage:Voter ID:server-77478171-4645-abd8-f3d5-dab048983d42 Address:77478171-4645-abd8-f3d5-dab048983d42} {Suffrage:Voter ID:server-b5c627f0-ae94-159f-fc4d-405bb2e75971 Address:b5c627f0-ae94-159f-fc4d-405bb2e75971}]" 2025-07-29T00:55:29.482Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: added peer, starting replication: peer=server-b5c627f0-ae94-159f-fc4d-405bb2e75971 2025-07-29T00:55:29.482Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:29.482Z [WARN] server-b5c627f0-ae94-159f-fc4d-405bb2e75971: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2025-07-29T00:55:29.482Z [WARN] server-77478171-4645-abd8-f3d5-dab048983d42: appendEntries rejected, sending older logs: peer="{Voter server-b5c627f0-ae94-159f-fc4d-405bb2e75971 b5c627f0-ae94-159f-fc4d-405bb2e75971}" next=1 2025-07-29T00:55:29.482Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: pipelining replication: peer="{Voter server-b5c627f0-ae94-159f-fc4d-405bb2e75971 b5c627f0-ae94-159f-fc4d-405bb2e75971}" 2025-07-29T00:55:29.588Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:29.482587777 +0000 UTC m=+9.171969161 (1 nodes), 840ns from start of poll, 179.38312ms from cluster start. Timeout at 2025-07-29 00:55:29.588123417 +0000 UTC m=+9.277504761, 105.5356ms after stability 2025-07-29T00:55:29.588Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: updating configuration: command=AddVoter server-id=server-c970ec9d-fc47-78bc-18ea-611f67215512 server-addr=c970ec9d-fc47-78bc-18ea-611f67215512 servers="[{Suffrage:Voter ID:server-77478171-4645-abd8-f3d5-dab048983d42 Address:77478171-4645-abd8-f3d5-dab048983d42} {Suffrage:Voter ID:server-b5c627f0-ae94-159f-fc4d-405bb2e75971 Address:b5c627f0-ae94-159f-fc4d-405bb2e75971} {Suffrage:Voter ID:server-c970ec9d-fc47-78bc-18ea-611f67215512 Address:c970ec9d-fc47-78bc-18ea-611f67215512}]" 2025-07-29T00:55:29.588Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: added peer, starting replication: peer=server-c970ec9d-fc47-78bc-18ea-611f67215512 2025-07-29T00:55:29.588Z [WARN] server-c970ec9d-fc47-78bc-18ea-611f67215512: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2025-07-29T00:55:29.588Z [WARN] server-77478171-4645-abd8-f3d5-dab048983d42: appendEntries rejected, sending older logs: peer="{Voter server-c970ec9d-fc47-78bc-18ea-611f67215512 c970ec9d-fc47-78bc-18ea-611f67215512}" next=1 2025-07-29T00:55:29.588Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: pipelining replication: peer="{Voter server-c970ec9d-fc47-78bc-18ea-611f67215512 c970ec9d-fc47-78bc-18ea-611f67215512}" 2025-07-29T00:55:29.588Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:29.694Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:29.588728177 +0000 UTC m=+9.278109521 (1 nodes), 640ns from start of poll, 285.52348ms from cluster start. Timeout at 2025-07-29 00:55:29.694099177 +0000 UTC m=+9.383480521, 105.371ms after stability 2025-07-29T00:55:29.694Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: aborting pipeline replication: peer="{Voter server-c970ec9d-fc47-78bc-18ea-611f67215512 c970ec9d-fc47-78bc-18ea-611f67215512}" 2025-07-29T00:55:29.694Z [INFO] server-77478171-4645-abd8-f3d5-dab048983d42: aborting pipeline replication: peer="{Voter server-b5c627f0-ae94-159f-fc4d-405bb2e75971 b5c627f0-ae94-159f-fc4d-405bb2e75971}" --- PASS: TestRaft_JoinNode_ConfigStore (0.39s) === RUN TestRaft_RemoveFollower 2025-07-29T00:55:29.695Z [INFO] server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 Address:d3ae4aa3-5c0e-cc84-ac08-a8589316a409} {Suffrage:Voter ID:server-382e2f17-3713-4fb7-8069-cc44b3ac3652 Address:382e2f17-3713-4fb7-8069-cc44b3ac3652} {Suffrage:Voter ID:server-911e85f3-c8c4-9cd1-516d-f9969b13da33 Address:911e85f3-c8c4-9cd1-516d-f9969b13da33}]" 2025-07-29T00:55:29.695Z [INFO] server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409: entering follower state: follower="Node at d3ae4aa3-5c0e-cc84-ac08-a8589316a409 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.695Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 Address:d3ae4aa3-5c0e-cc84-ac08-a8589316a409} {Suffrage:Voter ID:server-382e2f17-3713-4fb7-8069-cc44b3ac3652 Address:382e2f17-3713-4fb7-8069-cc44b3ac3652} {Suffrage:Voter ID:server-911e85f3-c8c4-9cd1-516d-f9969b13da33 Address:911e85f3-c8c4-9cd1-516d-f9969b13da33}]" 2025-07-29T00:55:29.695Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: entering follower state: follower="Node at 382e2f17-3713-4fb7-8069-cc44b3ac3652 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.695Z [INFO] server-911e85f3-c8c4-9cd1-516d-f9969b13da33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 Address:d3ae4aa3-5c0e-cc84-ac08-a8589316a409} {Suffrage:Voter ID:server-382e2f17-3713-4fb7-8069-cc44b3ac3652 Address:382e2f17-3713-4fb7-8069-cc44b3ac3652} {Suffrage:Voter ID:server-911e85f3-c8c4-9cd1-516d-f9969b13da33 Address:911e85f3-c8c4-9cd1-516d-f9969b13da33}]" 2025-07-29T00:55:29.695Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:29.695Z [INFO] server-911e85f3-c8c4-9cd1-516d-f9969b13da33: entering follower state: follower="Node at 911e85f3-c8c4-9cd1-516d-f9969b13da33 [Follower]" leader-address= leader-id= 2025-07-29T00:55:29.766Z [WARN] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:29.766Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: entering candidate state: node="Node at 382e2f17-3713-4fb7-8069-cc44b3ac3652 [Candidate]" term=2 2025-07-29T00:55:29.766Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: election won: term=2 tally=2 2025-07-29T00:55:29.767Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: entering leader state: leader="Node at 382e2f17-3713-4fb7-8069-cc44b3ac3652 [Leader]" 2025-07-29T00:55:29.767Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: added peer, starting replication: peer=server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 2025-07-29T00:55:29.767Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: added peer, starting replication: peer=server-911e85f3-c8c4-9cd1-516d-f9969b13da33 2025-07-29T00:55:29.767Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: pipelining replication: peer="{Voter server-911e85f3-c8c4-9cd1-516d-f9969b13da33 911e85f3-c8c4-9cd1-516d-f9969b13da33}" 2025-07-29T00:55:29.767Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: pipelining replication: peer="{Voter server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 d3ae4aa3-5c0e-cc84-ac08-a8589316a409}" 2025-07-29T00:55:29.873Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:29.767005737 +0000 UTC m=+9.456387081 (1 nodes), 71.52844ms from start of poll, 72.02ms from cluster start. Timeout at 2025-07-29 00:55:29.872995697 +0000 UTC m=+9.562377041, 105.98996ms after stability 2025-07-29T00:55:29.878Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:29.983Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:29.878598697 +0000 UTC m=+9.567980081 (2 nodes), 2.6µs from start of poll, 183.613ms from cluster start. Timeout at 2025-07-29 00:55:29.983761497 +0000 UTC m=+9.673142841, 105.16276ms after stability 2025-07-29T00:55:29.983Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: updating configuration: command=RemoveServer server-id=server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 server-addr= servers="[{Suffrage:Voter ID:server-382e2f17-3713-4fb7-8069-cc44b3ac3652 Address:382e2f17-3713-4fb7-8069-cc44b3ac3652} {Suffrage:Voter ID:server-911e85f3-c8c4-9cd1-516d-f9969b13da33 Address:911e85f3-c8c4-9cd1-516d-f9969b13da33}]" 2025-07-29T00:55:29.983Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: removed peer, stopping replication: peer=server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 last-index=3 2025-07-29T00:55:29.984Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: aborting pipeline replication: peer="{Voter server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409 d3ae4aa3-5c0e-cc84-ac08-a8589316a409}" 2025-07-29T00:55:30.051Z [WARN] server-d3ae4aa3-5c0e-cc84-ac08-a8589316a409: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-07-29T00:55:30.089Z [INFO] server-382e2f17-3713-4fb7-8069-cc44b3ac3652: aborting pipeline replication: peer="{Voter server-911e85f3-c8c4-9cd1-516d-f9969b13da33 911e85f3-c8c4-9cd1-516d-f9969b13da33}" --- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader 2025-07-29T00:55:30.090Z [INFO] server-d19042f5-f7fe-f249-76a8-783c61476ea4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d19042f5-f7fe-f249-76a8-783c61476ea4 Address:d19042f5-f7fe-f249-76a8-783c61476ea4} {Suffrage:Voter ID:server-be22523d-f375-22be-fae8-f7fc25ccc294 Address:be22523d-f375-22be-fae8-f7fc25ccc294} {Suffrage:Voter ID:server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 Address:fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1}]" 2025-07-29T00:55:30.090Z [INFO] server-d19042f5-f7fe-f249-76a8-783c61476ea4: entering follower state: follower="Node at d19042f5-f7fe-f249-76a8-783c61476ea4 [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.090Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d19042f5-f7fe-f249-76a8-783c61476ea4 Address:d19042f5-f7fe-f249-76a8-783c61476ea4} {Suffrage:Voter ID:server-be22523d-f375-22be-fae8-f7fc25ccc294 Address:be22523d-f375-22be-fae8-f7fc25ccc294} {Suffrage:Voter ID:server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 Address:fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1}]" 2025-07-29T00:55:30.090Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: entering follower state: follower="Node at be22523d-f375-22be-fae8-f7fc25ccc294 [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.090Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d19042f5-f7fe-f249-76a8-783c61476ea4 Address:d19042f5-f7fe-f249-76a8-783c61476ea4} {Suffrage:Voter ID:server-be22523d-f375-22be-fae8-f7fc25ccc294 Address:be22523d-f375-22be-fae8-f7fc25ccc294} {Suffrage:Voter ID:server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 Address:fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1}]" 2025-07-29T00:55:30.090Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:30.090Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: entering follower state: follower="Node at fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.146Z [WARN] server-be22523d-f375-22be-fae8-f7fc25ccc294: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:30.146Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: entering candidate state: node="Node at be22523d-f375-22be-fae8-f7fc25ccc294 [Candidate]" term=2 2025-07-29T00:55:30.146Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: election won: term=2 tally=2 2025-07-29T00:55:30.146Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: entering leader state: leader="Node at be22523d-f375-22be-fae8-f7fc25ccc294 [Leader]" 2025-07-29T00:55:30.146Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: added peer, starting replication: peer=server-d19042f5-f7fe-f249-76a8-783c61476ea4 2025-07-29T00:55:30.146Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: added peer, starting replication: peer=server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 2025-07-29T00:55:30.146Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: pipelining replication: peer="{Voter server-d19042f5-f7fe-f249-76a8-783c61476ea4 d19042f5-f7fe-f249-76a8-783c61476ea4}" 2025-07-29T00:55:30.146Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: pipelining replication: peer="{Voter server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1}" 2025-07-29T00:55:30.252Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:30.146424217 +0000 UTC m=+9.835805561 (1 nodes), 55.43264ms from start of poll, 55.89452ms from cluster start. Timeout at 2025-07-29 00:55:30.251995097 +0000 UTC m=+9.941376441, 105.57088ms after stability 2025-07-29T00:55:30.257Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:30.362Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:30.257408897 +0000 UTC m=+9.946790241 (2 nodes), 1.72µs from start of poll, 166.8792ms from cluster start. Timeout at 2025-07-29 00:55:30.362505697 +0000 UTC m=+10.051887041, 105.0968ms after stability 2025-07-29T00:55:30.362Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: updating configuration: command=RemoveServer server-id=server-be22523d-f375-22be-fae8-f7fc25ccc294 server-addr= servers="[{Suffrage:Voter ID:server-d19042f5-f7fe-f249-76a8-783c61476ea4 Address:d19042f5-f7fe-f249-76a8-783c61476ea4} {Suffrage:Voter ID:server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 Address:fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1}]" 2025-07-29T00:55:30.362Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: removed ourself, shutting down 2025-07-29T00:55:30.362Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: aborting pipeline replication: peer="{Voter server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1}" 2025-07-29T00:55:30.362Z [INFO] server-be22523d-f375-22be-fae8-f7fc25ccc294: aborting pipeline replication: peer="{Voter server-d19042f5-f7fe-f249-76a8-783c61476ea4 d19042f5-f7fe-f249-76a8-783c61476ea4}" 2025-07-29T00:55:30.417Z [WARN] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: heartbeat timeout reached, starting election: last-leader-addr=be22523d-f375-22be-fae8-f7fc25ccc294 last-leader-id=server-be22523d-f375-22be-fae8-f7fc25ccc294 2025-07-29T00:55:30.417Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: entering candidate state: node="Node at fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 [Candidate]" term=3 2025-07-29T00:55:30.417Z [WARN] server-d19042f5-f7fe-f249-76a8-783c61476ea4: rejecting vote request since we have a leader: from=fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 leader=be22523d-f375-22be-fae8-f7fc25ccc294 leader-id=server-be22523d-f375-22be-fae8-f7fc25ccc294 2025-07-29T00:55:30.459Z [WARN] server-d19042f5-f7fe-f249-76a8-783c61476ea4: heartbeat timeout reached, starting election: last-leader-addr=be22523d-f375-22be-fae8-f7fc25ccc294 last-leader-id=server-be22523d-f375-22be-fae8-f7fc25ccc294 2025-07-29T00:55:30.459Z [INFO] server-d19042f5-f7fe-f249-76a8-783c61476ea4: entering candidate state: node="Node at d19042f5-f7fe-f249-76a8-783c61476ea4 [Candidate]" term=3 2025-07-29T00:55:30.459Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: duplicate requestVote for same term: term=3 2025-07-29T00:55:30.514Z [WARN] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: Election timeout reached, restarting election 2025-07-29T00:55:30.514Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: entering candidate state: node="Node at fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 [Candidate]" term=4 2025-07-29T00:55:30.514Z [INFO] server-d19042f5-f7fe-f249-76a8-783c61476ea4: entering follower state: follower="Node at d19042f5-f7fe-f249-76a8-783c61476ea4 [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.514Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: election won: term=4 tally=2 2025-07-29T00:55:30.514Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: entering leader state: leader="Node at fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1 [Leader]" 2025-07-29T00:55:30.514Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: added peer, starting replication: peer=server-d19042f5-f7fe-f249-76a8-783c61476ea4 2025-07-29T00:55:30.514Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: pipelining replication: peer="{Voter server-d19042f5-f7fe-f249-76a8-783c61476ea4 d19042f5-f7fe-f249-76a8-783c61476ea4}" 2025-07-29T00:55:30.574Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:30.679Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:30.574132177 +0000 UTC m=+10.263513561 (1 nodes), 70.8µs from start of poll, 483.60252ms from cluster start. Timeout at 2025-07-29 00:55:30.679432097 +0000 UTC m=+10.368813441, 105.29988ms after stability 2025-07-29T00:55:30.679Z [INFO] server-fa5ea92d-c43a-ffb1-e438-9a0a88fbc3b1: aborting pipeline replication: peer="{Voter server-d19042f5-f7fe-f249-76a8-783c61476ea4 d19042f5-f7fe-f249-76a8-783c61476ea4}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown 2025-07-29T00:55:30.680Z [INFO] server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed Address:c1f9f1c1-5418-d1f1-5b61-84adf3f601ed} {Suffrage:Voter ID:server-b92c1c73-938a-42f1-287f-1b84b8d147b8 Address:b92c1c73-938a-42f1-287f-1b84b8d147b8} {Suffrage:Voter ID:server-0c4ce656-9974-388d-693f-e7a85d23aabf Address:0c4ce656-9974-388d-693f-e7a85d23aabf}]" 2025-07-29T00:55:30.680Z [INFO] server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed: entering follower state: follower="Node at c1f9f1c1-5418-d1f1-5b61-84adf3f601ed [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.681Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed Address:c1f9f1c1-5418-d1f1-5b61-84adf3f601ed} {Suffrage:Voter ID:server-b92c1c73-938a-42f1-287f-1b84b8d147b8 Address:b92c1c73-938a-42f1-287f-1b84b8d147b8} {Suffrage:Voter ID:server-0c4ce656-9974-388d-693f-e7a85d23aabf Address:0c4ce656-9974-388d-693f-e7a85d23aabf}]" 2025-07-29T00:55:30.681Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: entering follower state: follower="Node at b92c1c73-938a-42f1-287f-1b84b8d147b8 [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.681Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed Address:c1f9f1c1-5418-d1f1-5b61-84adf3f601ed} {Suffrage:Voter ID:server-b92c1c73-938a-42f1-287f-1b84b8d147b8 Address:b92c1c73-938a-42f1-287f-1b84b8d147b8} {Suffrage:Voter ID:server-0c4ce656-9974-388d-693f-e7a85d23aabf Address:0c4ce656-9974-388d-693f-e7a85d23aabf}]" 2025-07-29T00:55:30.681Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:30.681Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: entering follower state: follower="Node at 0c4ce656-9974-388d-693f-e7a85d23aabf [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.738Z [WARN] server-0c4ce656-9974-388d-693f-e7a85d23aabf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:30.738Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: entering candidate state: node="Node at 0c4ce656-9974-388d-693f-e7a85d23aabf [Candidate]" term=2 2025-07-29T00:55:30.738Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: election won: term=2 tally=2 2025-07-29T00:55:30.738Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: entering leader state: leader="Node at 0c4ce656-9974-388d-693f-e7a85d23aabf [Leader]" 2025-07-29T00:55:30.738Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: added peer, starting replication: peer=server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed 2025-07-29T00:55:30.738Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: added peer, starting replication: peer=server-b92c1c73-938a-42f1-287f-1b84b8d147b8 2025-07-29T00:55:30.738Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: pipelining replication: peer="{Voter server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed c1f9f1c1-5418-d1f1-5b61-84adf3f601ed}" 2025-07-29T00:55:30.738Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: pipelining replication: peer="{Voter server-b92c1c73-938a-42f1-287f-1b84b8d147b8 b92c1c73-938a-42f1-287f-1b84b8d147b8}" 2025-07-29T00:55:30.844Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:30.738790777 +0000 UTC m=+10.428172121 (2 nodes), 56.87148ms from start of poll, 58.4672ms from cluster start. Timeout at 2025-07-29 00:55:30.844544297 +0000 UTC m=+10.533925681, 105.75356ms after stability 2025-07-29T00:55:30.844Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:30.950Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:30.844685897 +0000 UTC m=+10.534067281 (1 nodes), 1.28µs from start of poll, 164.36236ms from cluster start. Timeout at 2025-07-29 00:55:30.950574377 +0000 UTC m=+10.639955721, 105.88844ms after stability 2025-07-29T00:55:30.955Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: updating configuration: command=RemoveServer server-id=server-0c4ce656-9974-388d-693f-e7a85d23aabf server-addr= servers="[{Suffrage:Voter ID:server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed Address:c1f9f1c1-5418-d1f1-5b61-84adf3f601ed} {Suffrage:Voter ID:server-b92c1c73-938a-42f1-287f-1b84b8d147b8 Address:b92c1c73-938a-42f1-287f-1b84b8d147b8}]" 2025-07-29T00:55:30.955Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: removed ourself, transitioning to follower 2025-07-29T00:55:30.955Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: entering follower state: follower="Node at 0c4ce656-9974-388d-693f-e7a85d23aabf [Follower]" leader-address= leader-id= 2025-07-29T00:55:30.955Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: aborting pipeline replication: peer="{Voter server-b92c1c73-938a-42f1-287f-1b84b8d147b8 b92c1c73-938a-42f1-287f-1b84b8d147b8}" 2025-07-29T00:55:30.955Z [INFO] server-0c4ce656-9974-388d-693f-e7a85d23aabf: aborting pipeline replication: peer="{Voter server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed c1f9f1c1-5418-d1f1-5b61-84adf3f601ed}" 2025-07-29T00:55:31.006Z [WARN] server-0c4ce656-9974-388d-693f-e7a85d23aabf: not part of stable configuration, aborting election 2025-07-29T00:55:31.046Z [WARN] server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed: heartbeat timeout reached, starting election: last-leader-addr=0c4ce656-9974-388d-693f-e7a85d23aabf last-leader-id=server-0c4ce656-9974-388d-693f-e7a85d23aabf 2025-07-29T00:55:31.046Z [INFO] server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed: entering candidate state: node="Node at c1f9f1c1-5418-d1f1-5b61-84adf3f601ed [Candidate]" term=3 2025-07-29T00:55:31.047Z [WARN] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: rejecting vote request since we have a leader: from=c1f9f1c1-5418-d1f1-5b61-84adf3f601ed leader=0c4ce656-9974-388d-693f-e7a85d23aabf leader-id=server-0c4ce656-9974-388d-693f-e7a85d23aabf 2025-07-29T00:55:31.061Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:31.067Z [WARN] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: heartbeat timeout reached, starting election: last-leader-addr=0c4ce656-9974-388d-693f-e7a85d23aabf last-leader-id=server-0c4ce656-9974-388d-693f-e7a85d23aabf 2025-07-29T00:55:31.067Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: entering candidate state: node="Node at b92c1c73-938a-42f1-287f-1b84b8d147b8 [Candidate]" term=3 2025-07-29T00:55:31.068Z [INFO] server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed: duplicate requestVote for same term: term=3 2025-07-29T00:55:31.130Z [WARN] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: Election timeout reached, restarting election 2025-07-29T00:55:31.130Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: entering candidate state: node="Node at b92c1c73-938a-42f1-287f-1b84b8d147b8 [Candidate]" term=4 2025-07-29T00:55:31.130Z [INFO] server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed: entering follower state: follower="Node at c1f9f1c1-5418-d1f1-5b61-84adf3f601ed [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.130Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: election won: term=4 tally=2 2025-07-29T00:55:31.130Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: entering leader state: leader="Node at b92c1c73-938a-42f1-287f-1b84b8d147b8 [Leader]" 2025-07-29T00:55:31.130Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: added peer, starting replication: peer=server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed 2025-07-29T00:55:31.130Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: pipelining replication: peer="{Voter server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed c1f9f1c1-5418-d1f1-5b61-84adf3f601ed}" 2025-07-29T00:55:31.235Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:31.130683137 +0000 UTC m=+10.820064481 (1 nodes), 69.28444ms from start of poll, 450.35956ms from cluster start. Timeout at 2025-07-29 00:55:31.235708017 +0000 UTC m=+10.925089361, 105.02488ms after stability 2025-07-29T00:55:31.340Z [INFO] server-b92c1c73-938a-42f1-287f-1b84b8d147b8: aborting pipeline replication: peer="{Voter server-c1f9f1c1-5418-d1f1-5b61-84adf3f601ed c1f9f1c1-5418-d1f1-5b61-84adf3f601ed}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.66s) === RUN TestRaft_RemoveFollower_SplitCluster 2025-07-29T00:55:31.342Z [INFO] server-edcad3d0-f25a-de50-f0a3-9631d24ade7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edcad3d0-f25a-de50-f0a3-9631d24ade7c Address:edcad3d0-f25a-de50-f0a3-9631d24ade7c} {Suffrage:Voter ID:server-1ce75d95-a882-37e7-28a2-f0666e6928f2 Address:1ce75d95-a882-37e7-28a2-f0666e6928f2} {Suffrage:Voter ID:server-72cc640c-7d68-24de-59ec-f45f39388384 Address:72cc640c-7d68-24de-59ec-f45f39388384} {Suffrage:Voter ID:server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 Address:56f377ae-4785-be16-bfc2-f0c409b1e5f9}]" 2025-07-29T00:55:31.342Z [INFO] server-edcad3d0-f25a-de50-f0a3-9631d24ade7c: entering follower state: follower="Node at edcad3d0-f25a-de50-f0a3-9631d24ade7c [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.342Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edcad3d0-f25a-de50-f0a3-9631d24ade7c Address:edcad3d0-f25a-de50-f0a3-9631d24ade7c} {Suffrage:Voter ID:server-1ce75d95-a882-37e7-28a2-f0666e6928f2 Address:1ce75d95-a882-37e7-28a2-f0666e6928f2} {Suffrage:Voter ID:server-72cc640c-7d68-24de-59ec-f45f39388384 Address:72cc640c-7d68-24de-59ec-f45f39388384} {Suffrage:Voter ID:server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 Address:56f377ae-4785-be16-bfc2-f0c409b1e5f9}]" 2025-07-29T00:55:31.342Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: entering follower state: follower="Node at 1ce75d95-a882-37e7-28a2-f0666e6928f2 [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.342Z [INFO] server-72cc640c-7d68-24de-59ec-f45f39388384: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edcad3d0-f25a-de50-f0a3-9631d24ade7c Address:edcad3d0-f25a-de50-f0a3-9631d24ade7c} {Suffrage:Voter ID:server-1ce75d95-a882-37e7-28a2-f0666e6928f2 Address:1ce75d95-a882-37e7-28a2-f0666e6928f2} {Suffrage:Voter ID:server-72cc640c-7d68-24de-59ec-f45f39388384 Address:72cc640c-7d68-24de-59ec-f45f39388384} {Suffrage:Voter ID:server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 Address:56f377ae-4785-be16-bfc2-f0c409b1e5f9}]" 2025-07-29T00:55:31.342Z [INFO] server-72cc640c-7d68-24de-59ec-f45f39388384: entering follower state: follower="Node at 72cc640c-7d68-24de-59ec-f45f39388384 [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.342Z [INFO] server-56f377ae-4785-be16-bfc2-f0c409b1e5f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edcad3d0-f25a-de50-f0a3-9631d24ade7c Address:edcad3d0-f25a-de50-f0a3-9631d24ade7c} {Suffrage:Voter ID:server-1ce75d95-a882-37e7-28a2-f0666e6928f2 Address:1ce75d95-a882-37e7-28a2-f0666e6928f2} {Suffrage:Voter ID:server-72cc640c-7d68-24de-59ec-f45f39388384 Address:72cc640c-7d68-24de-59ec-f45f39388384} {Suffrage:Voter ID:server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 Address:56f377ae-4785-be16-bfc2-f0c409b1e5f9}]" 2025-07-29T00:55:31.342Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:31.342Z [INFO] server-56f377ae-4785-be16-bfc2-f0c409b1e5f9: entering follower state: follower="Node at 56f377ae-4785-be16-bfc2-f0c409b1e5f9 [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.393Z [WARN] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:31.393Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: entering candidate state: node="Node at 1ce75d95-a882-37e7-28a2-f0666e6928f2 [Candidate]" term=2 2025-07-29T00:55:31.393Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: election won: term=2 tally=3 2025-07-29T00:55:31.393Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: entering leader state: leader="Node at 1ce75d95-a882-37e7-28a2-f0666e6928f2 [Leader]" 2025-07-29T00:55:31.393Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: added peer, starting replication: peer=server-edcad3d0-f25a-de50-f0a3-9631d24ade7c 2025-07-29T00:55:31.393Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: added peer, starting replication: peer=server-72cc640c-7d68-24de-59ec-f45f39388384 2025-07-29T00:55:31.393Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: added peer, starting replication: peer=server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 2025-07-29T00:55:31.394Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: pipelining replication: peer="{Voter server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 56f377ae-4785-be16-bfc2-f0c409b1e5f9}" 2025-07-29T00:55:31.394Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: pipelining replication: peer="{Voter server-edcad3d0-f25a-de50-f0a3-9631d24ade7c edcad3d0-f25a-de50-f0a3-9631d24ade7c}" 2025-07-29T00:55:31.394Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: pipelining replication: peer="{Voter server-72cc640c-7d68-24de-59ec-f45f39388384 72cc640c-7d68-24de-59ec-f45f39388384}" 2025-07-29T00:55:31.500Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:31.393940337 +0000 UTC m=+11.083321721 (1 nodes), 51.5024ms from start of poll, 52.10264ms from cluster start. Timeout at 2025-07-29 00:55:31.500073057 +0000 UTC m=+11.189454401, 106.13268ms after stability 2025-07-29T00:55:31.605Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:31.711Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:31.605786737 +0000 UTC m=+11.295168081 (3 nodes), 1.6µs from start of poll, 263.949ms from cluster start. Timeout at 2025-07-29 00:55:31.711043817 +0000 UTC m=+11.400425161, 105.25708ms after stability 2025-07-29T00:55:31.711Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: updating configuration: command=RemoveServer server-id=server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 server-addr= servers="[{Suffrage:Voter ID:server-edcad3d0-f25a-de50-f0a3-9631d24ade7c Address:edcad3d0-f25a-de50-f0a3-9631d24ade7c} {Suffrage:Voter ID:server-1ce75d95-a882-37e7-28a2-f0666e6928f2 Address:1ce75d95-a882-37e7-28a2-f0666e6928f2} {Suffrage:Voter ID:server-72cc640c-7d68-24de-59ec-f45f39388384 Address:72cc640c-7d68-24de-59ec-f45f39388384}]" 2025-07-29T00:55:31.711Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: removed peer, stopping replication: peer=server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 last-index=3 2025-07-29T00:55:31.711Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to pipeline appendEntries: peer="{Voter server-72cc640c-7d68-24de-59ec-f45f39388384 72cc640c-7d68-24de-59ec-f45f39388384}" error="append pipeline closed" 2025-07-29T00:55:31.711Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: aborting pipeline replication: peer="{Voter server-72cc640c-7d68-24de-59ec-f45f39388384 72cc640c-7d68-24de-59ec-f45f39388384}" 2025-07-29T00:55:31.711Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to pipeline appendEntries: peer="{Voter server-edcad3d0-f25a-de50-f0a3-9631d24ade7c edcad3d0-f25a-de50-f0a3-9631d24ade7c}" error="append pipeline closed" 2025-07-29T00:55:31.711Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: aborting pipeline replication: peer="{Voter server-edcad3d0-f25a-de50-f0a3-9631d24ade7c edcad3d0-f25a-de50-f0a3-9631d24ade7c}" 2025-07-29T00:55:31.711Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: aborting pipeline replication: peer="{Voter server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 56f377ae-4785-be16-bfc2-f0c409b1e5f9}" 2025-07-29T00:55:31.716Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to heartbeat to: peer=72cc640c-7d68-24de-59ec-f45f39388384 backoff time=10ms error="failed to connect to peer: 72cc640c-7d68-24de-59ec-f45f39388384" 2025-07-29T00:55:31.717Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to heartbeat to: peer=edcad3d0-f25a-de50-f0a3-9631d24ade7c backoff time=10ms error="failed to connect to peer: edcad3d0-f25a-de50-f0a3-9631d24ade7c" 2025-07-29T00:55:31.718Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-72cc640c-7d68-24de-59ec-f45f39388384 72cc640c-7d68-24de-59ec-f45f39388384}" error="failed to connect to peer: 72cc640c-7d68-24de-59ec-f45f39388384" 2025-07-29T00:55:31.721Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-edcad3d0-f25a-de50-f0a3-9631d24ade7c edcad3d0-f25a-de50-f0a3-9631d24ade7c}" error="failed to connect to peer: edcad3d0-f25a-de50-f0a3-9631d24ade7c" 2025-07-29T00:55:31.736Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to heartbeat to: peer=edcad3d0-f25a-de50-f0a3-9631d24ade7c backoff time=10ms error="failed to connect to peer: edcad3d0-f25a-de50-f0a3-9631d24ade7c" 2025-07-29T00:55:31.738Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to heartbeat to: peer=72cc640c-7d68-24de-59ec-f45f39388384 backoff time=10ms error="failed to connect to peer: 72cc640c-7d68-24de-59ec-f45f39388384" 2025-07-29T00:55:31.738Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-72cc640c-7d68-24de-59ec-f45f39388384 72cc640c-7d68-24de-59ec-f45f39388384}" error="failed to connect to peer: 72cc640c-7d68-24de-59ec-f45f39388384" 2025-07-29T00:55:31.739Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-edcad3d0-f25a-de50-f0a3-9631d24ade7c edcad3d0-f25a-de50-f0a3-9631d24ade7c}" error="failed to connect to peer: edcad3d0-f25a-de50-f0a3-9631d24ade7c" 2025-07-29T00:55:31.754Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to heartbeat to: peer=72cc640c-7d68-24de-59ec-f45f39388384 backoff time=10ms error="failed to connect to peer: 72cc640c-7d68-24de-59ec-f45f39388384" 2025-07-29T00:55:31.757Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-edcad3d0-f25a-de50-f0a3-9631d24ade7c edcad3d0-f25a-de50-f0a3-9631d24ade7c}" error="failed to connect to peer: edcad3d0-f25a-de50-f0a3-9631d24ade7c" 2025-07-29T00:55:31.757Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to heartbeat to: peer=edcad3d0-f25a-de50-f0a3-9631d24ade7c backoff time=10ms error="failed to connect to peer: edcad3d0-f25a-de50-f0a3-9631d24ade7c" 2025-07-29T00:55:31.758Z [WARN] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to contact: server-id=server-72cc640c-7d68-24de-59ec-f45f39388384 time=50.52656ms 2025-07-29T00:55:31.759Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-72cc640c-7d68-24de-59ec-f45f39388384 72cc640c-7d68-24de-59ec-f45f39388384}" error="failed to connect to peer: 72cc640c-7d68-24de-59ec-f45f39388384" 2025-07-29T00:55:31.765Z [WARN] server-edcad3d0-f25a-de50-f0a3-9631d24ade7c: heartbeat timeout reached, starting election: last-leader-addr=1ce75d95-a882-37e7-28a2-f0666e6928f2 last-leader-id=server-1ce75d95-a882-37e7-28a2-f0666e6928f2 2025-07-29T00:55:31.765Z [INFO] server-edcad3d0-f25a-de50-f0a3-9631d24ade7c: entering candidate state: node="Node at edcad3d0-f25a-de50-f0a3-9631d24ade7c [Candidate]" term=3 2025-07-29T00:55:31.766Z [ERROR] server-edcad3d0-f25a-de50-f0a3-9631d24ade7c: failed to make requestVote RPC: target="{Voter server-56f377ae-4785-be16-bfc2-f0c409b1e5f9 56f377ae-4785-be16-bfc2-f0c409b1e5f9}" error="failed to connect to peer: 56f377ae-4785-be16-bfc2-f0c409b1e5f9" term=3 2025-07-29T00:55:31.766Z [WARN] server-72cc640c-7d68-24de-59ec-f45f39388384: rejecting vote request since we have a leader: from=edcad3d0-f25a-de50-f0a3-9631d24ade7c leader=1ce75d95-a882-37e7-28a2-f0666e6928f2 leader-id=server-1ce75d95-a882-37e7-28a2-f0666e6928f2 2025-07-29T00:55:31.766Z [ERROR] server-edcad3d0-f25a-de50-f0a3-9631d24ade7c: failed to make requestVote RPC: target="{Voter server-1ce75d95-a882-37e7-28a2-f0666e6928f2 1ce75d95-a882-37e7-28a2-f0666e6928f2}" error="failed to connect to peer: 1ce75d95-a882-37e7-28a2-f0666e6928f2" term=3 2025-07-29T00:55:31.768Z [WARN] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to contact: server-id=server-edcad3d0-f25a-de50-f0a3-9631d24ade7c time=59.39604ms 2025-07-29T00:55:31.768Z [WARN] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to contact: server-id=server-72cc640c-7d68-24de-59ec-f45f39388384 time=60.69008ms 2025-07-29T00:55:31.768Z [WARN] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to contact quorum of nodes, stepping down 2025-07-29T00:55:31.768Z [INFO] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: entering follower state: follower="Node at 1ce75d95-a882-37e7-28a2-f0666e6928f2 [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.768Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-72cc640c-7d68-24de-59ec-f45f39388384 72cc640c-7d68-24de-59ec-f45f39388384}" error="failed to connect to peer: 72cc640c-7d68-24de-59ec-f45f39388384" 2025-07-29T00:55:31.768Z [ERROR] server-1ce75d95-a882-37e7-28a2-f0666e6928f2: failed to appendEntries to: peer="{Voter server-edcad3d0-f25a-de50-f0a3-9631d24ade7c edcad3d0-f25a-de50-f0a3-9631d24ade7c}" error="failed to connect to peer: edcad3d0-f25a-de50-f0a3-9631d24ade7c" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) === RUN TestRaft_AddKnownPeer 2025-07-29T00:55:31.769Z [INFO] server-f71a58bd-6531-ab14-f134-be92296fb5ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f71a58bd-6531-ab14-f134-be92296fb5ea Address:f71a58bd-6531-ab14-f134-be92296fb5ea} {Suffrage:Voter ID:server-e10776b3-9cba-a69e-1baf-8111fb4cffa1 Address:e10776b3-9cba-a69e-1baf-8111fb4cffa1} {Suffrage:Voter ID:server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba Address:c4e3699e-6b5d-f993-ed73-4d7cc00c9eba}]" 2025-07-29T00:55:31.769Z [INFO] server-f71a58bd-6531-ab14-f134-be92296fb5ea: entering follower state: follower="Node at f71a58bd-6531-ab14-f134-be92296fb5ea [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.769Z [INFO] server-e10776b3-9cba-a69e-1baf-8111fb4cffa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f71a58bd-6531-ab14-f134-be92296fb5ea Address:f71a58bd-6531-ab14-f134-be92296fb5ea} {Suffrage:Voter ID:server-e10776b3-9cba-a69e-1baf-8111fb4cffa1 Address:e10776b3-9cba-a69e-1baf-8111fb4cffa1} {Suffrage:Voter ID:server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba Address:c4e3699e-6b5d-f993-ed73-4d7cc00c9eba}]" 2025-07-29T00:55:31.769Z [INFO] server-e10776b3-9cba-a69e-1baf-8111fb4cffa1: entering follower state: follower="Node at e10776b3-9cba-a69e-1baf-8111fb4cffa1 [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.769Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f71a58bd-6531-ab14-f134-be92296fb5ea Address:f71a58bd-6531-ab14-f134-be92296fb5ea} {Suffrage:Voter ID:server-e10776b3-9cba-a69e-1baf-8111fb4cffa1 Address:e10776b3-9cba-a69e-1baf-8111fb4cffa1} {Suffrage:Voter ID:server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba Address:c4e3699e-6b5d-f993-ed73-4d7cc00c9eba}]" 2025-07-29T00:55:31.769Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:31.769Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: entering follower state: follower="Node at c4e3699e-6b5d-f993-ed73-4d7cc00c9eba [Follower]" leader-address= leader-id= 2025-07-29T00:55:31.837Z [WARN] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:31.837Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: entering candidate state: node="Node at c4e3699e-6b5d-f993-ed73-4d7cc00c9eba [Candidate]" term=2 2025-07-29T00:55:31.837Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: election won: term=2 tally=2 2025-07-29T00:55:31.837Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: entering leader state: leader="Node at c4e3699e-6b5d-f993-ed73-4d7cc00c9eba [Leader]" 2025-07-29T00:55:31.837Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: added peer, starting replication: peer=server-f71a58bd-6531-ab14-f134-be92296fb5ea 2025-07-29T00:55:31.837Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: added peer, starting replication: peer=server-e10776b3-9cba-a69e-1baf-8111fb4cffa1 2025-07-29T00:55:31.838Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: pipelining replication: peer="{Voter server-f71a58bd-6531-ab14-f134-be92296fb5ea f71a58bd-6531-ab14-f134-be92296fb5ea}" 2025-07-29T00:55:31.838Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: pipelining replication: peer="{Voter server-e10776b3-9cba-a69e-1baf-8111fb4cffa1 e10776b3-9cba-a69e-1baf-8111fb4cffa1}" 2025-07-29T00:55:31.943Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:31.837971217 +0000 UTC m=+11.527352561 (1 nodes), 68.20212ms from start of poll, 68.63748ms from cluster start. Timeout at 2025-07-29 00:55:31.943065297 +0000 UTC m=+11.632446641, 105.09408ms after stability 2025-07-29T00:55:31.943Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:32.048Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:31.943163137 +0000 UTC m=+11.632544481 (2 nodes), 3.44µs from start of poll, 173.8294ms from cluster start. Timeout at 2025-07-29 00:55:32.048184857 +0000 UTC m=+11.737566201, 105.02172ms after stability 2025-07-29T00:55:32.048Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: updating configuration: command=AddVoter server-id=server-f71a58bd-6531-ab14-f134-be92296fb5ea server-addr=f71a58bd-6531-ab14-f134-be92296fb5ea servers="[{Suffrage:Voter ID:server-f71a58bd-6531-ab14-f134-be92296fb5ea Address:f71a58bd-6531-ab14-f134-be92296fb5ea} {Suffrage:Voter ID:server-e10776b3-9cba-a69e-1baf-8111fb4cffa1 Address:e10776b3-9cba-a69e-1baf-8111fb4cffa1} {Suffrage:Voter ID:server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba Address:c4e3699e-6b5d-f993-ed73-4d7cc00c9eba}]" 2025-07-29T00:55:32.048Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: aborting pipeline replication: peer="{Voter server-e10776b3-9cba-a69e-1baf-8111fb4cffa1 e10776b3-9cba-a69e-1baf-8111fb4cffa1}" 2025-07-29T00:55:32.048Z [INFO] server-c4e3699e-6b5d-f993-ed73-4d7cc00c9eba: aborting pipeline replication: peer="{Voter server-f71a58bd-6531-ab14-f134-be92296fb5ea f71a58bd-6531-ab14-f134-be92296fb5ea}" --- PASS: TestRaft_AddKnownPeer (0.28s) === RUN TestRaft_RemoveUnknownPeer 2025-07-29T00:55:32.049Z [INFO] server-704e95b2-c813-8f25-def5-400ea0ce71c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-704e95b2-c813-8f25-def5-400ea0ce71c7 Address:704e95b2-c813-8f25-def5-400ea0ce71c7} {Suffrage:Voter ID:server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec Address:7d7b0074-35be-fc50-3fc7-d8aee4f372ec} {Suffrage:Voter ID:server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf Address:9bf420f0-a7fd-be47-ecdd-4f1be230ddcf}]" 2025-07-29T00:55:32.049Z [INFO] server-704e95b2-c813-8f25-def5-400ea0ce71c7: entering follower state: follower="Node at 704e95b2-c813-8f25-def5-400ea0ce71c7 [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.049Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-704e95b2-c813-8f25-def5-400ea0ce71c7 Address:704e95b2-c813-8f25-def5-400ea0ce71c7} {Suffrage:Voter ID:server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec Address:7d7b0074-35be-fc50-3fc7-d8aee4f372ec} {Suffrage:Voter ID:server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf Address:9bf420f0-a7fd-be47-ecdd-4f1be230ddcf}]" 2025-07-29T00:55:32.049Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: entering follower state: follower="Node at 7d7b0074-35be-fc50-3fc7-d8aee4f372ec [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.049Z [INFO] server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-704e95b2-c813-8f25-def5-400ea0ce71c7 Address:704e95b2-c813-8f25-def5-400ea0ce71c7} {Suffrage:Voter ID:server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec Address:7d7b0074-35be-fc50-3fc7-d8aee4f372ec} {Suffrage:Voter ID:server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf Address:9bf420f0-a7fd-be47-ecdd-4f1be230ddcf}]" 2025-07-29T00:55:32.049Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:32.049Z [INFO] server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf: entering follower state: follower="Node at 9bf420f0-a7fd-be47-ecdd-4f1be230ddcf [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.103Z [WARN] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:32.103Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: entering candidate state: node="Node at 7d7b0074-35be-fc50-3fc7-d8aee4f372ec [Candidate]" term=2 2025-07-29T00:55:32.103Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: election won: term=2 tally=2 2025-07-29T00:55:32.104Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: entering leader state: leader="Node at 7d7b0074-35be-fc50-3fc7-d8aee4f372ec [Leader]" 2025-07-29T00:55:32.104Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: added peer, starting replication: peer=server-704e95b2-c813-8f25-def5-400ea0ce71c7 2025-07-29T00:55:32.104Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: added peer, starting replication: peer=server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf 2025-07-29T00:55:32.104Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: pipelining replication: peer="{Voter server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf 9bf420f0-a7fd-be47-ecdd-4f1be230ddcf}" 2025-07-29T00:55:32.104Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: pipelining replication: peer="{Voter server-704e95b2-c813-8f25-def5-400ea0ce71c7 704e95b2-c813-8f25-def5-400ea0ce71c7}" 2025-07-29T00:55:32.209Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:32.104008017 +0000 UTC m=+11.793389361 (1 nodes), 54.1586ms from start of poll, 54.63724ms from cluster start. Timeout at 2025-07-29 00:55:32.209913417 +0000 UTC m=+11.899294761, 105.9054ms after stability 2025-07-29T00:55:32.210Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: updating configuration: command=RemoveServer server-id=776f95bb-ed8f-7a7f-c40c-3135c499e6c7 server-addr= servers="[{Suffrage:Voter ID:server-704e95b2-c813-8f25-def5-400ea0ce71c7 Address:704e95b2-c813-8f25-def5-400ea0ce71c7} {Suffrage:Voter ID:server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec Address:7d7b0074-35be-fc50-3fc7-d8aee4f372ec} {Suffrage:Voter ID:server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf Address:9bf420f0-a7fd-be47-ecdd-4f1be230ddcf}]" 2025-07-29T00:55:32.210Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: aborting pipeline replication: peer="{Voter server-9bf420f0-a7fd-be47-ecdd-4f1be230ddcf 9bf420f0-a7fd-be47-ecdd-4f1be230ddcf}" 2025-07-29T00:55:32.210Z [INFO] server-7d7b0074-35be-fc50-3fc7-d8aee4f372ec: aborting pipeline replication: peer="{Voter server-704e95b2-c813-8f25-def5-400ea0ce71c7 704e95b2-c813-8f25-def5-400ea0ce71c7}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore 2025-07-29T00:55:32.211Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5cf6307-ae6e-ea26-667d-812b93f8dcab Address:f5cf6307-ae6e-ea26-667d-812b93f8dcab}]" 2025-07-29T00:55:32.211Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:32.211Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: entering follower state: follower="Node at f5cf6307-ae6e-ea26-667d-812b93f8dcab [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.289Z [WARN] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:32.290Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: entering candidate state: node="Node at f5cf6307-ae6e-ea26-667d-812b93f8dcab [Candidate]" term=2 2025-07-29T00:55:32.290Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: election won: term=2 tally=1 2025-07-29T00:55:32.290Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: entering leader state: leader="Node at f5cf6307-ae6e-ea26-667d-812b93f8dcab [Leader]" 2025-07-29T00:55:32.395Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:32.290131817 +0000 UTC m=+11.979513161 (1 nodes), 78.96004ms from start of poll, 79.15828ms from cluster start. Timeout at 2025-07-29 00:55:32.395951217 +0000 UTC m=+12.085332601, 105.81944ms after stability 2025-07-29T00:55:32.398Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: starting snapshot up to: index=102 2025-07-29T00:55:32.399Z [INFO] creating new snapshot: path=/tmp/raft773507222/snapshots/2-102-1753750532398.tmp 2025-07-29T00:55:32.399Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: compacting logs: from=1 to=92 2025-07-29T00:55:32.399Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: snapshot complete up to: index=102 2025-07-29T00:55:32.399Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: starting restore from snapshot: id=2-102-1753750532398 last-index=102 last-term=2 size-in-bytes=693 2025-07-29T00:55:32.400Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: snapshot restore progress: id=2-102-1753750532398 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-07-29T00:55:32.400Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: restored from snapshot: id=2-102-1753750532398 last-index=102 last-term=2 size-in-bytes=693 2025-07-29T00:55:32.400Z [INFO] server-f5cf6307-ae6e-ea26-667d-812b93f8dcab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5cf6307-ae6e-ea26-667d-812b93f8dcab Address:f5cf6307-ae6e-ea26-667d-812b93f8dcab}]" --- PASS: TestRaft_SnapshotRestore (0.19s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2025-07-29T00:55:32.400Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f7fa864-66ef-77de-648e-071818c70cee Address:3f7fa864-66ef-77de-648e-071818c70cee}]" 2025-07-29T00:55:32.400Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:32.400Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: entering follower state: follower="Node at 3f7fa864-66ef-77de-648e-071818c70cee [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.460Z [WARN] server-3f7fa864-66ef-77de-648e-071818c70cee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:32.460Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: entering candidate state: node="Node at 3f7fa864-66ef-77de-648e-071818c70cee [Candidate]" term=2 2025-07-29T00:55:32.460Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: election won: term=2 tally=1 2025-07-29T00:55:32.460Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: entering leader state: leader="Node at 3f7fa864-66ef-77de-648e-071818c70cee [Leader]" 2025-07-29T00:55:32.566Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:32.460340297 +0000 UTC m=+12.149721641 (1 nodes), 59.48484ms from start of poll, 59.67384ms from cluster start. Timeout at 2025-07-29 00:55:32.566237937 +0000 UTC m=+12.255619281, 105.89764ms after stability 2025-07-29T00:55:32.569Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: starting snapshot up to: index=102 2025-07-29T00:55:32.569Z [INFO] creating new snapshot: path=/tmp/raft4192711528/snapshots/2-102-1753750532569.tmp 2025-07-29T00:55:32.569Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: compacting logs: from=1 to=92 2025-07-29T00:55:32.569Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: snapshot complete up to: index=102 2025-07-29T00:55:32.570Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: starting restore from snapshot: id=2-102-1753750532569 last-index=102 last-term=2 size-in-bytes=693 2025-07-29T00:55:32.570Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: snapshot restore progress: id=2-102-1753750532569 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-07-29T00:55:32.570Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: restored from snapshot: id=2-102-1753750532569 last-index=102 last-term=2 size-in-bytes=693 2025-07-29T00:55:32.570Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f7fa864-66ef-77de-648e-071818c70cee Address:3f7fa864-66ef-77de-648e-071818c70cee}]" 2025-07-29T00:55:32.570Z [INFO] server-3f7fa864-66ef-77de-648e-071818c70cee: entering follower state: follower="Node at 3f7fa864-66ef-77de-648e-071818c70cee [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.17s) === RUN TestRaft_SnapshotRestore_Progress 2025-07-29T00:55:32.572Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-164d9231-34a6-0817-b3e1-a89c61baf01e Address:164d9231-34a6-0817-b3e1-a89c61baf01e}]" 2025-07-29T00:55:32.572Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:32.572Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: entering follower state: follower="Node at 164d9231-34a6-0817-b3e1-a89c61baf01e [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.646Z [WARN] server-164d9231-34a6-0817-b3e1-a89c61baf01e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:32.646Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: entering candidate state: node="Node at 164d9231-34a6-0817-b3e1-a89c61baf01e [Candidate]" term=2 2025-07-29T00:55:32.646Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: election won: term=2 tally=1 2025-07-29T00:55:32.646Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: entering leader state: leader="Node at 164d9231-34a6-0817-b3e1-a89c61baf01e [Leader]" 2025-07-29T00:55:32.752Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:32.646913537 +0000 UTC m=+12.336294881 (1 nodes), 74.60036ms from start of poll, 74.83712ms from cluster start. Timeout at 2025-07-29 00:55:32.751987737 +0000 UTC m=+12.441369081, 105.0742ms after stability 2025-07-29T00:55:32.754Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: starting snapshot up to: index=102 2025-07-29T00:55:32.754Z [INFO] creating new snapshot: path=/tmp/raft2647896342/snapshots/2-102-1753750532754.tmp 2025-07-29T00:55:32.755Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: compacting logs: from=1 to=92 2025-07-29T00:55:32.755Z [INFO] server-164d9231-34a6-0817-b3e1-a89c61baf01e: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.18s) === RUN TestRaft_NoRestoreOnStart 2025-07-29T00:55:32.756Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b871daf4-1a9c-ab3e-50f5-0937d6852d52 Address:b871daf4-1a9c-ab3e-50f5-0937d6852d52}]" 2025-07-29T00:55:32.756Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:32.756Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: entering follower state: follower="Node at b871daf4-1a9c-ab3e-50f5-0937d6852d52 [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.809Z [WARN] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:32.809Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: entering candidate state: node="Node at b871daf4-1a9c-ab3e-50f5-0937d6852d52 [Candidate]" term=2 2025-07-29T00:55:32.809Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: election won: term=2 tally=1 2025-07-29T00:55:32.809Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: entering leader state: leader="Node at b871daf4-1a9c-ab3e-50f5-0937d6852d52 [Leader]" 2025-07-29T00:55:32.915Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:32.809872657 +0000 UTC m=+12.499254001 (1 nodes), 53.07272ms from start of poll, 53.2632ms from cluster start. Timeout at 2025-07-29 00:55:32.915187217 +0000 UTC m=+12.604568601, 105.3146ms after stability 2025-07-29T00:55:32.918Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: starting snapshot up to: index=102 2025-07-29T00:55:32.918Z [INFO] creating new snapshot: path=/tmp/raft3385971669/snapshots/2-102-1753750532918.tmp 2025-07-29T00:55:32.918Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: compacting logs: from=1 to=92 2025-07-29T00:55:32.918Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: snapshot complete up to: index=102 2025-07-29T00:55:32.918Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b871daf4-1a9c-ab3e-50f5-0937d6852d52 Address:b871daf4-1a9c-ab3e-50f5-0937d6852d52}]" --- PASS: TestRaft_NoRestoreOnStart (0.16s) === RUN TestRaft_SnapshotRestore_PeerChange 2025-07-29T00:55:32.918Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: entering follower state: follower="Node at b871daf4-1a9c-ab3e-50f5-0937d6852d52 [Follower]" leader-address= leader-id= raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2025-07-29T00:55:32.919Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8568ba25-7a05-611a-0114-af64b33674ef Address:8568ba25-7a05-611a-0114-af64b33674ef}]" 2025-07-29T00:55:32.919Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:32.919Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: entering follower state: follower="Node at 8568ba25-7a05-611a-0114-af64b33674ef [Follower]" leader-address= leader-id= 2025-07-29T00:55:32.976Z [WARN] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:32.976Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: entering candidate state: node="Node at b871daf4-1a9c-ab3e-50f5-0937d6852d52 [Candidate]" term=3 2025-07-29T00:55:32.976Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: election won: term=3 tally=1 2025-07-29T00:55:32.976Z [INFO] server-b871daf4-1a9c-ab3e-50f5-0937d6852d52: entering leader state: leader="Node at b871daf4-1a9c-ab3e-50f5-0937d6852d52 [Leader]" 2025-07-29T00:55:33.017Z [WARN] server-8568ba25-7a05-611a-0114-af64b33674ef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:33.017Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: entering candidate state: node="Node at 8568ba25-7a05-611a-0114-af64b33674ef [Candidate]" term=2 2025-07-29T00:55:33.017Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: election won: term=2 tally=1 2025-07-29T00:55:33.017Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: entering leader state: leader="Node at 8568ba25-7a05-611a-0114-af64b33674ef [Leader]" 2025-07-29T00:55:33.123Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:33.017927817 +0000 UTC m=+12.707309161 (1 nodes), 98.43812ms from start of poll, 98.6588ms from cluster start. Timeout at 2025-07-29 00:55:33.123639857 +0000 UTC m=+12.813021201, 105.71204ms after stability 2025-07-29T00:55:33.139Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: starting snapshot up to: index=102 2025-07-29T00:55:33.139Z [INFO] creating new snapshot: path=/tmp/raft150161715/snapshots/2-102-1753750533139.tmp 2025-07-29T00:55:33.139Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: compacting logs: from=1 to=92 2025-07-29T00:55:33.139Z [INFO] server-8568ba25-7a05-611a-0114-af64b33674ef: snapshot complete up to: index=102 --- PASS: TestRaft_AutoSnapshot (0.31s) === RUN TestRaft_UserSnapshot 2025-07-29T00:55:33.234Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da59bd28-4bbe-20a9-a617-08d8a801b831 Address:da59bd28-4bbe-20a9-a617-08d8a801b831}]" 2025-07-29T00:55:33.234Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:33.234Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: entering follower state: follower="Node at da59bd28-4bbe-20a9-a617-08d8a801b831 [Follower]" leader-address= leader-id= 2025-07-29T00:55:33.286Z [WARN] server-da59bd28-4bbe-20a9-a617-08d8a801b831: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:33.286Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: entering candidate state: node="Node at da59bd28-4bbe-20a9-a617-08d8a801b831 [Candidate]" term=2 2025-07-29T00:55:33.286Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: election won: term=2 tally=1 2025-07-29T00:55:33.286Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: entering leader state: leader="Node at da59bd28-4bbe-20a9-a617-08d8a801b831 [Leader]" 2025-07-29T00:55:33.392Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:33.286515417 +0000 UTC m=+12.975896761 (1 nodes), 52.27504ms from start of poll, 52.49372ms from cluster start. Timeout at 2025-07-29 00:55:33.392220737 +0000 UTC m=+13.081602081, 105.70532ms after stability 2025-07-29T00:55:33.392Z [ERROR] server-da59bd28-4bbe-20a9-a617-08d8a801b831: failed to take snapshot: error="nothing new to snapshot" 2025-07-29T00:55:33.392Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: starting snapshot up to: index=12 2025-07-29T00:55:33.392Z [INFO] creating new snapshot: path=/tmp/raft3192677071/snapshots/2-12-1753750533392.tmp 2025-07-29T00:55:33.393Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: compacting logs: from=1 to=2 2025-07-29T00:55:33.393Z [INFO] server-da59bd28-4bbe-20a9-a617-08d8a801b831: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.16s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2025-07-29T00:55:33.394Z [INFO] server-ef067dc3-7cea-802f-7df4-05da36928498: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef067dc3-7cea-802f-7df4-05da36928498 Address:ef067dc3-7cea-802f-7df4-05da36928498} {Suffrage:Voter ID:server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd Address:69dd8592-89d2-7181-9b86-cbf4f36ee2fd} {Suffrage:Voter ID:server-54b68123-446c-923c-3910-d69b588ed3ea Address:54b68123-446c-923c-3910-d69b588ed3ea}]" 2025-07-29T00:55:33.394Z [INFO] server-ef067dc3-7cea-802f-7df4-05da36928498: entering follower state: follower="Node at ef067dc3-7cea-802f-7df4-05da36928498 [Follower]" leader-address= leader-id= 2025-07-29T00:55:33.394Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef067dc3-7cea-802f-7df4-05da36928498 Address:ef067dc3-7cea-802f-7df4-05da36928498} {Suffrage:Voter ID:server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd Address:69dd8592-89d2-7181-9b86-cbf4f36ee2fd} {Suffrage:Voter ID:server-54b68123-446c-923c-3910-d69b588ed3ea Address:54b68123-446c-923c-3910-d69b588ed3ea}]" 2025-07-29T00:55:33.394Z [INFO] server-54b68123-446c-923c-3910-d69b588ed3ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef067dc3-7cea-802f-7df4-05da36928498 Address:ef067dc3-7cea-802f-7df4-05da36928498} {Suffrage:Voter ID:server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd Address:69dd8592-89d2-7181-9b86-cbf4f36ee2fd} {Suffrage:Voter ID:server-54b68123-446c-923c-3910-d69b588ed3ea Address:54b68123-446c-923c-3910-d69b588ed3ea}]" 2025-07-29T00:55:33.394Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: entering follower state: follower="Node at 69dd8592-89d2-7181-9b86-cbf4f36ee2fd [Follower]" leader-address= leader-id= 2025-07-29T00:55:33.394Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:33.394Z [INFO] server-54b68123-446c-923c-3910-d69b588ed3ea: entering follower state: follower="Node at 54b68123-446c-923c-3910-d69b588ed3ea [Follower]" leader-address= leader-id= 2025-07-29T00:55:34.111Z [WARN] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:34.111Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: entering candidate state: node="Node at 69dd8592-89d2-7181-9b86-cbf4f36ee2fd [Candidate]" term=2 2025-07-29T00:55:34.111Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: election won: term=2 tally=2 2025-07-29T00:55:34.112Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: entering leader state: leader="Node at 69dd8592-89d2-7181-9b86-cbf4f36ee2fd [Leader]" 2025-07-29T00:55:34.112Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: added peer, starting replication: peer=server-ef067dc3-7cea-802f-7df4-05da36928498 2025-07-29T00:55:34.112Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: added peer, starting replication: peer=server-54b68123-446c-923c-3910-d69b588ed3ea 2025-07-29T00:55:34.112Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: pipelining replication: peer="{Voter server-ef067dc3-7cea-802f-7df4-05da36928498 ef067dc3-7cea-802f-7df4-05da36928498}" 2025-07-29T00:55:34.112Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: pipelining replication: peer="{Voter server-54b68123-446c-923c-3910-d69b588ed3ea 54b68123-446c-923c-3910-d69b588ed3ea}" 2025-07-29T00:55:35.117Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:34.112110017 +0000 UTC m=+13.801491361 (1 nodes), 717.66112ms from start of poll, 718.17008ms from cluster start. Timeout at 2025-07-29 00:55:35.117898617 +0000 UTC m=+14.807279961, 1.0057886s after stability 2025-07-29T00:55:35.118Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: starting snapshot up to: index=12 2025-07-29T00:55:35.118Z [INFO] creating new snapshot: path=/tmp/raft680183038/snapshots/2-12-1753750535118.tmp 2025-07-29T00:55:35.119Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: snapshot complete up to: index=12 2025-07-29T00:55:35.120Z [INFO] creating new snapshot: path=/tmp/raft680183038/snapshots/2-23-1753750535120.tmp 2025-07-29T00:55:35.120Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: copied to local snapshot: bytes=71 2025-07-29T00:55:35.120Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: snapshot restore progress: id=2-23-1753750535120 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:35.120Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: restored user snapshot: index=23 2025-07-29T00:55:35.120Z [ERROR] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: failed to get log: index=23 error="log not found" 2025-07-29T00:55:35.120Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: aborting pipeline replication: peer="{Voter server-54b68123-446c-923c-3910-d69b588ed3ea 54b68123-446c-923c-3910-d69b588ed3ea}" 2025-07-29T00:55:35.120Z [ERROR] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: failed to get log: index=23 error="log not found" 2025-07-29T00:55:35.121Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: aborting pipeline replication: peer="{Voter server-ef067dc3-7cea-802f-7df4-05da36928498 ef067dc3-7cea-802f-7df4-05da36928498}" 2025-07-29T00:55:35.127Z [ERROR] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: failed to get log: index=23 error="log not found" 2025-07-29T00:55:35.128Z [INFO] creating new snapshot: path=/tmp/raft1934065586/snapshots/2-23-1753750535128.tmp 2025-07-29T00:55:35.128Z [INFO] server-54b68123-446c-923c-3910-d69b588ed3ea: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:35.128Z [INFO] server-54b68123-446c-923c-3910-d69b588ed3ea: copied to local snapshot: bytes=71 2025-07-29T00:55:35.128Z [INFO] server-54b68123-446c-923c-3910-d69b588ed3ea: snapshot restore progress: id=2-23-1753750535128 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:35.128Z [INFO] server-54b68123-446c-923c-3910-d69b588ed3ea: Installed remote snapshot 2025-07-29T00:55:35.128Z [WARN] server-54b68123-446c-923c-3910-d69b588ed3ea: skipping application of old log: index=22 2025-07-29T00:55:35.129Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: pipelining replication: peer="{Voter server-54b68123-446c-923c-3910-d69b588ed3ea 54b68123-446c-923c-3910-d69b588ed3ea}" 2025-07-29T00:55:35.129Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:35.130Z [ERROR] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: failed to get log: index=23 error="log not found" 2025-07-29T00:55:35.131Z [INFO] creating new snapshot: path=/tmp/raft29597861/snapshots/2-23-1753750535131.tmp 2025-07-29T00:55:35.131Z [INFO] server-ef067dc3-7cea-802f-7df4-05da36928498: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:35.131Z [INFO] server-ef067dc3-7cea-802f-7df4-05da36928498: copied to local snapshot: bytes=71 2025-07-29T00:55:35.132Z [INFO] server-ef067dc3-7cea-802f-7df4-05da36928498: snapshot restore progress: id=2-23-1753750535131 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:35.132Z [INFO] server-ef067dc3-7cea-802f-7df4-05da36928498: Installed remote snapshot 2025-07-29T00:55:35.132Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: pipelining replication: peer="{Voter server-ef067dc3-7cea-802f-7df4-05da36928498 ef067dc3-7cea-802f-7df4-05da36928498}" 2025-07-29T00:55:36.134Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:35.129113617 +0000 UTC m=+14.818494961 (2 nodes), 1.08µs from start of poll, 1.73517368s from cluster start. Timeout at 2025-07-29 00:55:36.134602017 +0000 UTC m=+15.823983361, 1.0054884s after stability 2025-07-29T00:55:36.147Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: aborting pipeline replication: peer="{Voter server-54b68123-446c-923c-3910-d69b588ed3ea 54b68123-446c-923c-3910-d69b588ed3ea}" 2025-07-29T00:55:36.147Z [INFO] server-69dd8592-89d2-7181-9b86-cbf4f36ee2fd: aborting pipeline replication: peer="{Voter server-ef067dc3-7cea-802f-7df4-05da36928498 ef067dc3-7cea-802f-7df4-05da36928498}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2025-07-29T00:55:36.148Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94753750-9e39-7d6e-7726-b8e8e36ab429 Address:94753750-9e39-7d6e-7726-b8e8e36ab429} {Suffrage:Voter ID:server-e3405e20-90ab-9d1b-22a7-37d6b458c23c Address:e3405e20-90ab-9d1b-22a7-37d6b458c23c} {Suffrage:Voter ID:server-c3eec121-7ed8-83f4-9707-987e8cbb351f Address:c3eec121-7ed8-83f4-9707-987e8cbb351f}]" 2025-07-29T00:55:36.148Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: entering follower state: follower="Node at 94753750-9e39-7d6e-7726-b8e8e36ab429 [Follower]" leader-address= leader-id= 2025-07-29T00:55:36.148Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94753750-9e39-7d6e-7726-b8e8e36ab429 Address:94753750-9e39-7d6e-7726-b8e8e36ab429} {Suffrage:Voter ID:server-e3405e20-90ab-9d1b-22a7-37d6b458c23c Address:e3405e20-90ab-9d1b-22a7-37d6b458c23c} {Suffrage:Voter ID:server-c3eec121-7ed8-83f4-9707-987e8cbb351f Address:c3eec121-7ed8-83f4-9707-987e8cbb351f}]" 2025-07-29T00:55:36.148Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: entering follower state: follower="Node at e3405e20-90ab-9d1b-22a7-37d6b458c23c [Follower]" leader-address= leader-id= 2025-07-29T00:55:36.149Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94753750-9e39-7d6e-7726-b8e8e36ab429 Address:94753750-9e39-7d6e-7726-b8e8e36ab429} {Suffrage:Voter ID:server-e3405e20-90ab-9d1b-22a7-37d6b458c23c Address:e3405e20-90ab-9d1b-22a7-37d6b458c23c} {Suffrage:Voter ID:server-c3eec121-7ed8-83f4-9707-987e8cbb351f Address:c3eec121-7ed8-83f4-9707-987e8cbb351f}]" 2025-07-29T00:55:36.149Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:36.149Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: entering follower state: follower="Node at c3eec121-7ed8-83f4-9707-987e8cbb351f [Follower]" leader-address= leader-id= 2025-07-29T00:55:36.859Z [WARN] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:36.859Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: entering candidate state: node="Node at e3405e20-90ab-9d1b-22a7-37d6b458c23c [Candidate]" term=2 2025-07-29T00:55:36.859Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: election won: term=2 tally=2 2025-07-29T00:55:36.859Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: entering leader state: leader="Node at e3405e20-90ab-9d1b-22a7-37d6b458c23c [Leader]" 2025-07-29T00:55:36.859Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: added peer, starting replication: peer=server-94753750-9e39-7d6e-7726-b8e8e36ab429 2025-07-29T00:55:36.859Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: added peer, starting replication: peer=server-c3eec121-7ed8-83f4-9707-987e8cbb351f 2025-07-29T00:55:36.859Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: pipelining replication: peer="{Voter server-c3eec121-7ed8-83f4-9707-987e8cbb351f c3eec121-7ed8-83f4-9707-987e8cbb351f}" 2025-07-29T00:55:36.859Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: pipelining replication: peer="{Voter server-94753750-9e39-7d6e-7726-b8e8e36ab429 94753750-9e39-7d6e-7726-b8e8e36ab429}" 2025-07-29T00:55:37.864Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:36.859243377 +0000 UTC m=+16.548624721 (1 nodes), 710.11352ms from start of poll, 710.74584ms from cluster start. Timeout at 2025-07-29 00:55:37.864273897 +0000 UTC m=+17.553655241, 1.00503052s after stability 2025-07-29T00:55:37.864Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: starting snapshot up to: index=12 2025-07-29T00:55:37.864Z [INFO] creating new snapshot: path=/tmp/raft2223549647/snapshots/2-12-1753750537864.tmp 2025-07-29T00:55:37.865Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: snapshot complete up to: index=12 2025-07-29T00:55:37.866Z [INFO] creating new snapshot: path=/tmp/raft2223549647/snapshots/2-23-1753750537866.tmp 2025-07-29T00:55:37.866Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: copied to local snapshot: bytes=71 2025-07-29T00:55:37.867Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: snapshot restore progress: id=2-23-1753750537866 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:37.867Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: removing all old logs from log store 2025-07-29T00:55:37.867Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: compacting logs: from=1 to=22 2025-07-29T00:55:37.867Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: restored user snapshot: index=23 2025-07-29T00:55:37.867Z [ERROR] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: failed to get log: index=22 error="log not found" 2025-07-29T00:55:37.867Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: aborting pipeline replication: peer="{Voter server-c3eec121-7ed8-83f4-9707-987e8cbb351f c3eec121-7ed8-83f4-9707-987e8cbb351f}" 2025-07-29T00:55:37.867Z [ERROR] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: failed to get log: index=22 error="log not found" 2025-07-29T00:55:37.867Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: aborting pipeline replication: peer="{Voter server-94753750-9e39-7d6e-7726-b8e8e36ab429 94753750-9e39-7d6e-7726-b8e8e36ab429}" 2025-07-29T00:55:37.874Z [ERROR] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: failed to get log: index=22 error="log not found" 2025-07-29T00:55:37.874Z [INFO] creating new snapshot: path=/tmp/raft594316155/snapshots/2-23-1753750537874.tmp 2025-07-29T00:55:37.875Z [ERROR] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: failed to get log: index=22 error="log not found" 2025-07-29T00:55:37.875Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:37.875Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: copied to local snapshot: bytes=71 2025-07-29T00:55:37.875Z [INFO] creating new snapshot: path=/tmp/raft2809403155/snapshots/2-23-1753750537875.tmp 2025-07-29T00:55:37.876Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:37.876Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: snapshot restore progress: id=2-23-1753750537874 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:37.876Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: removing all old logs from log store 2025-07-29T00:55:37.876Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: compacting logs: from=1 to=22 2025-07-29T00:55:37.876Z [INFO] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: Installed remote snapshot 2025-07-29T00:55:37.876Z [WARN] server-c3eec121-7ed8-83f4-9707-987e8cbb351f: skipping application of old log: index=22 2025-07-29T00:55:37.876Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: pipelining replication: peer="{Voter server-c3eec121-7ed8-83f4-9707-987e8cbb351f c3eec121-7ed8-83f4-9707-987e8cbb351f}" 2025-07-29T00:55:37.876Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: copied to local snapshot: bytes=71 2025-07-29T00:55:37.876Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:37.877Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: snapshot restore progress: id=2-23-1753750537875 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:37.877Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: removing all old logs from log store 2025-07-29T00:55:37.877Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: compacting logs: from=1 to=22 2025-07-29T00:55:37.877Z [INFO] server-94753750-9e39-7d6e-7726-b8e8e36ab429: Installed remote snapshot 2025-07-29T00:55:37.877Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: pipelining replication: peer="{Voter server-94753750-9e39-7d6e-7726-b8e8e36ab429 94753750-9e39-7d6e-7726-b8e8e36ab429}" 2025-07-29T00:55:38.881Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:37.876839577 +0000 UTC m=+17.566220961 (2 nodes), 1.88µs from start of poll, 1.72834208s from cluster start. Timeout at 2025-07-29 00:55:38.881910817 +0000 UTC m=+18.571292161, 1.0050712s after stability 2025-07-29T00:55:38.894Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: aborting pipeline replication: peer="{Voter server-c3eec121-7ed8-83f4-9707-987e8cbb351f c3eec121-7ed8-83f4-9707-987e8cbb351f}" 2025-07-29T00:55:38.894Z [INFO] server-e3405e20-90ab-9d1b-22a7-37d6b458c23c: aborting pipeline replication: peer="{Voter server-94753750-9e39-7d6e-7726-b8e8e36ab429 94753750-9e39-7d6e-7726-b8e8e36ab429}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2025-07-29T00:55:38.895Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3a22394-28d3-0f04-daf6-379e60eef4da Address:a3a22394-28d3-0f04-daf6-379e60eef4da} {Suffrage:Voter ID:server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8 Address:d21d4e71-1dbc-87eb-5596-cd09a85afbb8} {Suffrage:Voter ID:server-05652cc9-0056-e382-2a0b-3120d0ac3049 Address:05652cc9-0056-e382-2a0b-3120d0ac3049}]" 2025-07-29T00:55:38.895Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: entering follower state: follower="Node at a3a22394-28d3-0f04-daf6-379e60eef4da [Follower]" leader-address= leader-id= 2025-07-29T00:55:38.895Z [INFO] server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3a22394-28d3-0f04-daf6-379e60eef4da Address:a3a22394-28d3-0f04-daf6-379e60eef4da} {Suffrage:Voter ID:server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8 Address:d21d4e71-1dbc-87eb-5596-cd09a85afbb8} {Suffrage:Voter ID:server-05652cc9-0056-e382-2a0b-3120d0ac3049 Address:05652cc9-0056-e382-2a0b-3120d0ac3049}]" 2025-07-29T00:55:38.895Z [INFO] server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8: entering follower state: follower="Node at d21d4e71-1dbc-87eb-5596-cd09a85afbb8 [Follower]" leader-address= leader-id= 2025-07-29T00:55:38.895Z [INFO] server-05652cc9-0056-e382-2a0b-3120d0ac3049: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3a22394-28d3-0f04-daf6-379e60eef4da Address:a3a22394-28d3-0f04-daf6-379e60eef4da} {Suffrage:Voter ID:server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8 Address:d21d4e71-1dbc-87eb-5596-cd09a85afbb8} {Suffrage:Voter ID:server-05652cc9-0056-e382-2a0b-3120d0ac3049 Address:05652cc9-0056-e382-2a0b-3120d0ac3049}]" 2025-07-29T00:55:38.895Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:38.895Z [INFO] server-05652cc9-0056-e382-2a0b-3120d0ac3049: entering follower state: follower="Node at 05652cc9-0056-e382-2a0b-3120d0ac3049 [Follower]" leader-address= leader-id= 2025-07-29T00:55:39.412Z [WARN] server-a3a22394-28d3-0f04-daf6-379e60eef4da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:39.412Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: entering candidate state: node="Node at a3a22394-28d3-0f04-daf6-379e60eef4da [Candidate]" term=2 2025-07-29T00:55:39.412Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: election won: term=2 tally=2 2025-07-29T00:55:39.412Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: entering leader state: leader="Node at a3a22394-28d3-0f04-daf6-379e60eef4da [Leader]" 2025-07-29T00:55:39.412Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: added peer, starting replication: peer=server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8 2025-07-29T00:55:39.412Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: added peer, starting replication: peer=server-05652cc9-0056-e382-2a0b-3120d0ac3049 2025-07-29T00:55:39.413Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: pipelining replication: peer="{Voter server-05652cc9-0056-e382-2a0b-3120d0ac3049 05652cc9-0056-e382-2a0b-3120d0ac3049}" 2025-07-29T00:55:39.413Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: pipelining replication: peer="{Voter server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8 d21d4e71-1dbc-87eb-5596-cd09a85afbb8}" 2025-07-29T00:55:40.418Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:39.412899777 +0000 UTC m=+19.102281121 (1 nodes), 517.30268ms from start of poll, 517.7842ms from cluster start. Timeout at 2025-07-29 00:55:40.418295697 +0000 UTC m=+20.107677041, 1.00539592s after stability 2025-07-29T00:55:40.418Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: starting snapshot up to: index=12 2025-07-29T00:55:40.419Z [INFO] creating new snapshot: path=/tmp/raft3357805336/snapshots/2-12-1753750540419.tmp 2025-07-29T00:55:40.419Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: snapshot complete up to: index=12 2025-07-29T00:55:40.421Z [INFO] server-e7ab1cfc-5fda-1d14-1367-35fcf53af884: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7ab1cfc-5fda-1d14-1367-35fcf53af884 Address:e7ab1cfc-5fda-1d14-1367-35fcf53af884} {Suffrage:Voter ID:server-e6220984-f6c8-f7f0-1653-3849643467ac Address:e6220984-f6c8-f7f0-1653-3849643467ac} {Suffrage:Voter ID:server-6206c28a-dc2e-3d13-7396-04ec35301740 Address:6206c28a-dc2e-3d13-7396-04ec35301740}]" 2025-07-29T00:55:40.421Z [INFO] server-e7ab1cfc-5fda-1d14-1367-35fcf53af884: entering follower state: follower="Node at e7ab1cfc-5fda-1d14-1367-35fcf53af884 [Follower]" leader-address= leader-id= 2025-07-29T00:55:40.421Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7ab1cfc-5fda-1d14-1367-35fcf53af884 Address:e7ab1cfc-5fda-1d14-1367-35fcf53af884} {Suffrage:Voter ID:server-e6220984-f6c8-f7f0-1653-3849643467ac Address:e6220984-f6c8-f7f0-1653-3849643467ac} {Suffrage:Voter ID:server-6206c28a-dc2e-3d13-7396-04ec35301740 Address:6206c28a-dc2e-3d13-7396-04ec35301740}]" 2025-07-29T00:55:40.421Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: entering follower state: follower="Node at e6220984-f6c8-f7f0-1653-3849643467ac [Follower]" leader-address= leader-id= 2025-07-29T00:55:40.421Z [INFO] server-6206c28a-dc2e-3d13-7396-04ec35301740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7ab1cfc-5fda-1d14-1367-35fcf53af884 Address:e7ab1cfc-5fda-1d14-1367-35fcf53af884} {Suffrage:Voter ID:server-e6220984-f6c8-f7f0-1653-3849643467ac Address:e6220984-f6c8-f7f0-1653-3849643467ac} {Suffrage:Voter ID:server-6206c28a-dc2e-3d13-7396-04ec35301740 Address:6206c28a-dc2e-3d13-7396-04ec35301740}]" 2025-07-29T00:55:40.422Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:40.422Z [INFO] server-6206c28a-dc2e-3d13-7396-04ec35301740: entering follower state: follower="Node at 6206c28a-dc2e-3d13-7396-04ec35301740 [Follower]" leader-address= leader-id= 2025-07-29T00:55:41.111Z [WARN] server-e6220984-f6c8-f7f0-1653-3849643467ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:41.111Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: entering candidate state: node="Node at e6220984-f6c8-f7f0-1653-3849643467ac [Candidate]" term=2 2025-07-29T00:55:41.112Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: election won: term=2 tally=2 2025-07-29T00:55:41.112Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: entering leader state: leader="Node at e6220984-f6c8-f7f0-1653-3849643467ac [Leader]" 2025-07-29T00:55:41.112Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: added peer, starting replication: peer=server-e7ab1cfc-5fda-1d14-1367-35fcf53af884 2025-07-29T00:55:41.112Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: added peer, starting replication: peer=server-6206c28a-dc2e-3d13-7396-04ec35301740 2025-07-29T00:55:41.112Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: pipelining replication: peer="{Voter server-6206c28a-dc2e-3d13-7396-04ec35301740 6206c28a-dc2e-3d13-7396-04ec35301740}" 2025-07-29T00:55:41.112Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: pipelining replication: peer="{Voter server-e7ab1cfc-5fda-1d14-1367-35fcf53af884 e7ab1cfc-5fda-1d14-1367-35fcf53af884}" 2025-07-29T00:55:42.118Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:41.112131777 +0000 UTC m=+20.801513121 (1 nodes), 690.1046ms from start of poll, 690.64444ms from cluster start. Timeout at 2025-07-29 00:55:42.118016737 +0000 UTC m=+21.807398121, 1.005885s after stability 2025-07-29T00:55:42.118Z [INFO] creating new snapshot: path=/tmp/raft2978145738/snapshots/2-13-1753750542118.tmp 2025-07-29T00:55:42.118Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: copied to local snapshot: bytes=71 2025-07-29T00:55:42.119Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: snapshot restore progress: id=2-13-1753750542118 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:42.119Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: restored user snapshot: index=13 2025-07-29T00:55:42.119Z [ERROR] server-e6220984-f6c8-f7f0-1653-3849643467ac: failed to get log: index=3 error="log not found" 2025-07-29T00:55:42.119Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: aborting pipeline replication: peer="{Voter server-6206c28a-dc2e-3d13-7396-04ec35301740 6206c28a-dc2e-3d13-7396-04ec35301740}" 2025-07-29T00:55:42.119Z [ERROR] server-e6220984-f6c8-f7f0-1653-3849643467ac: failed to get log: index=3 error="log not found" 2025-07-29T00:55:42.119Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: aborting pipeline replication: peer="{Voter server-e7ab1cfc-5fda-1d14-1367-35fcf53af884 e7ab1cfc-5fda-1d14-1367-35fcf53af884}" 2025-07-29T00:55:42.129Z [ERROR] server-e6220984-f6c8-f7f0-1653-3849643467ac: failed to get log: index=3 error="log not found" 2025-07-29T00:55:42.129Z [INFO] creating new snapshot: path=/tmp/raft4002129932/snapshots/2-13-1753750542129.tmp 2025-07-29T00:55:42.129Z [ERROR] server-e6220984-f6c8-f7f0-1653-3849643467ac: failed to get log: index=3 error="log not found" 2025-07-29T00:55:42.129Z [INFO] server-6206c28a-dc2e-3d13-7396-04ec35301740: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:42.130Z [INFO] creating new snapshot: path=/tmp/raft2478602867/snapshots/2-13-1753750542130.tmp 2025-07-29T00:55:42.130Z [INFO] server-6206c28a-dc2e-3d13-7396-04ec35301740: copied to local snapshot: bytes=71 2025-07-29T00:55:42.130Z [INFO] server-e7ab1cfc-5fda-1d14-1367-35fcf53af884: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:42.130Z [INFO] server-6206c28a-dc2e-3d13-7396-04ec35301740: snapshot restore progress: id=2-13-1753750542129 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:42.130Z [INFO] server-6206c28a-dc2e-3d13-7396-04ec35301740: Installed remote snapshot 2025-07-29T00:55:42.130Z [INFO] server-e7ab1cfc-5fda-1d14-1367-35fcf53af884: copied to local snapshot: bytes=71 2025-07-29T00:55:42.130Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: pipelining replication: peer="{Voter server-6206c28a-dc2e-3d13-7396-04ec35301740 6206c28a-dc2e-3d13-7396-04ec35301740}" 2025-07-29T00:55:42.130Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:42.130Z [INFO] server-e7ab1cfc-5fda-1d14-1367-35fcf53af884: snapshot restore progress: id=2-13-1753750542130 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:42.130Z [INFO] server-e7ab1cfc-5fda-1d14-1367-35fcf53af884: Installed remote snapshot 2025-07-29T00:55:42.131Z [INFO] server-e6220984-f6c8-f7f0-1653-3849643467ac: pipelining replication: peer="{Voter server-e7ab1cfc-5fda-1d14-1367-35fcf53af884 e7ab1cfc-5fda-1d14-1367-35fcf53af884}" 2025-07-29T00:55:43.143Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:42.130637017 +0000 UTC m=+21.820018361 (2 nodes), 1.04µs from start of poll, 1.70914968s from cluster start. Timeout at 2025-07-29 00:55:43.143448857 +0000 UTC m=+22.832830161, 1.0128118s after stability 2025-07-29T00:55:43.161Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: aborting pipeline replication: peer="{Voter server-05652cc9-0056-e382-2a0b-3120d0ac3049 05652cc9-0056-e382-2a0b-3120d0ac3049}" 2025-07-29T00:55:43.161Z [INFO] server-a3a22394-28d3-0f04-daf6-379e60eef4da: aborting pipeline replication: peer="{Voter server-d21d4e71-1dbc-87eb-5596-cd09a85afbb8 d21d4e71-1dbc-87eb-5596-cd09a85afbb8}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2025-07-29T00:55:43.162Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9819056d-4f66-320d-3ecb-ea6df2e379fa Address:9819056d-4f66-320d-3ecb-ea6df2e379fa} {Suffrage:Voter ID:server-c58ead7f-14bd-f765-0a1c-effb58848f0d Address:c58ead7f-14bd-f765-0a1c-effb58848f0d} {Suffrage:Voter ID:server-c02fd069-8913-f6fd-f13b-6d2827e57272 Address:c02fd069-8913-f6fd-f13b-6d2827e57272}]" 2025-07-29T00:55:43.162Z [INFO] server-c58ead7f-14bd-f765-0a1c-effb58848f0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9819056d-4f66-320d-3ecb-ea6df2e379fa Address:9819056d-4f66-320d-3ecb-ea6df2e379fa} {Suffrage:Voter ID:server-c58ead7f-14bd-f765-0a1c-effb58848f0d Address:c58ead7f-14bd-f765-0a1c-effb58848f0d} {Suffrage:Voter ID:server-c02fd069-8913-f6fd-f13b-6d2827e57272 Address:c02fd069-8913-f6fd-f13b-6d2827e57272}]" 2025-07-29T00:55:43.162Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: entering follower state: follower="Node at 9819056d-4f66-320d-3ecb-ea6df2e379fa [Follower]" leader-address= leader-id= 2025-07-29T00:55:43.162Z [INFO] server-c02fd069-8913-f6fd-f13b-6d2827e57272: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9819056d-4f66-320d-3ecb-ea6df2e379fa Address:9819056d-4f66-320d-3ecb-ea6df2e379fa} {Suffrage:Voter ID:server-c58ead7f-14bd-f765-0a1c-effb58848f0d Address:c58ead7f-14bd-f765-0a1c-effb58848f0d} {Suffrage:Voter ID:server-c02fd069-8913-f6fd-f13b-6d2827e57272 Address:c02fd069-8913-f6fd-f13b-6d2827e57272}]" 2025-07-29T00:55:43.162Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:43.162Z [INFO] server-c58ead7f-14bd-f765-0a1c-effb58848f0d: entering follower state: follower="Node at c58ead7f-14bd-f765-0a1c-effb58848f0d [Follower]" leader-address= leader-id= 2025-07-29T00:55:43.162Z [INFO] server-c02fd069-8913-f6fd-f13b-6d2827e57272: entering follower state: follower="Node at c02fd069-8913-f6fd-f13b-6d2827e57272 [Follower]" leader-address= leader-id= 2025-07-29T00:55:43.737Z [WARN] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:43.737Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: entering candidate state: node="Node at 9819056d-4f66-320d-3ecb-ea6df2e379fa [Candidate]" term=2 2025-07-29T00:55:43.737Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: election won: term=2 tally=2 2025-07-29T00:55:43.737Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: entering leader state: leader="Node at 9819056d-4f66-320d-3ecb-ea6df2e379fa [Leader]" 2025-07-29T00:55:43.737Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: added peer, starting replication: peer=server-c58ead7f-14bd-f765-0a1c-effb58848f0d 2025-07-29T00:55:43.737Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: added peer, starting replication: peer=server-c02fd069-8913-f6fd-f13b-6d2827e57272 2025-07-29T00:55:43.737Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: pipelining replication: peer="{Voter server-c02fd069-8913-f6fd-f13b-6d2827e57272 c02fd069-8913-f6fd-f13b-6d2827e57272}" 2025-07-29T00:55:43.738Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: pipelining replication: peer="{Voter server-c58ead7f-14bd-f765-0a1c-effb58848f0d c58ead7f-14bd-f765-0a1c-effb58848f0d}" 2025-07-29T00:55:44.742Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:43.737609937 +0000 UTC m=+23.426991281 (1 nodes), 575.29628ms from start of poll, 575.55152ms from cluster start. Timeout at 2025-07-29 00:55:44.742671897 +0000 UTC m=+24.432053241, 1.00506196s after stability 2025-07-29T00:55:44.744Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: starting snapshot up to: index=12 2025-07-29T00:55:44.744Z [INFO] creating new snapshot: path=/tmp/raft1144449513/snapshots/2-12-1753750544744.tmp 2025-07-29T00:55:44.745Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: snapshot complete up to: index=12 2025-07-29T00:55:44.746Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9536d81b-bc38-4c64-868c-09ae1b64b3a7 Address:9536d81b-bc38-4c64-868c-09ae1b64b3a7} {Suffrage:Voter ID:server-8022aeba-e2cc-ad1a-eadb-a8ac62112430 Address:8022aeba-e2cc-ad1a-eadb-a8ac62112430} {Suffrage:Voter ID:server-a91ddb55-5290-d115-a400-42f8b063fcd7 Address:a91ddb55-5290-d115-a400-42f8b063fcd7}]" 2025-07-29T00:55:44.746Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: entering follower state: follower="Node at 9536d81b-bc38-4c64-868c-09ae1b64b3a7 [Follower]" leader-address= leader-id= 2025-07-29T00:55:44.746Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9536d81b-bc38-4c64-868c-09ae1b64b3a7 Address:9536d81b-bc38-4c64-868c-09ae1b64b3a7} {Suffrage:Voter ID:server-8022aeba-e2cc-ad1a-eadb-a8ac62112430 Address:8022aeba-e2cc-ad1a-eadb-a8ac62112430} {Suffrage:Voter ID:server-a91ddb55-5290-d115-a400-42f8b063fcd7 Address:a91ddb55-5290-d115-a400-42f8b063fcd7}]" 2025-07-29T00:55:44.746Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: entering follower state: follower="Node at 8022aeba-e2cc-ad1a-eadb-a8ac62112430 [Follower]" leader-address= leader-id= 2025-07-29T00:55:44.746Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9536d81b-bc38-4c64-868c-09ae1b64b3a7 Address:9536d81b-bc38-4c64-868c-09ae1b64b3a7} {Suffrage:Voter ID:server-8022aeba-e2cc-ad1a-eadb-a8ac62112430 Address:8022aeba-e2cc-ad1a-eadb-a8ac62112430} {Suffrage:Voter ID:server-a91ddb55-5290-d115-a400-42f8b063fcd7 Address:a91ddb55-5290-d115-a400-42f8b063fcd7}]" 2025-07-29T00:55:44.746Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:44.746Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: entering follower state: follower="Node at a91ddb55-5290-d115-a400-42f8b063fcd7 [Follower]" leader-address= leader-id= 2025-07-29T00:55:45.495Z [WARN] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:45.495Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: entering candidate state: node="Node at 8022aeba-e2cc-ad1a-eadb-a8ac62112430 [Candidate]" term=2 2025-07-29T00:55:45.495Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: election won: term=2 tally=2 2025-07-29T00:55:45.495Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: entering leader state: leader="Node at 8022aeba-e2cc-ad1a-eadb-a8ac62112430 [Leader]" 2025-07-29T00:55:45.495Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: added peer, starting replication: peer=server-9536d81b-bc38-4c64-868c-09ae1b64b3a7 2025-07-29T00:55:45.495Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: added peer, starting replication: peer=server-a91ddb55-5290-d115-a400-42f8b063fcd7 2025-07-29T00:55:45.495Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: pipelining replication: peer="{Voter server-a91ddb55-5290-d115-a400-42f8b063fcd7 a91ddb55-5290-d115-a400-42f8b063fcd7}" 2025-07-29T00:55:45.496Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: pipelining replication: peer="{Voter server-9536d81b-bc38-4c64-868c-09ae1b64b3a7 9536d81b-bc38-4c64-868c-09ae1b64b3a7}" 2025-07-29T00:55:46.500Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:45.495863777 +0000 UTC m=+25.185245161 (1 nodes), 749.16168ms from start of poll, 749.81856ms from cluster start. Timeout at 2025-07-29 00:55:46.500888937 +0000 UTC m=+26.190270281, 1.00502512s after stability 2025-07-29T00:55:46.501Z [INFO] creating new snapshot: path=/tmp/raft4271628510/snapshots/2-13-1753750546501.tmp 2025-07-29T00:55:46.501Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: copied to local snapshot: bytes=71 2025-07-29T00:55:46.501Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: snapshot restore progress: id=2-13-1753750546501 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:46.502Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: removing all old logs from log store 2025-07-29T00:55:46.502Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: compacting logs: from=1 to=2 2025-07-29T00:55:46.502Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: restored user snapshot: index=13 2025-07-29T00:55:46.502Z [ERROR] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: failed to get log: index=2 error="log not found" 2025-07-29T00:55:46.502Z [ERROR] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: failed to get log: index=2 error="log not found" 2025-07-29T00:55:46.502Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: aborting pipeline replication: peer="{Voter server-9536d81b-bc38-4c64-868c-09ae1b64b3a7 9536d81b-bc38-4c64-868c-09ae1b64b3a7}" 2025-07-29T00:55:46.502Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: aborting pipeline replication: peer="{Voter server-a91ddb55-5290-d115-a400-42f8b063fcd7 a91ddb55-5290-d115-a400-42f8b063fcd7}" 2025-07-29T00:55:46.508Z [ERROR] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: failed to get log: index=2 error="log not found" 2025-07-29T00:55:46.509Z [INFO] creating new snapshot: path=/tmp/raft4165442049/snapshots/2-13-1753750546509.tmp 2025-07-29T00:55:46.509Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:46.509Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: copied to local snapshot: bytes=71 2025-07-29T00:55:46.510Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: snapshot restore progress: id=2-13-1753750546509 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:46.510Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: removing all old logs from log store 2025-07-29T00:55:46.510Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: compacting logs: from=1 to=2 2025-07-29T00:55:46.510Z [INFO] server-a91ddb55-5290-d115-a400-42f8b063fcd7: Installed remote snapshot 2025-07-29T00:55:46.510Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: pipelining replication: peer="{Voter server-a91ddb55-5290-d115-a400-42f8b063fcd7 a91ddb55-5290-d115-a400-42f8b063fcd7}" 2025-07-29T00:55:46.510Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:46.512Z [ERROR] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: failed to get log: index=2 error="log not found" 2025-07-29T00:55:46.512Z [INFO] creating new snapshot: path=/tmp/raft3162340329/snapshots/2-13-1753750546512.tmp 2025-07-29T00:55:46.512Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:46.513Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: copied to local snapshot: bytes=71 2025-07-29T00:55:46.513Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: snapshot restore progress: id=2-13-1753750546512 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:46.513Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: removing all old logs from log store 2025-07-29T00:55:46.513Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: compacting logs: from=1 to=2 2025-07-29T00:55:46.513Z [INFO] server-9536d81b-bc38-4c64-868c-09ae1b64b3a7: Installed remote snapshot 2025-07-29T00:55:46.513Z [INFO] server-8022aeba-e2cc-ad1a-eadb-a8ac62112430: pipelining replication: peer="{Voter server-9536d81b-bc38-4c64-868c-09ae1b64b3a7 9536d81b-bc38-4c64-868c-09ae1b64b3a7}" 2025-07-29T00:55:47.516Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:46.510397297 +0000 UTC m=+26.199778641 (2 nodes), 880ns from start of poll, 1.76435204s from cluster start. Timeout at 2025-07-29 00:55:47.516240977 +0000 UTC m=+27.205622321, 1.00584368s after stability 2025-07-29T00:55:47.528Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: aborting pipeline replication: peer="{Voter server-c02fd069-8913-f6fd-f13b-6d2827e57272 c02fd069-8913-f6fd-f13b-6d2827e57272}" 2025-07-29T00:55:47.528Z [INFO] server-9819056d-4f66-320d-3ecb-ea6df2e379fa: aborting pipeline replication: peer="{Voter server-c58ead7f-14bd-f765-0a1c-effb58848f0d c58ead7f-14bd-f765-0a1c-effb58848f0d}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2025-07-29T00:55:47.530Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd067e57-bcff-d6d6-20c0-187fb47ccb60 Address:fd067e57-bcff-d6d6-20c0-187fb47ccb60} {Suffrage:Voter ID:server-8a0107db-f79f-065d-065e-cae3b37eb4c3 Address:8a0107db-f79f-065d-065e-cae3b37eb4c3} {Suffrage:Voter ID:server-1244d4fe-99a1-bab5-c499-7f848644886c Address:1244d4fe-99a1-bab5-c499-7f848644886c}]" 2025-07-29T00:55:47.530Z [INFO] server-8a0107db-f79f-065d-065e-cae3b37eb4c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd067e57-bcff-d6d6-20c0-187fb47ccb60 Address:fd067e57-bcff-d6d6-20c0-187fb47ccb60} {Suffrage:Voter ID:server-8a0107db-f79f-065d-065e-cae3b37eb4c3 Address:8a0107db-f79f-065d-065e-cae3b37eb4c3} {Suffrage:Voter ID:server-1244d4fe-99a1-bab5-c499-7f848644886c Address:1244d4fe-99a1-bab5-c499-7f848644886c}]" 2025-07-29T00:55:47.530Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: entering follower state: follower="Node at fd067e57-bcff-d6d6-20c0-187fb47ccb60 [Follower]" leader-address= leader-id= 2025-07-29T00:55:47.531Z [INFO] server-8a0107db-f79f-065d-065e-cae3b37eb4c3: entering follower state: follower="Node at 8a0107db-f79f-065d-065e-cae3b37eb4c3 [Follower]" leader-address= leader-id= 2025-07-29T00:55:47.531Z [INFO] server-1244d4fe-99a1-bab5-c499-7f848644886c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd067e57-bcff-d6d6-20c0-187fb47ccb60 Address:fd067e57-bcff-d6d6-20c0-187fb47ccb60} {Suffrage:Voter ID:server-8a0107db-f79f-065d-065e-cae3b37eb4c3 Address:8a0107db-f79f-065d-065e-cae3b37eb4c3} {Suffrage:Voter ID:server-1244d4fe-99a1-bab5-c499-7f848644886c Address:1244d4fe-99a1-bab5-c499-7f848644886c}]" 2025-07-29T00:55:47.531Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:47.531Z [INFO] server-1244d4fe-99a1-bab5-c499-7f848644886c: entering follower state: follower="Node at 1244d4fe-99a1-bab5-c499-7f848644886c [Follower]" leader-address= leader-id= 2025-07-29T00:55:48.046Z [WARN] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:48.046Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: entering candidate state: node="Node at fd067e57-bcff-d6d6-20c0-187fb47ccb60 [Candidate]" term=2 2025-07-29T00:55:48.046Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: election won: term=2 tally=2 2025-07-29T00:55:48.046Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: entering leader state: leader="Node at fd067e57-bcff-d6d6-20c0-187fb47ccb60 [Leader]" 2025-07-29T00:55:48.046Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: added peer, starting replication: peer=server-8a0107db-f79f-065d-065e-cae3b37eb4c3 2025-07-29T00:55:48.046Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: added peer, starting replication: peer=server-1244d4fe-99a1-bab5-c499-7f848644886c 2025-07-29T00:55:48.046Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: pipelining replication: peer="{Voter server-1244d4fe-99a1-bab5-c499-7f848644886c 1244d4fe-99a1-bab5-c499-7f848644886c}" 2025-07-29T00:55:48.046Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: pipelining replication: peer="{Voter server-8a0107db-f79f-065d-065e-cae3b37eb4c3 8a0107db-f79f-065d-065e-cae3b37eb4c3}" 2025-07-29T00:55:49.052Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:48.046876737 +0000 UTC m=+27.736258081 (1 nodes), 515.73548ms from start of poll, 516.36292ms from cluster start. Timeout at 2025-07-29 00:55:49.052755577 +0000 UTC m=+28.742136921, 1.00587884s after stability 2025-07-29T00:55:49.054Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: starting snapshot up to: index=12 2025-07-29T00:55:49.054Z [INFO] creating new snapshot: path=/tmp/raft1886709491/snapshots/2-12-1753750549054.tmp 2025-07-29T00:55:49.055Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: snapshot complete up to: index=12 2025-07-29T00:55:49.056Z [INFO] creating new snapshot: path=/tmp/raft1886709491/snapshots/2-23-1753750549056.tmp 2025-07-29T00:55:49.056Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: copied to local snapshot: bytes=71 2025-07-29T00:55:49.056Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: snapshot restore progress: id=2-23-1753750549056 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:49.056Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: restored user snapshot: index=23 2025-07-29T00:55:49.056Z [ERROR] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: failed to get log: index=23 error="log not found" 2025-07-29T00:55:49.056Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: aborting pipeline replication: peer="{Voter server-1244d4fe-99a1-bab5-c499-7f848644886c 1244d4fe-99a1-bab5-c499-7f848644886c}" 2025-07-29T00:55:49.056Z [ERROR] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: failed to get log: index=23 error="log not found" 2025-07-29T00:55:49.056Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: aborting pipeline replication: peer="{Voter server-8a0107db-f79f-065d-065e-cae3b37eb4c3 8a0107db-f79f-065d-065e-cae3b37eb4c3}" 2025-07-29T00:55:49.065Z [ERROR] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: failed to get log: index=23 error="log not found" 2025-07-29T00:55:49.066Z [INFO] creating new snapshot: path=/tmp/raft1744823510/snapshots/2-23-1753750549066.tmp 2025-07-29T00:55:49.066Z [ERROR] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: failed to get log: index=23 error="log not found" 2025-07-29T00:55:49.066Z [INFO] server-1244d4fe-99a1-bab5-c499-7f848644886c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:49.066Z [INFO] server-1244d4fe-99a1-bab5-c499-7f848644886c: copied to local snapshot: bytes=71 2025-07-29T00:55:49.067Z [INFO] creating new snapshot: path=/tmp/raft1274154246/snapshots/2-23-1753750549067.tmp 2025-07-29T00:55:49.067Z [INFO] server-1244d4fe-99a1-bab5-c499-7f848644886c: snapshot restore progress: id=2-23-1753750549066 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:49.067Z [INFO] server-1244d4fe-99a1-bab5-c499-7f848644886c: Installed remote snapshot 2025-07-29T00:55:49.067Z [INFO] server-8a0107db-f79f-065d-065e-cae3b37eb4c3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:49.067Z [WARN] server-1244d4fe-99a1-bab5-c499-7f848644886c: skipping application of old log: index=22 2025-07-29T00:55:49.067Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: pipelining replication: peer="{Voter server-1244d4fe-99a1-bab5-c499-7f848644886c 1244d4fe-99a1-bab5-c499-7f848644886c}" 2025-07-29T00:55:49.067Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:49.067Z [INFO] server-8a0107db-f79f-065d-065e-cae3b37eb4c3: copied to local snapshot: bytes=71 2025-07-29T00:55:49.067Z [INFO] server-8a0107db-f79f-065d-065e-cae3b37eb4c3: snapshot restore progress: id=2-23-1753750549067 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:49.067Z [INFO] server-8a0107db-f79f-065d-065e-cae3b37eb4c3: Installed remote snapshot 2025-07-29T00:55:49.067Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: pipelining replication: peer="{Voter server-8a0107db-f79f-065d-065e-cae3b37eb4c3 8a0107db-f79f-065d-065e-cae3b37eb4c3}" 2025-07-29T00:55:50.072Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:49.067444537 +0000 UTC m=+28.756825881 (2 nodes), 1.52µs from start of poll, 1.53693072s from cluster start. Timeout at 2025-07-29 00:55:50.072763817 +0000 UTC m=+29.762145161, 1.00531928s after stability 2025-07-29T00:55:50.084Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: aborting pipeline replication: peer="{Voter server-1244d4fe-99a1-bab5-c499-7f848644886c 1244d4fe-99a1-bab5-c499-7f848644886c}" 2025-07-29T00:55:50.084Z [INFO] server-fd067e57-bcff-d6d6-20c0-187fb47ccb60: aborting pipeline replication: peer="{Voter server-8a0107db-f79f-065d-065e-cae3b37eb4c3 8a0107db-f79f-065d-065e-cae3b37eb4c3}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2025-07-29T00:55:50.085Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 Address:9611a30e-2cda-9bb6-7849-7c3d86d22b80} {Suffrage:Voter ID:server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 Address:86f24c0e-ecbe-67ae-b9ed-c55b92289ab7} {Suffrage:Voter ID:server-b1e02173-adc6-4824-d7af-405d49214d28 Address:b1e02173-adc6-4824-d7af-405d49214d28}]" 2025-07-29T00:55:50.085Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: entering follower state: follower="Node at 9611a30e-2cda-9bb6-7849-7c3d86d22b80 [Follower]" leader-address= leader-id= 2025-07-29T00:55:50.085Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 Address:9611a30e-2cda-9bb6-7849-7c3d86d22b80} {Suffrage:Voter ID:server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 Address:86f24c0e-ecbe-67ae-b9ed-c55b92289ab7} {Suffrage:Voter ID:server-b1e02173-adc6-4824-d7af-405d49214d28 Address:b1e02173-adc6-4824-d7af-405d49214d28}]" 2025-07-29T00:55:50.085Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 Address:9611a30e-2cda-9bb6-7849-7c3d86d22b80} {Suffrage:Voter ID:server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 Address:86f24c0e-ecbe-67ae-b9ed-c55b92289ab7} {Suffrage:Voter ID:server-b1e02173-adc6-4824-d7af-405d49214d28 Address:b1e02173-adc6-4824-d7af-405d49214d28}]" 2025-07-29T00:55:50.085Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: entering follower state: follower="Node at 86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 [Follower]" leader-address= leader-id= 2025-07-29T00:55:50.086Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:50.086Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: entering follower state: follower="Node at b1e02173-adc6-4824-d7af-405d49214d28 [Follower]" leader-address= leader-id= 2025-07-29T00:55:50.656Z [WARN] server-b1e02173-adc6-4824-d7af-405d49214d28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:50.656Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: entering candidate state: node="Node at b1e02173-adc6-4824-d7af-405d49214d28 [Candidate]" term=2 2025-07-29T00:55:50.657Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: election won: term=2 tally=2 2025-07-29T00:55:50.657Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: entering leader state: leader="Node at b1e02173-adc6-4824-d7af-405d49214d28 [Leader]" 2025-07-29T00:55:50.657Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: added peer, starting replication: peer=server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 2025-07-29T00:55:50.657Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: added peer, starting replication: peer=server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 2025-07-29T00:55:50.657Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: pipelining replication: peer="{Voter server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 86f24c0e-ecbe-67ae-b9ed-c55b92289ab7}" 2025-07-29T00:55:50.657Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: pipelining replication: peer="{Voter server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 9611a30e-2cda-9bb6-7849-7c3d86d22b80}" 2025-07-29T00:55:51.662Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:50.657264857 +0000 UTC m=+30.346646201 (1 nodes), 571.2062ms from start of poll, 571.70336ms from cluster start. Timeout at 2025-07-29 00:55:51.662401857 +0000 UTC m=+31.351783241, 1.00513704s after stability 2025-07-29T00:55:51.663Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: starting snapshot up to: index=12 2025-07-29T00:55:51.663Z [INFO] creating new snapshot: path=/tmp/raft3827985618/snapshots/2-12-1753750551663.tmp 2025-07-29T00:55:51.663Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: snapshot complete up to: index=12 2025-07-29T00:55:51.664Z [INFO] creating new snapshot: path=/tmp/raft3827985618/snapshots/2-23-1753750551664.tmp 2025-07-29T00:55:51.665Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: copied to local snapshot: bytes=71 2025-07-29T00:55:51.665Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: snapshot restore progress: id=2-23-1753750551664 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:51.665Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: removing all old logs from log store 2025-07-29T00:55:51.665Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: compacting logs: from=1 to=22 2025-07-29T00:55:51.666Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: restored user snapshot: index=23 2025-07-29T00:55:51.666Z [ERROR] server-b1e02173-adc6-4824-d7af-405d49214d28: failed to get log: index=22 error="log not found" 2025-07-29T00:55:51.666Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: aborting pipeline replication: peer="{Voter server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 86f24c0e-ecbe-67ae-b9ed-c55b92289ab7}" 2025-07-29T00:55:51.666Z [ERROR] server-b1e02173-adc6-4824-d7af-405d49214d28: failed to get log: index=22 error="log not found" 2025-07-29T00:55:51.666Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: aborting pipeline replication: peer="{Voter server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 9611a30e-2cda-9bb6-7849-7c3d86d22b80}" 2025-07-29T00:55:51.675Z [ERROR] server-b1e02173-adc6-4824-d7af-405d49214d28: failed to get log: index=22 error="log not found" 2025-07-29T00:55:51.675Z [ERROR] server-b1e02173-adc6-4824-d7af-405d49214d28: failed to get log: index=22 error="log not found" 2025-07-29T00:55:51.676Z [INFO] creating new snapshot: path=/tmp/raft3057978222/snapshots/2-23-1753750551676.tmp 2025-07-29T00:55:51.676Z [INFO] creating new snapshot: path=/tmp/raft1851647527/snapshots/2-23-1753750551676.tmp 2025-07-29T00:55:51.676Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:51.676Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:51.676Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: copied to local snapshot: bytes=71 2025-07-29T00:55:51.676Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: copied to local snapshot: bytes=71 2025-07-29T00:55:51.677Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: snapshot restore progress: id=2-23-1753750551676 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:51.677Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: removing all old logs from log store 2025-07-29T00:55:51.677Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: compacting logs: from=1 to=22 2025-07-29T00:55:51.677Z [INFO] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: Installed remote snapshot 2025-07-29T00:55:51.677Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: snapshot restore progress: id=2-23-1753750551676 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:51.677Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: removing all old logs from log store 2025-07-29T00:55:51.677Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: compacting logs: from=1 to=22 2025-07-29T00:55:51.677Z [WARN] server-9611a30e-2cda-9bb6-7849-7c3d86d22b80: skipping application of old log: index=22 2025-07-29T00:55:51.677Z [INFO] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: Installed remote snapshot 2025-07-29T00:55:51.677Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: pipelining replication: peer="{Voter server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 9611a30e-2cda-9bb6-7849-7c3d86d22b80}" 2025-07-29T00:55:51.677Z [WARN] server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7: skipping application of old log: index=23 2025-07-29T00:55:51.677Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:51.677Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: pipelining replication: peer="{Voter server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 86f24c0e-ecbe-67ae-b9ed-c55b92289ab7}" 2025-07-29T00:55:52.682Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:51.677350097 +0000 UTC m=+31.366731441 (2 nodes), 880ns from start of poll, 1.5917886s from cluster start. Timeout at 2025-07-29 00:55:52.682801937 +0000 UTC m=+32.372183321, 1.00545188s after stability 2025-07-29T00:55:52.694Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: aborting pipeline replication: peer="{Voter server-86f24c0e-ecbe-67ae-b9ed-c55b92289ab7 86f24c0e-ecbe-67ae-b9ed-c55b92289ab7}" 2025-07-29T00:55:52.694Z [INFO] server-b1e02173-adc6-4824-d7af-405d49214d28: aborting pipeline replication: peer="{Voter server-9611a30e-2cda-9bb6-7849-7c3d86d22b80 9611a30e-2cda-9bb6-7849-7c3d86d22b80}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2025-07-29T00:55:52.695Z [INFO] server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8 Address:65a78e44-2b41-18c9-dc32-ac0d7459c8d8} {Suffrage:Voter ID:server-872b94ef-249c-a450-9465-d13e19d4d2b4 Address:872b94ef-249c-a450-9465-d13e19d4d2b4} {Suffrage:Voter ID:server-29ef1680-f4c0-1345-0a59-35d9664d90f2 Address:29ef1680-f4c0-1345-0a59-35d9664d90f2}]" 2025-07-29T00:55:52.695Z [INFO] server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8: entering follower state: follower="Node at 65a78e44-2b41-18c9-dc32-ac0d7459c8d8 [Follower]" leader-address= leader-id= 2025-07-29T00:55:52.695Z [INFO] server-872b94ef-249c-a450-9465-d13e19d4d2b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8 Address:65a78e44-2b41-18c9-dc32-ac0d7459c8d8} {Suffrage:Voter ID:server-872b94ef-249c-a450-9465-d13e19d4d2b4 Address:872b94ef-249c-a450-9465-d13e19d4d2b4} {Suffrage:Voter ID:server-29ef1680-f4c0-1345-0a59-35d9664d90f2 Address:29ef1680-f4c0-1345-0a59-35d9664d90f2}]" 2025-07-29T00:55:52.695Z [INFO] server-872b94ef-249c-a450-9465-d13e19d4d2b4: entering follower state: follower="Node at 872b94ef-249c-a450-9465-d13e19d4d2b4 [Follower]" leader-address= leader-id= 2025-07-29T00:55:52.696Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8 Address:65a78e44-2b41-18c9-dc32-ac0d7459c8d8} {Suffrage:Voter ID:server-872b94ef-249c-a450-9465-d13e19d4d2b4 Address:872b94ef-249c-a450-9465-d13e19d4d2b4} {Suffrage:Voter ID:server-29ef1680-f4c0-1345-0a59-35d9664d90f2 Address:29ef1680-f4c0-1345-0a59-35d9664d90f2}]" 2025-07-29T00:55:52.696Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:52.696Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: entering follower state: follower="Node at 29ef1680-f4c0-1345-0a59-35d9664d90f2 [Follower]" leader-address= leader-id= 2025-07-29T00:55:53.315Z [WARN] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:53.315Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: entering candidate state: node="Node at 29ef1680-f4c0-1345-0a59-35d9664d90f2 [Candidate]" term=2 2025-07-29T00:55:53.316Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: election won: term=2 tally=2 2025-07-29T00:55:53.316Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: entering leader state: leader="Node at 29ef1680-f4c0-1345-0a59-35d9664d90f2 [Leader]" 2025-07-29T00:55:53.316Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: added peer, starting replication: peer=server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8 2025-07-29T00:55:53.316Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: added peer, starting replication: peer=server-872b94ef-249c-a450-9465-d13e19d4d2b4 2025-07-29T00:55:53.316Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: pipelining replication: peer="{Voter server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8 65a78e44-2b41-18c9-dc32-ac0d7459c8d8}" 2025-07-29T00:55:53.316Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: pipelining replication: peer="{Voter server-872b94ef-249c-a450-9465-d13e19d4d2b4 872b94ef-249c-a450-9465-d13e19d4d2b4}" 2025-07-29T00:55:54.321Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:53.316255177 +0000 UTC m=+33.005636521 (1 nodes), 619.71608ms from start of poll, 620.8836ms from cluster start. Timeout at 2025-07-29 00:55:54.321697057 +0000 UTC m=+34.011078401, 1.00544188s after stability 2025-07-29T00:55:54.323Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: starting snapshot up to: index=12 2025-07-29T00:55:54.323Z [INFO] creating new snapshot: path=/tmp/raft2724808675/snapshots/2-12-1753750554323.tmp 2025-07-29T00:55:54.324Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: snapshot complete up to: index=12 2025-07-29T00:55:54.325Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55 Address:f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55} {Suffrage:Voter ID:server-b825177a-d826-094d-21fc-b4c531081ce5 Address:b825177a-d826-094d-21fc-b4c531081ce5} {Suffrage:Voter ID:server-fddbf8f1-88b0-c403-6932-dbdc552312b0 Address:fddbf8f1-88b0-c403-6932-dbdc552312b0}]" 2025-07-29T00:55:54.325Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: entering follower state: follower="Node at f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55 [Follower]" leader-address= leader-id= 2025-07-29T00:55:54.325Z [INFO] server-b825177a-d826-094d-21fc-b4c531081ce5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55 Address:f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55} {Suffrage:Voter ID:server-b825177a-d826-094d-21fc-b4c531081ce5 Address:b825177a-d826-094d-21fc-b4c531081ce5} {Suffrage:Voter ID:server-fddbf8f1-88b0-c403-6932-dbdc552312b0 Address:fddbf8f1-88b0-c403-6932-dbdc552312b0}]" 2025-07-29T00:55:54.325Z [INFO] server-fddbf8f1-88b0-c403-6932-dbdc552312b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55 Address:f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55} {Suffrage:Voter ID:server-b825177a-d826-094d-21fc-b4c531081ce5 Address:b825177a-d826-094d-21fc-b4c531081ce5} {Suffrage:Voter ID:server-fddbf8f1-88b0-c403-6932-dbdc552312b0 Address:fddbf8f1-88b0-c403-6932-dbdc552312b0}]" 2025-07-29T00:55:54.325Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:54.325Z [INFO] server-b825177a-d826-094d-21fc-b4c531081ce5: entering follower state: follower="Node at b825177a-d826-094d-21fc-b4c531081ce5 [Follower]" leader-address= leader-id= 2025-07-29T00:55:54.325Z [INFO] server-fddbf8f1-88b0-c403-6932-dbdc552312b0: entering follower state: follower="Node at fddbf8f1-88b0-c403-6932-dbdc552312b0 [Follower]" leader-address= leader-id= 2025-07-29T00:55:54.853Z [WARN] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:54.853Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: entering candidate state: node="Node at f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55 [Candidate]" term=2 2025-07-29T00:55:54.853Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: election won: term=2 tally=2 2025-07-29T00:55:54.853Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: entering leader state: leader="Node at f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55 [Leader]" 2025-07-29T00:55:54.853Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: added peer, starting replication: peer=server-b825177a-d826-094d-21fc-b4c531081ce5 2025-07-29T00:55:54.853Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: added peer, starting replication: peer=server-fddbf8f1-88b0-c403-6932-dbdc552312b0 2025-07-29T00:55:54.853Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: pipelining replication: peer="{Voter server-fddbf8f1-88b0-c403-6932-dbdc552312b0 fddbf8f1-88b0-c403-6932-dbdc552312b0}" 2025-07-29T00:55:54.853Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: pipelining replication: peer="{Voter server-b825177a-d826-094d-21fc-b4c531081ce5 b825177a-d826-094d-21fc-b4c531081ce5}" 2025-07-29T00:55:55.859Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:54.853741417 +0000 UTC m=+34.543122801 (1 nodes), 527.84956ms from start of poll, 528.3184ms from cluster start. Timeout at 2025-07-29 00:55:55.859591337 +0000 UTC m=+35.548972681, 1.00584988s after stability 2025-07-29T00:55:55.859Z [INFO] creating new snapshot: path=/tmp/raft2934670951/snapshots/2-14-1753750555859.tmp 2025-07-29T00:55:55.860Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: copied to local snapshot: bytes=71 2025-07-29T00:55:55.860Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: snapshot restore progress: id=2-14-1753750555859 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:55.860Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: restored user snapshot: index=14 2025-07-29T00:55:55.860Z [ERROR] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: failed to get log: index=3 error="log not found" 2025-07-29T00:55:55.860Z [ERROR] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: failed to get log: index=3 error="log not found" 2025-07-29T00:55:55.860Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: aborting pipeline replication: peer="{Voter server-b825177a-d826-094d-21fc-b4c531081ce5 b825177a-d826-094d-21fc-b4c531081ce5}" 2025-07-29T00:55:55.861Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: aborting pipeline replication: peer="{Voter server-fddbf8f1-88b0-c403-6932-dbdc552312b0 fddbf8f1-88b0-c403-6932-dbdc552312b0}" 2025-07-29T00:55:55.867Z [ERROR] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: failed to get log: index=3 error="log not found" 2025-07-29T00:55:55.867Z [ERROR] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: failed to get log: index=3 error="log not found" 2025-07-29T00:55:55.867Z [INFO] creating new snapshot: path=/tmp/raft926466837/snapshots/2-14-1753750555867.tmp 2025-07-29T00:55:55.867Z [INFO] creating new snapshot: path=/tmp/raft4051381475/snapshots/2-14-1753750555867.tmp 2025-07-29T00:55:55.868Z [INFO] server-b825177a-d826-094d-21fc-b4c531081ce5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:55.868Z [INFO] server-b825177a-d826-094d-21fc-b4c531081ce5: copied to local snapshot: bytes=71 2025-07-29T00:55:55.868Z [INFO] server-b825177a-d826-094d-21fc-b4c531081ce5: snapshot restore progress: id=2-14-1753750555867 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:55.868Z [INFO] server-fddbf8f1-88b0-c403-6932-dbdc552312b0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:55.868Z [INFO] server-b825177a-d826-094d-21fc-b4c531081ce5: Installed remote snapshot 2025-07-29T00:55:55.868Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: pipelining replication: peer="{Voter server-b825177a-d826-094d-21fc-b4c531081ce5 b825177a-d826-094d-21fc-b4c531081ce5}" 2025-07-29T00:55:55.868Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:55.869Z [INFO] server-fddbf8f1-88b0-c403-6932-dbdc552312b0: copied to local snapshot: bytes=71 2025-07-29T00:55:55.870Z [INFO] server-fddbf8f1-88b0-c403-6932-dbdc552312b0: snapshot restore progress: id=2-14-1753750555867 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:55.870Z [INFO] server-fddbf8f1-88b0-c403-6932-dbdc552312b0: Installed remote snapshot 2025-07-29T00:55:55.870Z [INFO] server-f1b68bb1-7ff4-2909-6ca7-c7209dfbfd55: pipelining replication: peer="{Voter server-fddbf8f1-88b0-c403-6932-dbdc552312b0 fddbf8f1-88b0-c403-6932-dbdc552312b0}" 2025-07-29T00:55:56.874Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:55.869019537 +0000 UTC m=+35.558400881 (2 nodes), 1.04µs from start of poll, 1.54359648s from cluster start. Timeout at 2025-07-29 00:55:56.874259297 +0000 UTC m=+36.563640641, 1.00523976s after stability 2025-07-29T00:55:56.886Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: aborting pipeline replication: peer="{Voter server-872b94ef-249c-a450-9465-d13e19d4d2b4 872b94ef-249c-a450-9465-d13e19d4d2b4}" 2025-07-29T00:55:56.886Z [INFO] server-29ef1680-f4c0-1345-0a59-35d9664d90f2: aborting pipeline replication: peer="{Voter server-65a78e44-2b41-18c9-dc32-ac0d7459c8d8 65a78e44-2b41-18c9-dc32-ac0d7459c8d8}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2025-07-29T00:55:56.887Z [INFO] server-60247c2a-43b3-4287-bf36-dc2642e078dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60247c2a-43b3-4287-bf36-dc2642e078dd Address:60247c2a-43b3-4287-bf36-dc2642e078dd} {Suffrage:Voter ID:server-8b53d449-dce0-ad01-4637-24490064a751 Address:8b53d449-dce0-ad01-4637-24490064a751} {Suffrage:Voter ID:server-ffc57319-15fc-9098-c3b6-0fd48b6892c1 Address:ffc57319-15fc-9098-c3b6-0fd48b6892c1}]" 2025-07-29T00:55:56.887Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60247c2a-43b3-4287-bf36-dc2642e078dd Address:60247c2a-43b3-4287-bf36-dc2642e078dd} {Suffrage:Voter ID:server-8b53d449-dce0-ad01-4637-24490064a751 Address:8b53d449-dce0-ad01-4637-24490064a751} {Suffrage:Voter ID:server-ffc57319-15fc-9098-c3b6-0fd48b6892c1 Address:ffc57319-15fc-9098-c3b6-0fd48b6892c1}]" 2025-07-29T00:55:56.887Z [INFO] server-60247c2a-43b3-4287-bf36-dc2642e078dd: entering follower state: follower="Node at 60247c2a-43b3-4287-bf36-dc2642e078dd [Follower]" leader-address= leader-id= 2025-07-29T00:55:56.887Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: entering follower state: follower="Node at 8b53d449-dce0-ad01-4637-24490064a751 [Follower]" leader-address= leader-id= 2025-07-29T00:55:56.887Z [INFO] server-ffc57319-15fc-9098-c3b6-0fd48b6892c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60247c2a-43b3-4287-bf36-dc2642e078dd Address:60247c2a-43b3-4287-bf36-dc2642e078dd} {Suffrage:Voter ID:server-8b53d449-dce0-ad01-4637-24490064a751 Address:8b53d449-dce0-ad01-4637-24490064a751} {Suffrage:Voter ID:server-ffc57319-15fc-9098-c3b6-0fd48b6892c1 Address:ffc57319-15fc-9098-c3b6-0fd48b6892c1}]" 2025-07-29T00:55:56.887Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:56.887Z [INFO] server-ffc57319-15fc-9098-c3b6-0fd48b6892c1: entering follower state: follower="Node at ffc57319-15fc-9098-c3b6-0fd48b6892c1 [Follower]" leader-address= leader-id= 2025-07-29T00:55:57.405Z [WARN] server-8b53d449-dce0-ad01-4637-24490064a751: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:57.405Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: entering candidate state: node="Node at 8b53d449-dce0-ad01-4637-24490064a751 [Candidate]" term=2 2025-07-29T00:55:57.405Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: election won: term=2 tally=2 2025-07-29T00:55:57.405Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: entering leader state: leader="Node at 8b53d449-dce0-ad01-4637-24490064a751 [Leader]" 2025-07-29T00:55:57.405Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: added peer, starting replication: peer=server-60247c2a-43b3-4287-bf36-dc2642e078dd 2025-07-29T00:55:57.405Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: added peer, starting replication: peer=server-ffc57319-15fc-9098-c3b6-0fd48b6892c1 2025-07-29T00:55:57.405Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: pipelining replication: peer="{Voter server-ffc57319-15fc-9098-c3b6-0fd48b6892c1 ffc57319-15fc-9098-c3b6-0fd48b6892c1}" 2025-07-29T00:55:57.405Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: pipelining replication: peer="{Voter server-60247c2a-43b3-4287-bf36-dc2642e078dd 60247c2a-43b3-4287-bf36-dc2642e078dd}" 2025-07-29T00:55:58.411Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:57.405415697 +0000 UTC m=+37.094797081 (1 nodes), 517.84612ms from start of poll, 518.36816ms from cluster start. Timeout at 2025-07-29 00:55:58.411139977 +0000 UTC m=+38.100521321, 1.00572424s after stability 2025-07-29T00:55:58.411Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: starting snapshot up to: index=12 2025-07-29T00:55:58.411Z [INFO] creating new snapshot: path=/tmp/raft3780595993/snapshots/2-12-1753750558411.tmp 2025-07-29T00:55:58.412Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: snapshot complete up to: index=12 2025-07-29T00:55:58.414Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 Address:c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9} {Suffrage:Voter ID:server-f286acad-2aca-7343-6668-06092037f323 Address:f286acad-2aca-7343-6668-06092037f323} {Suffrage:Voter ID:server-4269c970-7db9-c916-1bfa-27b56eb6f580 Address:4269c970-7db9-c916-1bfa-27b56eb6f580}]" 2025-07-29T00:55:58.415Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 Address:c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9} {Suffrage:Voter ID:server-f286acad-2aca-7343-6668-06092037f323 Address:f286acad-2aca-7343-6668-06092037f323} {Suffrage:Voter ID:server-4269c970-7db9-c916-1bfa-27b56eb6f580 Address:4269c970-7db9-c916-1bfa-27b56eb6f580}]" 2025-07-29T00:55:58.415Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: entering follower state: follower="Node at c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 [Follower]" leader-address= leader-id= 2025-07-29T00:55:58.415Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: entering follower state: follower="Node at f286acad-2aca-7343-6668-06092037f323 [Follower]" leader-address= leader-id= 2025-07-29T00:55:58.415Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 Address:c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9} {Suffrage:Voter ID:server-f286acad-2aca-7343-6668-06092037f323 Address:f286acad-2aca-7343-6668-06092037f323} {Suffrage:Voter ID:server-4269c970-7db9-c916-1bfa-27b56eb6f580 Address:4269c970-7db9-c916-1bfa-27b56eb6f580}]" 2025-07-29T00:55:58.415Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:55:58.415Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: entering follower state: follower="Node at 4269c970-7db9-c916-1bfa-27b56eb6f580 [Follower]" leader-address= leader-id= 2025-07-29T00:55:58.966Z [WARN] server-4269c970-7db9-c916-1bfa-27b56eb6f580: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:55:58.966Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: entering candidate state: node="Node at 4269c970-7db9-c916-1bfa-27b56eb6f580 [Candidate]" term=2 2025-07-29T00:55:58.967Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: election won: term=2 tally=2 2025-07-29T00:55:58.967Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: entering leader state: leader="Node at 4269c970-7db9-c916-1bfa-27b56eb6f580 [Leader]" 2025-07-29T00:55:58.967Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: added peer, starting replication: peer=server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 2025-07-29T00:55:58.967Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: added peer, starting replication: peer=server-f286acad-2aca-7343-6668-06092037f323 2025-07-29T00:55:58.967Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: pipelining replication: peer="{Voter server-f286acad-2aca-7343-6668-06092037f323 f286acad-2aca-7343-6668-06092037f323}" 2025-07-29T00:55:58.968Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: pipelining replication: peer="{Voter server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9}" 2025-07-29T00:55:59.972Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:55:58.967541977 +0000 UTC m=+38.656923241 (1 nodes), 552.26188ms from start of poll, 552.7964ms from cluster start. Timeout at 2025-07-29 00:55:59.972790897 +0000 UTC m=+39.662172241, 1.005249s after stability 2025-07-29T00:55:59.973Z [INFO] creating new snapshot: path=/tmp/raft79816524/snapshots/2-14-1753750559973.tmp 2025-07-29T00:55:59.973Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: copied to local snapshot: bytes=71 2025-07-29T00:55:59.974Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: snapshot restore progress: id=2-14-1753750559973 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:59.974Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: removing all old logs from log store 2025-07-29T00:55:59.974Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: compacting logs: from=1 to=2 2025-07-29T00:55:59.974Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: restored user snapshot: index=14 2025-07-29T00:55:59.974Z [ERROR] server-4269c970-7db9-c916-1bfa-27b56eb6f580: failed to get log: index=2 error="log not found" 2025-07-29T00:55:59.974Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: aborting pipeline replication: peer="{Voter server-f286acad-2aca-7343-6668-06092037f323 f286acad-2aca-7343-6668-06092037f323}" 2025-07-29T00:55:59.974Z [ERROR] server-4269c970-7db9-c916-1bfa-27b56eb6f580: failed to get log: index=2 error="log not found" 2025-07-29T00:55:59.974Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: aborting pipeline replication: peer="{Voter server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9}" 2025-07-29T00:55:59.983Z [ERROR] server-4269c970-7db9-c916-1bfa-27b56eb6f580: failed to get log: index=2 error="log not found" 2025-07-29T00:55:59.983Z [ERROR] server-4269c970-7db9-c916-1bfa-27b56eb6f580: failed to get log: index=2 error="log not found" 2025-07-29T00:55:59.983Z [INFO] creating new snapshot: path=/tmp/raft515443646/snapshots/2-14-1753750559983.tmp 2025-07-29T00:55:59.983Z [INFO] creating new snapshot: path=/tmp/raft2465316192/snapshots/2-14-1753750559983.tmp 2025-07-29T00:55:59.984Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:59.984Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:59.984Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: copied to local snapshot: bytes=71 2025-07-29T00:55:59.984Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: copied to local snapshot: bytes=71 2025-07-29T00:55:59.984Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: snapshot restore progress: id=2-14-1753750559983 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:59.984Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: removing all old logs from log store 2025-07-29T00:55:59.984Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: compacting logs: from=1 to=2 2025-07-29T00:55:59.984Z [INFO] server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9: Installed remote snapshot 2025-07-29T00:55:59.984Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: snapshot restore progress: id=2-14-1753750559983 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:55:59.984Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: removing all old logs from log store 2025-07-29T00:55:59.984Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: compacting logs: from=1 to=2 2025-07-29T00:55:59.984Z [INFO] server-f286acad-2aca-7343-6668-06092037f323: Installed remote snapshot 2025-07-29T00:55:59.984Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: pipelining replication: peer="{Voter server-c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9 c0a139f0-82fa-cf6b-32d7-a3f42fe0efa9}" 2025-07-29T00:55:59.984Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:55:59.984Z [INFO] server-4269c970-7db9-c916-1bfa-27b56eb6f580: pipelining replication: peer="{Voter server-f286acad-2aca-7343-6668-06092037f323 f286acad-2aca-7343-6668-06092037f323}" 2025-07-29T00:56:00.990Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:55:59.984823257 +0000 UTC m=+39.674204601 (2 nodes), 1µs from start of poll, 1.57007776s from cluster start. Timeout at 2025-07-29 00:56:00.990635177 +0000 UTC m=+40.680016521, 1.00581192s after stability 2025-07-29T00:56:01.002Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: aborting pipeline replication: peer="{Voter server-ffc57319-15fc-9098-c3b6-0fd48b6892c1 ffc57319-15fc-9098-c3b6-0fd48b6892c1}" 2025-07-29T00:56:01.002Z [INFO] server-8b53d449-dce0-ad01-4637-24490064a751: aborting pipeline replication: peer="{Voter server-60247c2a-43b3-4287-bf36-dc2642e078dd 60247c2a-43b3-4287-bf36-dc2642e078dd}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2025-07-29T00:56:01.003Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4 Address:5116fab1-9532-4c2a-21ea-2dc04e6ff6e4} {Suffrage:Voter ID:server-7ffc7a5a-588f-b450-8f2b-66a4807232eb Address:7ffc7a5a-588f-b450-8f2b-66a4807232eb} {Suffrage:Voter ID:server-74f099af-bbc9-04ca-613b-118b877045bd Address:74f099af-bbc9-04ca-613b-118b877045bd}]" 2025-07-29T00:56:01.003Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: entering follower state: follower="Node at 5116fab1-9532-4c2a-21ea-2dc04e6ff6e4 [Follower]" leader-address= leader-id= 2025-07-29T00:56:01.003Z [INFO] server-7ffc7a5a-588f-b450-8f2b-66a4807232eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4 Address:5116fab1-9532-4c2a-21ea-2dc04e6ff6e4} {Suffrage:Voter ID:server-7ffc7a5a-588f-b450-8f2b-66a4807232eb Address:7ffc7a5a-588f-b450-8f2b-66a4807232eb} {Suffrage:Voter ID:server-74f099af-bbc9-04ca-613b-118b877045bd Address:74f099af-bbc9-04ca-613b-118b877045bd}]" 2025-07-29T00:56:01.003Z [INFO] server-7ffc7a5a-588f-b450-8f2b-66a4807232eb: entering follower state: follower="Node at 7ffc7a5a-588f-b450-8f2b-66a4807232eb [Follower]" leader-address= leader-id= 2025-07-29T00:56:01.003Z [INFO] server-74f099af-bbc9-04ca-613b-118b877045bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4 Address:5116fab1-9532-4c2a-21ea-2dc04e6ff6e4} {Suffrage:Voter ID:server-7ffc7a5a-588f-b450-8f2b-66a4807232eb Address:7ffc7a5a-588f-b450-8f2b-66a4807232eb} {Suffrage:Voter ID:server-74f099af-bbc9-04ca-613b-118b877045bd Address:74f099af-bbc9-04ca-613b-118b877045bd}]" 2025-07-29T00:56:01.003Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:01.003Z [INFO] server-74f099af-bbc9-04ca-613b-118b877045bd: entering follower state: follower="Node at 74f099af-bbc9-04ca-613b-118b877045bd [Follower]" leader-address= leader-id= 2025-07-29T00:56:01.619Z [WARN] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:01.619Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: entering candidate state: node="Node at 5116fab1-9532-4c2a-21ea-2dc04e6ff6e4 [Candidate]" term=2 2025-07-29T00:56:01.619Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: election won: term=2 tally=2 2025-07-29T00:56:01.619Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: entering leader state: leader="Node at 5116fab1-9532-4c2a-21ea-2dc04e6ff6e4 [Leader]" 2025-07-29T00:56:01.619Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: added peer, starting replication: peer=server-7ffc7a5a-588f-b450-8f2b-66a4807232eb 2025-07-29T00:56:01.619Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: added peer, starting replication: peer=server-74f099af-bbc9-04ca-613b-118b877045bd 2025-07-29T00:56:01.619Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: pipelining replication: peer="{Voter server-74f099af-bbc9-04ca-613b-118b877045bd 74f099af-bbc9-04ca-613b-118b877045bd}" 2025-07-29T00:56:01.619Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: pipelining replication: peer="{Voter server-7ffc7a5a-588f-b450-8f2b-66a4807232eb 7ffc7a5a-588f-b450-8f2b-66a4807232eb}" 2025-07-29T00:56:02.625Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:01.619501457 +0000 UTC m=+41.308882801 (1 nodes), 615.81204ms from start of poll, 616.3448ms from cluster start. Timeout at 2025-07-29 00:56:02.625158337 +0000 UTC m=+42.314539641, 1.00565684s after stability 2025-07-29T00:56:02.626Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: starting snapshot up to: index=12 2025-07-29T00:56:02.626Z [INFO] creating new snapshot: path=/tmp/raft3205213026/snapshots/2-12-1753750562626.tmp 2025-07-29T00:56:02.626Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: snapshot complete up to: index=12 2025-07-29T00:56:02.627Z [INFO] creating new snapshot: path=/tmp/raft3205213026/snapshots/2-23-1753750562627.tmp 2025-07-29T00:56:02.628Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: copied to local snapshot: bytes=71 2025-07-29T00:56:02.628Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: snapshot restore progress: id=2-23-1753750562627 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:02.628Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: restored user snapshot: index=23 2025-07-29T00:56:02.628Z [ERROR] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: failed to get log: index=23 error="log not found" 2025-07-29T00:56:02.628Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: aborting pipeline replication: peer="{Voter server-74f099af-bbc9-04ca-613b-118b877045bd 74f099af-bbc9-04ca-613b-118b877045bd}" 2025-07-29T00:56:02.628Z [ERROR] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: failed to get log: index=23 error="log not found" 2025-07-29T00:56:02.628Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: aborting pipeline replication: peer="{Voter server-7ffc7a5a-588f-b450-8f2b-66a4807232eb 7ffc7a5a-588f-b450-8f2b-66a4807232eb}" 2025-07-29T00:56:02.637Z [ERROR] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: failed to get log: index=23 error="log not found" 2025-07-29T00:56:02.637Z [INFO] creating new snapshot: path=/tmp/raft4260492197/snapshots/2-23-1753750562637.tmp 2025-07-29T00:56:02.638Z [INFO] server-7ffc7a5a-588f-b450-8f2b-66a4807232eb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:02.638Z [INFO] server-7ffc7a5a-588f-b450-8f2b-66a4807232eb: copied to local snapshot: bytes=71 2025-07-29T00:56:02.638Z [ERROR] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: failed to get log: index=23 error="log not found" 2025-07-29T00:56:02.638Z [INFO] server-7ffc7a5a-588f-b450-8f2b-66a4807232eb: snapshot restore progress: id=2-23-1753750562637 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:02.638Z [INFO] server-7ffc7a5a-588f-b450-8f2b-66a4807232eb: Installed remote snapshot 2025-07-29T00:56:02.638Z [WARN] server-7ffc7a5a-588f-b450-8f2b-66a4807232eb: skipping application of old log: index=22 2025-07-29T00:56:02.638Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: pipelining replication: peer="{Voter server-7ffc7a5a-588f-b450-8f2b-66a4807232eb 7ffc7a5a-588f-b450-8f2b-66a4807232eb}" 2025-07-29T00:56:02.638Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:02.638Z [INFO] creating new snapshot: path=/tmp/raft3868104109/snapshots/2-23-1753750562638.tmp 2025-07-29T00:56:02.639Z [INFO] server-74f099af-bbc9-04ca-613b-118b877045bd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:02.640Z [INFO] server-74f099af-bbc9-04ca-613b-118b877045bd: copied to local snapshot: bytes=71 2025-07-29T00:56:02.640Z [INFO] server-74f099af-bbc9-04ca-613b-118b877045bd: snapshot restore progress: id=2-23-1753750562638 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:02.640Z [INFO] server-74f099af-bbc9-04ca-613b-118b877045bd: Installed remote snapshot 2025-07-29T00:56:02.641Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: pipelining replication: peer="{Voter server-74f099af-bbc9-04ca-613b-118b877045bd 74f099af-bbc9-04ca-613b-118b877045bd}" 2025-07-29T00:56:03.644Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:02.638665217 +0000 UTC m=+42.328046561 (2 nodes), 960ns from start of poll, 1.63550856s from cluster start. Timeout at 2025-07-29 00:56:03.644330457 +0000 UTC m=+43.333711801, 1.00566524s after stability 2025-07-29T00:56:03.661Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: aborting pipeline replication: peer="{Voter server-74f099af-bbc9-04ca-613b-118b877045bd 74f099af-bbc9-04ca-613b-118b877045bd}" 2025-07-29T00:56:03.661Z [INFO] server-5116fab1-9532-4c2a-21ea-2dc04e6ff6e4: aborting pipeline replication: peer="{Voter server-7ffc7a5a-588f-b450-8f2b-66a4807232eb 7ffc7a5a-588f-b450-8f2b-66a4807232eb}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2025-07-29T00:56:03.662Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 Address:2010c9b3-e57a-8ea0-aca5-9ac31b946893} {Suffrage:Voter ID:server-f4e35cfa-1850-e156-78d0-4d4230fbcb39 Address:f4e35cfa-1850-e156-78d0-4d4230fbcb39} {Suffrage:Voter ID:server-8ad8719c-de66-c6f3-9041-3325113b624f Address:8ad8719c-de66-c6f3-9041-3325113b624f}]" 2025-07-29T00:56:03.662Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: entering follower state: follower="Node at 2010c9b3-e57a-8ea0-aca5-9ac31b946893 [Follower]" leader-address= leader-id= 2025-07-29T00:56:03.662Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 Address:2010c9b3-e57a-8ea0-aca5-9ac31b946893} {Suffrage:Voter ID:server-f4e35cfa-1850-e156-78d0-4d4230fbcb39 Address:f4e35cfa-1850-e156-78d0-4d4230fbcb39} {Suffrage:Voter ID:server-8ad8719c-de66-c6f3-9041-3325113b624f Address:8ad8719c-de66-c6f3-9041-3325113b624f}]" 2025-07-29T00:56:03.662Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: entering follower state: follower="Node at f4e35cfa-1850-e156-78d0-4d4230fbcb39 [Follower]" leader-address= leader-id= 2025-07-29T00:56:03.662Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 Address:2010c9b3-e57a-8ea0-aca5-9ac31b946893} {Suffrage:Voter ID:server-f4e35cfa-1850-e156-78d0-4d4230fbcb39 Address:f4e35cfa-1850-e156-78d0-4d4230fbcb39} {Suffrage:Voter ID:server-8ad8719c-de66-c6f3-9041-3325113b624f Address:8ad8719c-de66-c6f3-9041-3325113b624f}]" 2025-07-29T00:56:03.662Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:03.662Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: entering follower state: follower="Node at 8ad8719c-de66-c6f3-9041-3325113b624f [Follower]" leader-address= leader-id= 2025-07-29T00:56:04.337Z [WARN] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:04.337Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: entering candidate state: node="Node at f4e35cfa-1850-e156-78d0-4d4230fbcb39 [Candidate]" term=2 2025-07-29T00:56:04.337Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: election won: term=2 tally=2 2025-07-29T00:56:04.337Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: entering leader state: leader="Node at f4e35cfa-1850-e156-78d0-4d4230fbcb39 [Leader]" 2025-07-29T00:56:04.337Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: added peer, starting replication: peer=server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 2025-07-29T00:56:04.337Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: added peer, starting replication: peer=server-8ad8719c-de66-c6f3-9041-3325113b624f 2025-07-29T00:56:04.337Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: pipelining replication: peer="{Voter server-8ad8719c-de66-c6f3-9041-3325113b624f 8ad8719c-de66-c6f3-9041-3325113b624f}" 2025-07-29T00:56:04.337Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: pipelining replication: peer="{Voter server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 2010c9b3-e57a-8ea0-aca5-9ac31b946893}" 2025-07-29T00:56:05.342Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:04.337708937 +0000 UTC m=+44.027090281 (1 nodes), 675.2062ms from start of poll, 675.68128ms from cluster start. Timeout at 2025-07-29 00:56:05.342802817 +0000 UTC m=+45.032184161, 1.00509388s after stability 2025-07-29T00:56:05.343Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: starting snapshot up to: index=12 2025-07-29T00:56:05.343Z [INFO] creating new snapshot: path=/tmp/raft651736844/snapshots/2-12-1753750565343.tmp 2025-07-29T00:56:05.344Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: snapshot complete up to: index=12 2025-07-29T00:56:05.345Z [INFO] creating new snapshot: path=/tmp/raft651736844/snapshots/2-23-1753750565345.tmp 2025-07-29T00:56:05.346Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: copied to local snapshot: bytes=71 2025-07-29T00:56:05.347Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: snapshot restore progress: id=2-23-1753750565345 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:05.347Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: removing all old logs from log store 2025-07-29T00:56:05.347Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: compacting logs: from=1 to=22 2025-07-29T00:56:05.347Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: restored user snapshot: index=23 2025-07-29T00:56:05.347Z [ERROR] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: failed to get log: index=22 error="log not found" 2025-07-29T00:56:05.347Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: aborting pipeline replication: peer="{Voter server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 2010c9b3-e57a-8ea0-aca5-9ac31b946893}" 2025-07-29T00:56:05.347Z [ERROR] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: failed to get log: index=22 error="log not found" 2025-07-29T00:56:05.347Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: aborting pipeline replication: peer="{Voter server-8ad8719c-de66-c6f3-9041-3325113b624f 8ad8719c-de66-c6f3-9041-3325113b624f}" 2025-07-29T00:56:05.354Z [ERROR] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: failed to get log: index=22 error="log not found" 2025-07-29T00:56:05.355Z [INFO] creating new snapshot: path=/tmp/raft1467397868/snapshots/2-23-1753750565355.tmp 2025-07-29T00:56:05.355Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:05.355Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: copied to local snapshot: bytes=71 2025-07-29T00:56:05.355Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: snapshot restore progress: id=2-23-1753750565355 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:05.355Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: removing all old logs from log store 2025-07-29T00:56:05.355Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: compacting logs: from=1 to=22 2025-07-29T00:56:05.355Z [INFO] server-8ad8719c-de66-c6f3-9041-3325113b624f: Installed remote snapshot 2025-07-29T00:56:05.356Z [WARN] server-8ad8719c-de66-c6f3-9041-3325113b624f: skipping application of old log: index=22 2025-07-29T00:56:05.356Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: pipelining replication: peer="{Voter server-8ad8719c-de66-c6f3-9041-3325113b624f 8ad8719c-de66-c6f3-9041-3325113b624f}" 2025-07-29T00:56:05.356Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:05.358Z [ERROR] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: failed to get log: index=22 error="log not found" 2025-07-29T00:56:05.358Z [INFO] creating new snapshot: path=/tmp/raft555017693/snapshots/2-23-1753750565358.tmp 2025-07-29T00:56:05.358Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:05.359Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: copied to local snapshot: bytes=71 2025-07-29T00:56:05.359Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: snapshot restore progress: id=2-23-1753750565358 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:05.359Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: removing all old logs from log store 2025-07-29T00:56:05.359Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: compacting logs: from=1 to=22 2025-07-29T00:56:05.359Z [INFO] server-2010c9b3-e57a-8ea0-aca5-9ac31b946893: Installed remote snapshot 2025-07-29T00:56:05.359Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: pipelining replication: peer="{Voter server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 2010c9b3-e57a-8ea0-aca5-9ac31b946893}" 2025-07-29T00:56:06.361Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:05.356172337 +0000 UTC m=+45.045553681 (2 nodes), 1µs from start of poll, 1.69414468s from cluster start. Timeout at 2025-07-29 00:56:06.361257257 +0000 UTC m=+46.050638601, 1.00508492s after stability 2025-07-29T00:56:06.373Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: aborting pipeline replication: peer="{Voter server-2010c9b3-e57a-8ea0-aca5-9ac31b946893 2010c9b3-e57a-8ea0-aca5-9ac31b946893}" 2025-07-29T00:56:06.373Z [INFO] server-f4e35cfa-1850-e156-78d0-4d4230fbcb39: aborting pipeline replication: peer="{Voter server-8ad8719c-de66-c6f3-9041-3325113b624f 8ad8719c-de66-c6f3-9041-3325113b624f}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2025-07-29T00:56:06.374Z [INFO] server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e Address:6a9d0c70-907c-bed7-60bb-7a01fa92f23e} {Suffrage:Voter ID:server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe Address:c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe} {Suffrage:Voter ID:server-92aa7e55-16a5-0387-69d2-f321aae30392 Address:92aa7e55-16a5-0387-69d2-f321aae30392}]" 2025-07-29T00:56:06.374Z [INFO] server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e: entering follower state: follower="Node at 6a9d0c70-907c-bed7-60bb-7a01fa92f23e [Follower]" leader-address= leader-id= 2025-07-29T00:56:06.374Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e Address:6a9d0c70-907c-bed7-60bb-7a01fa92f23e} {Suffrage:Voter ID:server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe Address:c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe} {Suffrage:Voter ID:server-92aa7e55-16a5-0387-69d2-f321aae30392 Address:92aa7e55-16a5-0387-69d2-f321aae30392}]" 2025-07-29T00:56:06.374Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: entering follower state: follower="Node at c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe [Follower]" leader-address= leader-id= 2025-07-29T00:56:06.374Z [INFO] server-92aa7e55-16a5-0387-69d2-f321aae30392: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e Address:6a9d0c70-907c-bed7-60bb-7a01fa92f23e} {Suffrage:Voter ID:server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe Address:c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe} {Suffrage:Voter ID:server-92aa7e55-16a5-0387-69d2-f321aae30392 Address:92aa7e55-16a5-0387-69d2-f321aae30392}]" 2025-07-29T00:56:06.374Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:06.374Z [INFO] server-92aa7e55-16a5-0387-69d2-f321aae30392: entering follower state: follower="Node at 92aa7e55-16a5-0387-69d2-f321aae30392 [Follower]" leader-address= leader-id= 2025-07-29T00:56:06.904Z [WARN] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:06.904Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: entering candidate state: node="Node at c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe [Candidate]" term=2 2025-07-29T00:56:06.904Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: election won: term=2 tally=2 2025-07-29T00:56:06.904Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: entering leader state: leader="Node at c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe [Leader]" 2025-07-29T00:56:06.904Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: added peer, starting replication: peer=server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e 2025-07-29T00:56:06.904Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: added peer, starting replication: peer=server-92aa7e55-16a5-0387-69d2-f321aae30392 2025-07-29T00:56:06.904Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: pipelining replication: peer="{Voter server-92aa7e55-16a5-0387-69d2-f321aae30392 92aa7e55-16a5-0387-69d2-f321aae30392}" 2025-07-29T00:56:06.904Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: pipelining replication: peer="{Voter server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e 6a9d0c70-907c-bed7-60bb-7a01fa92f23e}" 2025-07-29T00:56:07.909Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:06.904397097 +0000 UTC m=+46.593778441 (1 nodes), 529.5652ms from start of poll, 530.03308ms from cluster start. Timeout at 2025-07-29 00:56:07.909890497 +0000 UTC m=+47.599271841, 1.0054934s after stability 2025-07-29T00:56:07.910Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: starting snapshot up to: index=12 2025-07-29T00:56:07.910Z [INFO] creating new snapshot: path=/tmp/raft3570905646/snapshots/2-12-1753750567910.tmp 2025-07-29T00:56:07.911Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: snapshot complete up to: index=12 2025-07-29T00:56:07.912Z [INFO] server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e Address:49ae46e0-6405-f0e7-87f8-f8cb3776a48e} {Suffrage:Voter ID:server-6c01faca-376d-ba63-197f-bbafe9760446 Address:6c01faca-376d-ba63-197f-bbafe9760446} {Suffrage:Voter ID:server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553 Address:80352aa0-3f8e-bc41-81cd-8fdb6da7c553}]" 2025-07-29T00:56:07.912Z [INFO] server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e: entering follower state: follower="Node at 49ae46e0-6405-f0e7-87f8-f8cb3776a48e [Follower]" leader-address= leader-id= 2025-07-29T00:56:07.912Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e Address:49ae46e0-6405-f0e7-87f8-f8cb3776a48e} {Suffrage:Voter ID:server-6c01faca-376d-ba63-197f-bbafe9760446 Address:6c01faca-376d-ba63-197f-bbafe9760446} {Suffrage:Voter ID:server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553 Address:80352aa0-3f8e-bc41-81cd-8fdb6da7c553}]" 2025-07-29T00:56:07.912Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: entering follower state: follower="Node at 6c01faca-376d-ba63-197f-bbafe9760446 [Follower]" leader-address= leader-id= 2025-07-29T00:56:07.912Z [INFO] server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e Address:49ae46e0-6405-f0e7-87f8-f8cb3776a48e} {Suffrage:Voter ID:server-6c01faca-376d-ba63-197f-bbafe9760446 Address:6c01faca-376d-ba63-197f-bbafe9760446} {Suffrage:Voter ID:server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553 Address:80352aa0-3f8e-bc41-81cd-8fdb6da7c553}]" 2025-07-29T00:56:07.912Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:07.912Z [INFO] server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553: entering follower state: follower="Node at 80352aa0-3f8e-bc41-81cd-8fdb6da7c553 [Follower]" leader-address= leader-id= 2025-07-29T00:56:08.423Z [WARN] server-6c01faca-376d-ba63-197f-bbafe9760446: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:08.423Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: entering candidate state: node="Node at 6c01faca-376d-ba63-197f-bbafe9760446 [Candidate]" term=2 2025-07-29T00:56:08.424Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: election won: term=2 tally=2 2025-07-29T00:56:08.424Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: entering leader state: leader="Node at 6c01faca-376d-ba63-197f-bbafe9760446 [Leader]" 2025-07-29T00:56:08.424Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: added peer, starting replication: peer=server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e 2025-07-29T00:56:08.424Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: added peer, starting replication: peer=server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553 2025-07-29T00:56:08.424Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: pipelining replication: peer="{Voter server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e 49ae46e0-6405-f0e7-87f8-f8cb3776a48e}" 2025-07-29T00:56:08.424Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: pipelining replication: peer="{Voter server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553 80352aa0-3f8e-bc41-81cd-8fdb6da7c553}" 2025-07-29T00:56:09.429Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:08.424092057 +0000 UTC m=+48.113473401 (1 nodes), 511.51952ms from start of poll, 511.9686ms from cluster start. Timeout at 2025-07-29 00:56:09.429238217 +0000 UTC m=+49.118619561, 1.00514616s after stability 2025-07-29T00:56:09.429Z [INFO] creating new snapshot: path=/tmp/raft3262283676/snapshots/2-15-1753750569429.tmp 2025-07-29T00:56:09.430Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: copied to local snapshot: bytes=71 2025-07-29T00:56:09.430Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: snapshot restore progress: id=2-15-1753750569429 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:09.430Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: restored user snapshot: index=15 2025-07-29T00:56:09.430Z [ERROR] server-6c01faca-376d-ba63-197f-bbafe9760446: failed to get log: index=3 error="log not found" 2025-07-29T00:56:09.430Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: aborting pipeline replication: peer="{Voter server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553 80352aa0-3f8e-bc41-81cd-8fdb6da7c553}" 2025-07-29T00:56:09.430Z [ERROR] server-6c01faca-376d-ba63-197f-bbafe9760446: failed to get log: index=3 error="log not found" 2025-07-29T00:56:09.430Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: aborting pipeline replication: peer="{Voter server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e 49ae46e0-6405-f0e7-87f8-f8cb3776a48e}" 2025-07-29T00:56:09.437Z [ERROR] server-6c01faca-376d-ba63-197f-bbafe9760446: failed to get log: index=3 error="log not found" 2025-07-29T00:56:09.437Z [INFO] creating new snapshot: path=/tmp/raft1531014120/snapshots/2-15-1753750569437.tmp 2025-07-29T00:56:09.437Z [ERROR] server-6c01faca-376d-ba63-197f-bbafe9760446: failed to get log: index=3 error="log not found" 2025-07-29T00:56:09.438Z [INFO] server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:09.438Z [INFO] creating new snapshot: path=/tmp/raft2100188135/snapshots/2-15-1753750569438.tmp 2025-07-29T00:56:09.438Z [INFO] server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e: copied to local snapshot: bytes=71 2025-07-29T00:56:09.438Z [INFO] server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:09.438Z [INFO] server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e: snapshot restore progress: id=2-15-1753750569437 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:09.438Z [INFO] server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e: Installed remote snapshot 2025-07-29T00:56:09.438Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: pipelining replication: peer="{Voter server-49ae46e0-6405-f0e7-87f8-f8cb3776a48e 49ae46e0-6405-f0e7-87f8-f8cb3776a48e}" 2025-07-29T00:56:09.438Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:09.438Z [INFO] server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553: copied to local snapshot: bytes=71 2025-07-29T00:56:09.439Z [INFO] server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553: snapshot restore progress: id=2-15-1753750569438 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:09.439Z [INFO] server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553: Installed remote snapshot 2025-07-29T00:56:09.439Z [INFO] server-6c01faca-376d-ba63-197f-bbafe9760446: pipelining replication: peer="{Voter server-80352aa0-3f8e-bc41-81cd-8fdb6da7c553 80352aa0-3f8e-bc41-81cd-8fdb6da7c553}" 2025-07-29T00:56:10.444Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:09.438919977 +0000 UTC m=+49.128301321 (2 nodes), 1.44µs from start of poll, 1.52679652s from cluster start. Timeout at 2025-07-29 00:56:10.444798777 +0000 UTC m=+50.134180121, 1.0058788s after stability 2025-07-29T00:56:10.457Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: aborting pipeline replication: peer="{Voter server-92aa7e55-16a5-0387-69d2-f321aae30392 92aa7e55-16a5-0387-69d2-f321aae30392}" 2025-07-29T00:56:10.457Z [INFO] server-c72edc4e-15a8-a9a2-7e7a-094e29cc9cbe: aborting pipeline replication: peer="{Voter server-6a9d0c70-907c-bed7-60bb-7a01fa92f23e 6a9d0c70-907c-bed7-60bb-7a01fa92f23e}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2025-07-29T00:56:10.458Z [INFO] server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0 Address:a6f69d8d-f8f5-efec-0a06-5ea43835f0d0} {Suffrage:Voter ID:server-044de399-a416-72ff-95e6-e5e72215aaeb Address:044de399-a416-72ff-95e6-e5e72215aaeb} {Suffrage:Voter ID:server-f96a4bcf-8b58-11f8-0982-582af0a3b05b Address:f96a4bcf-8b58-11f8-0982-582af0a3b05b}]" 2025-07-29T00:56:10.458Z [INFO] server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0: entering follower state: follower="Node at a6f69d8d-f8f5-efec-0a06-5ea43835f0d0 [Follower]" leader-address= leader-id= 2025-07-29T00:56:10.458Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0 Address:a6f69d8d-f8f5-efec-0a06-5ea43835f0d0} {Suffrage:Voter ID:server-044de399-a416-72ff-95e6-e5e72215aaeb Address:044de399-a416-72ff-95e6-e5e72215aaeb} {Suffrage:Voter ID:server-f96a4bcf-8b58-11f8-0982-582af0a3b05b Address:f96a4bcf-8b58-11f8-0982-582af0a3b05b}]" 2025-07-29T00:56:10.458Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: entering follower state: follower="Node at 044de399-a416-72ff-95e6-e5e72215aaeb [Follower]" leader-address= leader-id= 2025-07-29T00:56:10.458Z [INFO] server-f96a4bcf-8b58-11f8-0982-582af0a3b05b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0 Address:a6f69d8d-f8f5-efec-0a06-5ea43835f0d0} {Suffrage:Voter ID:server-044de399-a416-72ff-95e6-e5e72215aaeb Address:044de399-a416-72ff-95e6-e5e72215aaeb} {Suffrage:Voter ID:server-f96a4bcf-8b58-11f8-0982-582af0a3b05b Address:f96a4bcf-8b58-11f8-0982-582af0a3b05b}]" 2025-07-29T00:56:10.458Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:10.458Z [INFO] server-f96a4bcf-8b58-11f8-0982-582af0a3b05b: entering follower state: follower="Node at f96a4bcf-8b58-11f8-0982-582af0a3b05b [Follower]" leader-address= leader-id= 2025-07-29T00:56:11.105Z [WARN] server-044de399-a416-72ff-95e6-e5e72215aaeb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:11.105Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: entering candidate state: node="Node at 044de399-a416-72ff-95e6-e5e72215aaeb [Candidate]" term=2 2025-07-29T00:56:11.105Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: election won: term=2 tally=2 2025-07-29T00:56:11.105Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: entering leader state: leader="Node at 044de399-a416-72ff-95e6-e5e72215aaeb [Leader]" 2025-07-29T00:56:11.105Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: added peer, starting replication: peer=server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0 2025-07-29T00:56:11.105Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: added peer, starting replication: peer=server-f96a4bcf-8b58-11f8-0982-582af0a3b05b 2025-07-29T00:56:11.105Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: pipelining replication: peer="{Voter server-f96a4bcf-8b58-11f8-0982-582af0a3b05b f96a4bcf-8b58-11f8-0982-582af0a3b05b}" 2025-07-29T00:56:11.105Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: pipelining replication: peer="{Voter server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0 a6f69d8d-f8f5-efec-0a06-5ea43835f0d0}" 2025-07-29T00:56:12.111Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:11.105667297 +0000 UTC m=+50.795048641 (1 nodes), 647.23484ms from start of poll, 647.68952ms from cluster start. Timeout at 2025-07-29 00:56:12.111285897 +0000 UTC m=+51.800667241, 1.0056186s after stability 2025-07-29T00:56:12.111Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: starting snapshot up to: index=12 2025-07-29T00:56:12.111Z [INFO] creating new snapshot: path=/tmp/raft368211275/snapshots/2-12-1753750572111.tmp 2025-07-29T00:56:12.112Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: snapshot complete up to: index=12 2025-07-29T00:56:12.113Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08329cb3-436e-9246-4201-cc47fb78d85d Address:08329cb3-436e-9246-4201-cc47fb78d85d} {Suffrage:Voter ID:server-2f224177-b01d-8862-1050-2ff8ac2aaf56 Address:2f224177-b01d-8862-1050-2ff8ac2aaf56} {Suffrage:Voter ID:server-752b850d-f8ca-a0cb-e33a-27457d028e14 Address:752b850d-f8ca-a0cb-e33a-27457d028e14}]" 2025-07-29T00:56:12.113Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: entering follower state: follower="Node at 08329cb3-436e-9246-4201-cc47fb78d85d [Follower]" leader-address= leader-id= 2025-07-29T00:56:12.113Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08329cb3-436e-9246-4201-cc47fb78d85d Address:08329cb3-436e-9246-4201-cc47fb78d85d} {Suffrage:Voter ID:server-2f224177-b01d-8862-1050-2ff8ac2aaf56 Address:2f224177-b01d-8862-1050-2ff8ac2aaf56} {Suffrage:Voter ID:server-752b850d-f8ca-a0cb-e33a-27457d028e14 Address:752b850d-f8ca-a0cb-e33a-27457d028e14}]" 2025-07-29T00:56:12.114Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: entering follower state: follower="Node at 2f224177-b01d-8862-1050-2ff8ac2aaf56 [Follower]" leader-address= leader-id= 2025-07-29T00:56:12.114Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08329cb3-436e-9246-4201-cc47fb78d85d Address:08329cb3-436e-9246-4201-cc47fb78d85d} {Suffrage:Voter ID:server-2f224177-b01d-8862-1050-2ff8ac2aaf56 Address:2f224177-b01d-8862-1050-2ff8ac2aaf56} {Suffrage:Voter ID:server-752b850d-f8ca-a0cb-e33a-27457d028e14 Address:752b850d-f8ca-a0cb-e33a-27457d028e14}]" 2025-07-29T00:56:12.114Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:12.114Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: entering follower state: follower="Node at 752b850d-f8ca-a0cb-e33a-27457d028e14 [Follower]" leader-address= leader-id= 2025-07-29T00:56:12.671Z [WARN] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:12.671Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: entering candidate state: node="Node at 2f224177-b01d-8862-1050-2ff8ac2aaf56 [Candidate]" term=2 2025-07-29T00:56:12.671Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: election won: term=2 tally=2 2025-07-29T00:56:12.671Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: entering leader state: leader="Node at 2f224177-b01d-8862-1050-2ff8ac2aaf56 [Leader]" 2025-07-29T00:56:12.671Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: added peer, starting replication: peer=server-08329cb3-436e-9246-4201-cc47fb78d85d 2025-07-29T00:56:12.671Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: added peer, starting replication: peer=server-752b850d-f8ca-a0cb-e33a-27457d028e14 2025-07-29T00:56:12.671Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: pipelining replication: peer="{Voter server-08329cb3-436e-9246-4201-cc47fb78d85d 08329cb3-436e-9246-4201-cc47fb78d85d}" 2025-07-29T00:56:12.671Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: pipelining replication: peer="{Voter server-752b850d-f8ca-a0cb-e33a-27457d028e14 752b850d-f8ca-a0cb-e33a-27457d028e14}" 2025-07-29T00:56:13.677Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:12.671827217 +0000 UTC m=+52.361208561 (1 nodes), 557.73456ms from start of poll, 558.19676ms from cluster start. Timeout at 2025-07-29 00:56:13.677893977 +0000 UTC m=+53.367275321, 1.00606676s after stability 2025-07-29T00:56:13.678Z [INFO] creating new snapshot: path=/tmp/raft3824855631/snapshots/2-15-1753750573678.tmp 2025-07-29T00:56:13.678Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: copied to local snapshot: bytes=71 2025-07-29T00:56:13.679Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: snapshot restore progress: id=2-15-1753750573678 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:13.679Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: removing all old logs from log store 2025-07-29T00:56:13.679Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: compacting logs: from=1 to=2 2025-07-29T00:56:13.679Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: restored user snapshot: index=15 2025-07-29T00:56:13.679Z [ERROR] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: failed to get log: index=2 error="log not found" 2025-07-29T00:56:13.679Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: aborting pipeline replication: peer="{Voter server-752b850d-f8ca-a0cb-e33a-27457d028e14 752b850d-f8ca-a0cb-e33a-27457d028e14}" 2025-07-29T00:56:13.679Z [ERROR] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: failed to get log: index=2 error="log not found" 2025-07-29T00:56:13.679Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: aborting pipeline replication: peer="{Voter server-08329cb3-436e-9246-4201-cc47fb78d85d 08329cb3-436e-9246-4201-cc47fb78d85d}" 2025-07-29T00:56:13.686Z [ERROR] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: failed to get log: index=2 error="log not found" 2025-07-29T00:56:13.687Z [INFO] creating new snapshot: path=/tmp/raft3214177703/snapshots/2-15-1753750573687.tmp 2025-07-29T00:56:13.687Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:13.687Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: copied to local snapshot: bytes=71 2025-07-29T00:56:13.687Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: snapshot restore progress: id=2-15-1753750573687 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:13.687Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: removing all old logs from log store 2025-07-29T00:56:13.687Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: compacting logs: from=1 to=2 2025-07-29T00:56:13.687Z [INFO] server-752b850d-f8ca-a0cb-e33a-27457d028e14: Installed remote snapshot 2025-07-29T00:56:13.687Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: pipelining replication: peer="{Voter server-752b850d-f8ca-a0cb-e33a-27457d028e14 752b850d-f8ca-a0cb-e33a-27457d028e14}" 2025-07-29T00:56:13.688Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:13.688Z [ERROR] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: failed to get log: index=2 error="log not found" 2025-07-29T00:56:13.689Z [INFO] creating new snapshot: path=/tmp/raft2221703437/snapshots/2-15-1753750573689.tmp 2025-07-29T00:56:13.689Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:13.689Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: copied to local snapshot: bytes=71 2025-07-29T00:56:13.690Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: snapshot restore progress: id=2-15-1753750573689 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:13.690Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: removing all old logs from log store 2025-07-29T00:56:13.690Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: compacting logs: from=1 to=2 2025-07-29T00:56:13.690Z [INFO] server-08329cb3-436e-9246-4201-cc47fb78d85d: Installed remote snapshot 2025-07-29T00:56:13.690Z [INFO] server-2f224177-b01d-8862-1050-2ff8ac2aaf56: pipelining replication: peer="{Voter server-08329cb3-436e-9246-4201-cc47fb78d85d 08329cb3-436e-9246-4201-cc47fb78d85d}" 2025-07-29T00:56:14.694Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:13.688041817 +0000 UTC m=+53.377423161 (2 nodes), 920ns from start of poll, 1.57441136s from cluster start. Timeout at 2025-07-29 00:56:14.694019457 +0000 UTC m=+54.383400801, 1.00597764s after stability 2025-07-29T00:56:14.706Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: aborting pipeline replication: peer="{Voter server-f96a4bcf-8b58-11f8-0982-582af0a3b05b f96a4bcf-8b58-11f8-0982-582af0a3b05b}" 2025-07-29T00:56:14.706Z [INFO] server-044de399-a416-72ff-95e6-e5e72215aaeb: aborting pipeline replication: peer="{Voter server-a6f69d8d-f8f5-efec-0a06-5ea43835f0d0 a6f69d8d-f8f5-efec-0a06-5ea43835f0d0}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2025-07-29T00:56:14.707Z [INFO] server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab Address:9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab} {Suffrage:Voter ID:server-345d8521-a860-4b8b-7d11-7ba45e0ab286 Address:345d8521-a860-4b8b-7d11-7ba45e0ab286} {Suffrage:Voter ID:server-e42b4634-978d-c978-1216-a64dfb6e1224 Address:e42b4634-978d-c978-1216-a64dfb6e1224}]" 2025-07-29T00:56:14.707Z [INFO] server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab: entering follower state: follower="Node at 9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab [Follower]" leader-address= leader-id= 2025-07-29T00:56:14.707Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab Address:9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab} {Suffrage:Voter ID:server-345d8521-a860-4b8b-7d11-7ba45e0ab286 Address:345d8521-a860-4b8b-7d11-7ba45e0ab286} {Suffrage:Voter ID:server-e42b4634-978d-c978-1216-a64dfb6e1224 Address:e42b4634-978d-c978-1216-a64dfb6e1224}]" 2025-07-29T00:56:14.707Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: entering follower state: follower="Node at 345d8521-a860-4b8b-7d11-7ba45e0ab286 [Follower]" leader-address= leader-id= 2025-07-29T00:56:14.707Z [INFO] server-e42b4634-978d-c978-1216-a64dfb6e1224: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab Address:9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab} {Suffrage:Voter ID:server-345d8521-a860-4b8b-7d11-7ba45e0ab286 Address:345d8521-a860-4b8b-7d11-7ba45e0ab286} {Suffrage:Voter ID:server-e42b4634-978d-c978-1216-a64dfb6e1224 Address:e42b4634-978d-c978-1216-a64dfb6e1224}]" 2025-07-29T00:56:14.707Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:14.707Z [INFO] server-e42b4634-978d-c978-1216-a64dfb6e1224: entering follower state: follower="Node at e42b4634-978d-c978-1216-a64dfb6e1224 [Follower]" leader-address= leader-id= 2025-07-29T00:56:15.364Z [WARN] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:15.364Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: entering candidate state: node="Node at 345d8521-a860-4b8b-7d11-7ba45e0ab286 [Candidate]" term=2 2025-07-29T00:56:15.364Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: election won: term=2 tally=2 2025-07-29T00:56:15.364Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: entering leader state: leader="Node at 345d8521-a860-4b8b-7d11-7ba45e0ab286 [Leader]" 2025-07-29T00:56:15.364Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: added peer, starting replication: peer=server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab 2025-07-29T00:56:15.364Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: added peer, starting replication: peer=server-e42b4634-978d-c978-1216-a64dfb6e1224 2025-07-29T00:56:15.364Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: pipelining replication: peer="{Voter server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab 9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab}" 2025-07-29T00:56:15.365Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: pipelining replication: peer="{Voter server-e42b4634-978d-c978-1216-a64dfb6e1224 e42b4634-978d-c978-1216-a64dfb6e1224}" 2025-07-29T00:56:16.369Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:15.364682617 +0000 UTC m=+55.054063961 (1 nodes), 657.00236ms from start of poll, 657.56244ms from cluster start. Timeout at 2025-07-29 00:56:16.369738057 +0000 UTC m=+56.059119441, 1.00505548s after stability 2025-07-29T00:56:16.371Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: starting snapshot up to: index=12 2025-07-29T00:56:16.371Z [INFO] creating new snapshot: path=/tmp/raft2784404099/snapshots/2-12-1753750576371.tmp 2025-07-29T00:56:16.372Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: snapshot complete up to: index=12 2025-07-29T00:56:16.373Z [INFO] creating new snapshot: path=/tmp/raft2784404099/snapshots/2-113-1753750576373.tmp 2025-07-29T00:56:16.374Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: copied to local snapshot: bytes=71 2025-07-29T00:56:16.374Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: snapshot restore progress: id=2-113-1753750576373 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:16.374Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: restored user snapshot: index=113 2025-07-29T00:56:16.374Z [ERROR] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: failed to get log: index=23 error="log not found" 2025-07-29T00:56:16.374Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: aborting pipeline replication: peer="{Voter server-e42b4634-978d-c978-1216-a64dfb6e1224 e42b4634-978d-c978-1216-a64dfb6e1224}" 2025-07-29T00:56:16.374Z [ERROR] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: failed to get log: index=23 error="log not found" 2025-07-29T00:56:16.374Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: aborting pipeline replication: peer="{Voter server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab 9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab}" 2025-07-29T00:56:16.381Z [ERROR] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: failed to get log: index=23 error="log not found" 2025-07-29T00:56:16.381Z [INFO] creating new snapshot: path=/tmp/raft259815736/snapshots/2-113-1753750576381.tmp 2025-07-29T00:56:16.382Z [INFO] server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:16.382Z [ERROR] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: failed to get log: index=23 error="log not found" 2025-07-29T00:56:16.382Z [INFO] server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab: copied to local snapshot: bytes=71 2025-07-29T00:56:16.382Z [INFO] creating new snapshot: path=/tmp/raft2872552282/snapshots/2-113-1753750576382.tmp 2025-07-29T00:56:16.382Z [INFO] server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab: snapshot restore progress: id=2-113-1753750576381 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:16.383Z [INFO] server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab: Installed remote snapshot 2025-07-29T00:56:16.383Z [INFO] server-e42b4634-978d-c978-1216-a64dfb6e1224: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:16.383Z [WARN] server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab: skipping application of old log: index=22 2025-07-29T00:56:16.383Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: pipelining replication: peer="{Voter server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab 9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab}" 2025-07-29T00:56:16.383Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:16.383Z [INFO] server-e42b4634-978d-c978-1216-a64dfb6e1224: copied to local snapshot: bytes=71 2025-07-29T00:56:16.383Z [INFO] server-e42b4634-978d-c978-1216-a64dfb6e1224: snapshot restore progress: id=2-113-1753750576382 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:16.383Z [INFO] server-e42b4634-978d-c978-1216-a64dfb6e1224: Installed remote snapshot 2025-07-29T00:56:16.383Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: pipelining replication: peer="{Voter server-e42b4634-978d-c978-1216-a64dfb6e1224 e42b4634-978d-c978-1216-a64dfb6e1224}" 2025-07-29T00:56:17.388Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:16.383320417 +0000 UTC m=+56.072701761 (2 nodes), 1.8µs from start of poll, 1.67620024s from cluster start. Timeout at 2025-07-29 00:56:17.388479257 +0000 UTC m=+57.077860601, 1.00515884s after stability 2025-07-29T00:56:17.400Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: aborting pipeline replication: peer="{Voter server-e42b4634-978d-c978-1216-a64dfb6e1224 e42b4634-978d-c978-1216-a64dfb6e1224}" 2025-07-29T00:56:17.400Z [INFO] server-345d8521-a860-4b8b-7d11-7ba45e0ab286: aborting pipeline replication: peer="{Voter server-9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab 9add55e4-82f8-eb0e-d6e7-e0ef5cf6c8ab}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2025-07-29T00:56:17.401Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc676035-3c30-f727-6a0d-8a677a29ebef Address:fc676035-3c30-f727-6a0d-8a677a29ebef} {Suffrage:Voter ID:server-257cd9e1-3e08-6d42-f8ef-ad05db81613f Address:257cd9e1-3e08-6d42-f8ef-ad05db81613f} {Suffrage:Voter ID:server-95ba672c-4fac-95b3-e769-620d85d0786d Address:95ba672c-4fac-95b3-e769-620d85d0786d}]" 2025-07-29T00:56:17.401Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: entering follower state: follower="Node at fc676035-3c30-f727-6a0d-8a677a29ebef [Follower]" leader-address= leader-id= 2025-07-29T00:56:17.401Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc676035-3c30-f727-6a0d-8a677a29ebef Address:fc676035-3c30-f727-6a0d-8a677a29ebef} {Suffrage:Voter ID:server-257cd9e1-3e08-6d42-f8ef-ad05db81613f Address:257cd9e1-3e08-6d42-f8ef-ad05db81613f} {Suffrage:Voter ID:server-95ba672c-4fac-95b3-e769-620d85d0786d Address:95ba672c-4fac-95b3-e769-620d85d0786d}]" 2025-07-29T00:56:17.402Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: entering follower state: follower="Node at 257cd9e1-3e08-6d42-f8ef-ad05db81613f [Follower]" leader-address= leader-id= 2025-07-29T00:56:17.402Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc676035-3c30-f727-6a0d-8a677a29ebef Address:fc676035-3c30-f727-6a0d-8a677a29ebef} {Suffrage:Voter ID:server-257cd9e1-3e08-6d42-f8ef-ad05db81613f Address:257cd9e1-3e08-6d42-f8ef-ad05db81613f} {Suffrage:Voter ID:server-95ba672c-4fac-95b3-e769-620d85d0786d Address:95ba672c-4fac-95b3-e769-620d85d0786d}]" 2025-07-29T00:56:17.402Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:17.402Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: entering follower state: follower="Node at 95ba672c-4fac-95b3-e769-620d85d0786d [Follower]" leader-address= leader-id= 2025-07-29T00:56:17.968Z [WARN] server-fc676035-3c30-f727-6a0d-8a677a29ebef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:17.968Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: entering candidate state: node="Node at fc676035-3c30-f727-6a0d-8a677a29ebef [Candidate]" term=2 2025-07-29T00:56:17.968Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: election won: term=2 tally=2 2025-07-29T00:56:17.968Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: entering leader state: leader="Node at fc676035-3c30-f727-6a0d-8a677a29ebef [Leader]" 2025-07-29T00:56:17.968Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: added peer, starting replication: peer=server-257cd9e1-3e08-6d42-f8ef-ad05db81613f 2025-07-29T00:56:17.968Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: added peer, starting replication: peer=server-95ba672c-4fac-95b3-e769-620d85d0786d 2025-07-29T00:56:17.968Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: pipelining replication: peer="{Voter server-95ba672c-4fac-95b3-e769-620d85d0786d 95ba672c-4fac-95b3-e769-620d85d0786d}" 2025-07-29T00:56:17.968Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: pipelining replication: peer="{Voter server-257cd9e1-3e08-6d42-f8ef-ad05db81613f 257cd9e1-3e08-6d42-f8ef-ad05db81613f}" 2025-07-29T00:56:18.974Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:17.968348257 +0000 UTC m=+57.657729641 (1 nodes), 566.21844ms from start of poll, 566.67468ms from cluster start. Timeout at 2025-07-29 00:56:18.974377177 +0000 UTC m=+58.663758521, 1.00602888s after stability 2025-07-29T00:56:18.975Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: starting snapshot up to: index=12 2025-07-29T00:56:18.975Z [INFO] creating new snapshot: path=/tmp/raft1198278681/snapshots/2-12-1753750578975.tmp 2025-07-29T00:56:18.975Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: snapshot complete up to: index=12 2025-07-29T00:56:18.976Z [INFO] creating new snapshot: path=/tmp/raft1198278681/snapshots/2-113-1753750578976.tmp 2025-07-29T00:56:18.976Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: copied to local snapshot: bytes=71 2025-07-29T00:56:18.977Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: snapshot restore progress: id=2-113-1753750578976 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:18.977Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: removing all old logs from log store 2025-07-29T00:56:18.977Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: compacting logs: from=1 to=22 2025-07-29T00:56:18.977Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: restored user snapshot: index=113 2025-07-29T00:56:18.977Z [ERROR] server-fc676035-3c30-f727-6a0d-8a677a29ebef: failed to get log: index=22 error="log not found" 2025-07-29T00:56:18.977Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: aborting pipeline replication: peer="{Voter server-95ba672c-4fac-95b3-e769-620d85d0786d 95ba672c-4fac-95b3-e769-620d85d0786d}" 2025-07-29T00:56:18.977Z [ERROR] server-fc676035-3c30-f727-6a0d-8a677a29ebef: failed to get log: index=22 error="log not found" 2025-07-29T00:56:18.977Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: aborting pipeline replication: peer="{Voter server-257cd9e1-3e08-6d42-f8ef-ad05db81613f 257cd9e1-3e08-6d42-f8ef-ad05db81613f}" 2025-07-29T00:56:18.984Z [ERROR] server-fc676035-3c30-f727-6a0d-8a677a29ebef: failed to get log: index=22 error="log not found" 2025-07-29T00:56:18.984Z [INFO] creating new snapshot: path=/tmp/raft2205969766/snapshots/2-113-1753750578984.tmp 2025-07-29T00:56:18.984Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:18.985Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: copied to local snapshot: bytes=71 2025-07-29T00:56:18.985Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: snapshot restore progress: id=2-113-1753750578984 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:18.985Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: removing all old logs from log store 2025-07-29T00:56:18.985Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: compacting logs: from=1 to=22 2025-07-29T00:56:18.985Z [INFO] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: Installed remote snapshot 2025-07-29T00:56:18.985Z [ERROR] server-fc676035-3c30-f727-6a0d-8a677a29ebef: failed to get log: index=22 error="log not found" 2025-07-29T00:56:18.985Z [WARN] server-257cd9e1-3e08-6d42-f8ef-ad05db81613f: skipping application of old log: index=22 2025-07-29T00:56:18.985Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: pipelining replication: peer="{Voter server-257cd9e1-3e08-6d42-f8ef-ad05db81613f 257cd9e1-3e08-6d42-f8ef-ad05db81613f}" 2025-07-29T00:56:18.985Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:18.986Z [INFO] creating new snapshot: path=/tmp/raft3554320821/snapshots/2-113-1753750578986.tmp 2025-07-29T00:56:18.986Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:18.986Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: copied to local snapshot: bytes=71 2025-07-29T00:56:18.987Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: snapshot restore progress: id=2-113-1753750578986 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:18.987Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: removing all old logs from log store 2025-07-29T00:56:18.987Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: compacting logs: from=1 to=22 2025-07-29T00:56:18.987Z [INFO] server-95ba672c-4fac-95b3-e769-620d85d0786d: Installed remote snapshot 2025-07-29T00:56:18.987Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: pipelining replication: peer="{Voter server-95ba672c-4fac-95b3-e769-620d85d0786d 95ba672c-4fac-95b3-e769-620d85d0786d}" 2025-07-29T00:56:19.991Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:18.985943097 +0000 UTC m=+58.675324441 (2 nodes), 1µs from start of poll, 1.58426948s from cluster start. Timeout at 2025-07-29 00:56:19.991576017 +0000 UTC m=+59.680957401, 1.00563296s after stability 2025-07-29T00:56:20.007Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: aborting pipeline replication: peer="{Voter server-95ba672c-4fac-95b3-e769-620d85d0786d 95ba672c-4fac-95b3-e769-620d85d0786d}" 2025-07-29T00:56:20.007Z [INFO] server-fc676035-3c30-f727-6a0d-8a677a29ebef: aborting pipeline replication: peer="{Voter server-257cd9e1-3e08-6d42-f8ef-ad05db81613f 257cd9e1-3e08-6d42-f8ef-ad05db81613f}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2025-07-29T00:56:20.008Z [INFO] server-abb3077e-8e10-f679-7c08-b43f82746d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abb3077e-8e10-f679-7c08-b43f82746d6c Address:abb3077e-8e10-f679-7c08-b43f82746d6c} {Suffrage:Voter ID:server-cc26b094-b225-8723-2002-14830bbe7ff0 Address:cc26b094-b225-8723-2002-14830bbe7ff0} {Suffrage:Voter ID:server-d400633d-c2af-b08c-04d0-644f25b661e6 Address:d400633d-c2af-b08c-04d0-644f25b661e6}]" 2025-07-29T00:56:20.008Z [INFO] server-abb3077e-8e10-f679-7c08-b43f82746d6c: entering follower state: follower="Node at abb3077e-8e10-f679-7c08-b43f82746d6c [Follower]" leader-address= leader-id= 2025-07-29T00:56:20.009Z [INFO] server-cc26b094-b225-8723-2002-14830bbe7ff0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abb3077e-8e10-f679-7c08-b43f82746d6c Address:abb3077e-8e10-f679-7c08-b43f82746d6c} {Suffrage:Voter ID:server-cc26b094-b225-8723-2002-14830bbe7ff0 Address:cc26b094-b225-8723-2002-14830bbe7ff0} {Suffrage:Voter ID:server-d400633d-c2af-b08c-04d0-644f25b661e6 Address:d400633d-c2af-b08c-04d0-644f25b661e6}]" 2025-07-29T00:56:20.009Z [INFO] server-cc26b094-b225-8723-2002-14830bbe7ff0: entering follower state: follower="Node at cc26b094-b225-8723-2002-14830bbe7ff0 [Follower]" leader-address= leader-id= 2025-07-29T00:56:20.009Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abb3077e-8e10-f679-7c08-b43f82746d6c Address:abb3077e-8e10-f679-7c08-b43f82746d6c} {Suffrage:Voter ID:server-cc26b094-b225-8723-2002-14830bbe7ff0 Address:cc26b094-b225-8723-2002-14830bbe7ff0} {Suffrage:Voter ID:server-d400633d-c2af-b08c-04d0-644f25b661e6 Address:d400633d-c2af-b08c-04d0-644f25b661e6}]" 2025-07-29T00:56:20.009Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:20.009Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: entering follower state: follower="Node at d400633d-c2af-b08c-04d0-644f25b661e6 [Follower]" leader-address= leader-id= 2025-07-29T00:56:20.561Z [WARN] server-d400633d-c2af-b08c-04d0-644f25b661e6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:20.561Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: entering candidate state: node="Node at d400633d-c2af-b08c-04d0-644f25b661e6 [Candidate]" term=2 2025-07-29T00:56:20.561Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: election won: term=2 tally=2 2025-07-29T00:56:20.561Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: entering leader state: leader="Node at d400633d-c2af-b08c-04d0-644f25b661e6 [Leader]" 2025-07-29T00:56:20.561Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: added peer, starting replication: peer=server-abb3077e-8e10-f679-7c08-b43f82746d6c 2025-07-29T00:56:20.561Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: added peer, starting replication: peer=server-cc26b094-b225-8723-2002-14830bbe7ff0 2025-07-29T00:56:20.561Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: pipelining replication: peer="{Voter server-abb3077e-8e10-f679-7c08-b43f82746d6c abb3077e-8e10-f679-7c08-b43f82746d6c}" 2025-07-29T00:56:20.561Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: pipelining replication: peer="{Voter server-cc26b094-b225-8723-2002-14830bbe7ff0 cc26b094-b225-8723-2002-14830bbe7ff0}" 2025-07-29T00:56:21.567Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:20.561514777 +0000 UTC m=+60.250896121 (1 nodes), 552.26396ms from start of poll, 552.7654ms from cluster start. Timeout at 2025-07-29 00:56:21.567190537 +0000 UTC m=+61.256571881, 1.00567576s after stability 2025-07-29T00:56:21.568Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: starting snapshot up to: index=12 2025-07-29T00:56:21.568Z [INFO] creating new snapshot: path=/tmp/raft2505041025/snapshots/2-12-1753750581568.tmp 2025-07-29T00:56:21.568Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: snapshot complete up to: index=12 2025-07-29T00:56:21.569Z [INFO] server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc Address:9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc} {Suffrage:Voter ID:server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2 Address:4db4f534-b6ea-49aa-339d-a5f4e639ebd2} {Suffrage:Voter ID:server-8caba45b-aaf5-6639-8c87-b2f202d4adce Address:8caba45b-aaf5-6639-8c87-b2f202d4adce}]" 2025-07-29T00:56:21.569Z [INFO] server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc Address:9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc} {Suffrage:Voter ID:server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2 Address:4db4f534-b6ea-49aa-339d-a5f4e639ebd2} {Suffrage:Voter ID:server-8caba45b-aaf5-6639-8c87-b2f202d4adce Address:8caba45b-aaf5-6639-8c87-b2f202d4adce}]" 2025-07-29T00:56:21.569Z [INFO] server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc: entering follower state: follower="Node at 9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc [Follower]" leader-address= leader-id= 2025-07-29T00:56:21.569Z [INFO] server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2: entering follower state: follower="Node at 4db4f534-b6ea-49aa-339d-a5f4e639ebd2 [Follower]" leader-address= leader-id= 2025-07-29T00:56:21.569Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc Address:9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc} {Suffrage:Voter ID:server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2 Address:4db4f534-b6ea-49aa-339d-a5f4e639ebd2} {Suffrage:Voter ID:server-8caba45b-aaf5-6639-8c87-b2f202d4adce Address:8caba45b-aaf5-6639-8c87-b2f202d4adce}]" 2025-07-29T00:56:21.569Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:21.569Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: entering follower state: follower="Node at 8caba45b-aaf5-6639-8c87-b2f202d4adce [Follower]" leader-address= leader-id= 2025-07-29T00:56:22.244Z [WARN] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:22.244Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: entering candidate state: node="Node at 8caba45b-aaf5-6639-8c87-b2f202d4adce [Candidate]" term=2 2025-07-29T00:56:22.244Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: election won: term=2 tally=2 2025-07-29T00:56:22.244Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: entering leader state: leader="Node at 8caba45b-aaf5-6639-8c87-b2f202d4adce [Leader]" 2025-07-29T00:56:22.244Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: added peer, starting replication: peer=server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc 2025-07-29T00:56:22.244Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: added peer, starting replication: peer=server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2 2025-07-29T00:56:22.244Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: pipelining replication: peer="{Voter server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2 4db4f534-b6ea-49aa-339d-a5f4e639ebd2}" 2025-07-29T00:56:22.244Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: pipelining replication: peer="{Voter server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc 9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc}" 2025-07-29T00:56:23.250Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:22.244649497 +0000 UTC m=+61.934030881 (1 nodes), 674.81036ms from start of poll, 675.33776ms from cluster start. Timeout at 2025-07-29 00:56:23.250025857 +0000 UTC m=+62.939407241, 1.00537636s after stability 2025-07-29T00:56:23.250Z [INFO] creating new snapshot: path=/tmp/raft3920438965/snapshots/2-113-1753750583250.tmp 2025-07-29T00:56:23.250Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: copied to local snapshot: bytes=71 2025-07-29T00:56:23.251Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: snapshot restore progress: id=2-113-1753750583250 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:23.251Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: restored user snapshot: index=113 2025-07-29T00:56:23.251Z [ERROR] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: failed to get log: index=3 error="log not found" 2025-07-29T00:56:23.251Z [ERROR] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: failed to get log: index=3 error="log not found" 2025-07-29T00:56:23.251Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: aborting pipeline replication: peer="{Voter server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc 9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc}" 2025-07-29T00:56:23.251Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: aborting pipeline replication: peer="{Voter server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2 4db4f534-b6ea-49aa-339d-a5f4e639ebd2}" 2025-07-29T00:56:23.259Z [ERROR] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: failed to get log: index=3 error="log not found" 2025-07-29T00:56:23.259Z [ERROR] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: failed to get log: index=3 error="log not found" 2025-07-29T00:56:23.259Z [INFO] creating new snapshot: path=/tmp/raft2815166318/snapshots/2-113-1753750583259.tmp 2025-07-29T00:56:23.259Z [INFO] creating new snapshot: path=/tmp/raft1064385428/snapshots/2-113-1753750583259.tmp 2025-07-29T00:56:23.259Z [INFO] server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:23.260Z [INFO] server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:23.260Z [INFO] server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2: copied to local snapshot: bytes=71 2025-07-29T00:56:23.260Z [INFO] server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc: copied to local snapshot: bytes=71 2025-07-29T00:56:23.260Z [INFO] server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc: snapshot restore progress: id=2-113-1753750583259 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:23.260Z [INFO] server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2: snapshot restore progress: id=2-113-1753750583259 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:23.260Z [INFO] server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc: Installed remote snapshot 2025-07-29T00:56:23.260Z [INFO] server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2: Installed remote snapshot 2025-07-29T00:56:23.260Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: pipelining replication: peer="{Voter server-9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc 9e4b7d2d-29cb-e5ba-aed8-47cbd66c1dbc}" 2025-07-29T00:56:23.260Z [WARN] server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2: skipping application of old log: index=113 2025-07-29T00:56:23.261Z [INFO] server-8caba45b-aaf5-6639-8c87-b2f202d4adce: pipelining replication: peer="{Voter server-4db4f534-b6ea-49aa-339d-a5f4e639ebd2 4db4f534-b6ea-49aa-339d-a5f4e639ebd2}" 2025-07-29T00:56:23.260Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:24.266Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:23.261071017 +0000 UTC m=+62.950452361 (2 nodes), 1.28µs from start of poll, 1.69175924s from cluster start. Timeout at 2025-07-29 00:56:24.266449177 +0000 UTC m=+63.955830561, 1.0053782s after stability 2025-07-29T00:56:24.277Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: aborting pipeline replication: peer="{Voter server-cc26b094-b225-8723-2002-14830bbe7ff0 cc26b094-b225-8723-2002-14830bbe7ff0}" 2025-07-29T00:56:24.277Z [INFO] server-d400633d-c2af-b08c-04d0-644f25b661e6: aborting pipeline replication: peer="{Voter server-abb3077e-8e10-f679-7c08-b43f82746d6c abb3077e-8e10-f679-7c08-b43f82746d6c}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2025-07-29T00:56:24.278Z [INFO] server-a3bc388f-8670-58d7-669e-41e677e1778c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3bc388f-8670-58d7-669e-41e677e1778c Address:a3bc388f-8670-58d7-669e-41e677e1778c} {Suffrage:Voter ID:server-d11f84a7-b314-32b0-8582-d49dbb23383f Address:d11f84a7-b314-32b0-8582-d49dbb23383f} {Suffrage:Voter ID:server-cbcdb452-bcb2-ac45-b510-5a7b00833408 Address:cbcdb452-bcb2-ac45-b510-5a7b00833408}]" 2025-07-29T00:56:24.278Z [INFO] server-a3bc388f-8670-58d7-669e-41e677e1778c: entering follower state: follower="Node at a3bc388f-8670-58d7-669e-41e677e1778c [Follower]" leader-address= leader-id= 2025-07-29T00:56:24.278Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3bc388f-8670-58d7-669e-41e677e1778c Address:a3bc388f-8670-58d7-669e-41e677e1778c} {Suffrage:Voter ID:server-d11f84a7-b314-32b0-8582-d49dbb23383f Address:d11f84a7-b314-32b0-8582-d49dbb23383f} {Suffrage:Voter ID:server-cbcdb452-bcb2-ac45-b510-5a7b00833408 Address:cbcdb452-bcb2-ac45-b510-5a7b00833408}]" 2025-07-29T00:56:24.278Z [INFO] server-cbcdb452-bcb2-ac45-b510-5a7b00833408: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3bc388f-8670-58d7-669e-41e677e1778c Address:a3bc388f-8670-58d7-669e-41e677e1778c} {Suffrage:Voter ID:server-d11f84a7-b314-32b0-8582-d49dbb23383f Address:d11f84a7-b314-32b0-8582-d49dbb23383f} {Suffrage:Voter ID:server-cbcdb452-bcb2-ac45-b510-5a7b00833408 Address:cbcdb452-bcb2-ac45-b510-5a7b00833408}]" 2025-07-29T00:56:24.278Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: entering follower state: follower="Node at d11f84a7-b314-32b0-8582-d49dbb23383f [Follower]" leader-address= leader-id= 2025-07-29T00:56:24.278Z [INFO] server-cbcdb452-bcb2-ac45-b510-5a7b00833408: entering follower state: follower="Node at cbcdb452-bcb2-ac45-b510-5a7b00833408 [Follower]" leader-address= leader-id= 2025-07-29T00:56:24.278Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:24.971Z [WARN] server-d11f84a7-b314-32b0-8582-d49dbb23383f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:24.971Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: entering candidate state: node="Node at d11f84a7-b314-32b0-8582-d49dbb23383f [Candidate]" term=2 2025-07-29T00:56:24.971Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: election won: term=2 tally=2 2025-07-29T00:56:24.971Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: entering leader state: leader="Node at d11f84a7-b314-32b0-8582-d49dbb23383f [Leader]" 2025-07-29T00:56:24.971Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: added peer, starting replication: peer=server-a3bc388f-8670-58d7-669e-41e677e1778c 2025-07-29T00:56:24.971Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: added peer, starting replication: peer=server-cbcdb452-bcb2-ac45-b510-5a7b00833408 2025-07-29T00:56:24.972Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: pipelining replication: peer="{Voter server-cbcdb452-bcb2-ac45-b510-5a7b00833408 cbcdb452-bcb2-ac45-b510-5a7b00833408}" 2025-07-29T00:56:24.972Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: pipelining replication: peer="{Voter server-a3bc388f-8670-58d7-669e-41e677e1778c a3bc388f-8670-58d7-669e-41e677e1778c}" 2025-07-29T00:56:25.977Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:24.971908137 +0000 UTC m=+64.661289481 (1 nodes), 692.92972ms from start of poll, 693.43016ms from cluster start. Timeout at 2025-07-29 00:56:25.977375377 +0000 UTC m=+65.666756721, 1.00546724s after stability 2025-07-29T00:56:25.978Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: starting snapshot up to: index=12 2025-07-29T00:56:25.978Z [INFO] creating new snapshot: path=/tmp/raft508850889/snapshots/2-12-1753750585978.tmp 2025-07-29T00:56:25.978Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: snapshot complete up to: index=12 2025-07-29T00:56:25.979Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fc301b7-2772-dfd1-7a3a-1f463835f13e Address:8fc301b7-2772-dfd1-7a3a-1f463835f13e} {Suffrage:Voter ID:server-3c106d9a-436b-9a66-cdbf-06a53095387b Address:3c106d9a-436b-9a66-cdbf-06a53095387b} {Suffrage:Voter ID:server-8b55393c-6d33-120f-d679-50014ddc8620 Address:8b55393c-6d33-120f-d679-50014ddc8620}]" 2025-07-29T00:56:25.979Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: entering follower state: follower="Node at 8fc301b7-2772-dfd1-7a3a-1f463835f13e [Follower]" leader-address= leader-id= 2025-07-29T00:56:25.979Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fc301b7-2772-dfd1-7a3a-1f463835f13e Address:8fc301b7-2772-dfd1-7a3a-1f463835f13e} {Suffrage:Voter ID:server-3c106d9a-436b-9a66-cdbf-06a53095387b Address:3c106d9a-436b-9a66-cdbf-06a53095387b} {Suffrage:Voter ID:server-8b55393c-6d33-120f-d679-50014ddc8620 Address:8b55393c-6d33-120f-d679-50014ddc8620}]" 2025-07-29T00:56:25.979Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: entering follower state: follower="Node at 3c106d9a-436b-9a66-cdbf-06a53095387b [Follower]" leader-address= leader-id= 2025-07-29T00:56:25.979Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8fc301b7-2772-dfd1-7a3a-1f463835f13e Address:8fc301b7-2772-dfd1-7a3a-1f463835f13e} {Suffrage:Voter ID:server-3c106d9a-436b-9a66-cdbf-06a53095387b Address:3c106d9a-436b-9a66-cdbf-06a53095387b} {Suffrage:Voter ID:server-8b55393c-6d33-120f-d679-50014ddc8620 Address:8b55393c-6d33-120f-d679-50014ddc8620}]" 2025-07-29T00:56:25.979Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:25.979Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: entering follower state: follower="Node at 8b55393c-6d33-120f-d679-50014ddc8620 [Follower]" leader-address= leader-id= 2025-07-29T00:56:26.617Z [WARN] server-3c106d9a-436b-9a66-cdbf-06a53095387b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:26.617Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: entering candidate state: node="Node at 3c106d9a-436b-9a66-cdbf-06a53095387b [Candidate]" term=2 2025-07-29T00:56:26.617Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: election won: term=2 tally=2 2025-07-29T00:56:26.617Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: entering leader state: leader="Node at 3c106d9a-436b-9a66-cdbf-06a53095387b [Leader]" 2025-07-29T00:56:26.617Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: added peer, starting replication: peer=server-8fc301b7-2772-dfd1-7a3a-1f463835f13e 2025-07-29T00:56:26.617Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: added peer, starting replication: peer=server-8b55393c-6d33-120f-d679-50014ddc8620 2025-07-29T00:56:26.617Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: pipelining replication: peer="{Voter server-8b55393c-6d33-120f-d679-50014ddc8620 8b55393c-6d33-120f-d679-50014ddc8620}" 2025-07-29T00:56:26.617Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: pipelining replication: peer="{Voter server-8fc301b7-2772-dfd1-7a3a-1f463835f13e 8fc301b7-2772-dfd1-7a3a-1f463835f13e}" 2025-07-29T00:56:27.623Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:26.617519017 +0000 UTC m=+66.306900401 (1 nodes), 637.51148ms from start of poll, 637.97712ms from cluster start. Timeout at 2025-07-29 00:56:27.623280057 +0000 UTC m=+67.312661361, 1.00576096s after stability 2025-07-29T00:56:27.623Z [INFO] creating new snapshot: path=/tmp/raft1784838365/snapshots/2-113-1753750587623.tmp 2025-07-29T00:56:27.623Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: copied to local snapshot: bytes=71 2025-07-29T00:56:27.624Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: snapshot restore progress: id=2-113-1753750587623 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:27.624Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: removing all old logs from log store 2025-07-29T00:56:27.624Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: compacting logs: from=1 to=2 2025-07-29T00:56:27.624Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: restored user snapshot: index=113 2025-07-29T00:56:27.624Z [ERROR] server-3c106d9a-436b-9a66-cdbf-06a53095387b: failed to get log: index=2 error="log not found" 2025-07-29T00:56:27.624Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: aborting pipeline replication: peer="{Voter server-8b55393c-6d33-120f-d679-50014ddc8620 8b55393c-6d33-120f-d679-50014ddc8620}" 2025-07-29T00:56:27.624Z [ERROR] server-3c106d9a-436b-9a66-cdbf-06a53095387b: failed to get log: index=2 error="log not found" 2025-07-29T00:56:27.624Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: aborting pipeline replication: peer="{Voter server-8fc301b7-2772-dfd1-7a3a-1f463835f13e 8fc301b7-2772-dfd1-7a3a-1f463835f13e}" 2025-07-29T00:56:27.631Z [ERROR] server-3c106d9a-436b-9a66-cdbf-06a53095387b: failed to get log: index=2 error="log not found" 2025-07-29T00:56:27.631Z [INFO] creating new snapshot: path=/tmp/raft238901939/snapshots/2-113-1753750587631.tmp 2025-07-29T00:56:27.631Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:27.632Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: copied to local snapshot: bytes=71 2025-07-29T00:56:27.632Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: snapshot restore progress: id=2-113-1753750587631 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:27.632Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: removing all old logs from log store 2025-07-29T00:56:27.632Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: compacting logs: from=1 to=2 2025-07-29T00:56:27.632Z [INFO] server-8fc301b7-2772-dfd1-7a3a-1f463835f13e: Installed remote snapshot 2025-07-29T00:56:27.632Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: pipelining replication: peer="{Voter server-8fc301b7-2772-dfd1-7a3a-1f463835f13e 8fc301b7-2772-dfd1-7a3a-1f463835f13e}" 2025-07-29T00:56:27.632Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:27.633Z [ERROR] server-3c106d9a-436b-9a66-cdbf-06a53095387b: failed to get log: index=2 error="log not found" 2025-07-29T00:56:27.633Z [INFO] creating new snapshot: path=/tmp/raft637207748/snapshots/2-113-1753750587633.tmp 2025-07-29T00:56:27.633Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:27.634Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: copied to local snapshot: bytes=71 2025-07-29T00:56:27.634Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: snapshot restore progress: id=2-113-1753750587633 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:27.634Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: removing all old logs from log store 2025-07-29T00:56:27.634Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: compacting logs: from=1 to=2 2025-07-29T00:56:27.634Z [INFO] server-8b55393c-6d33-120f-d679-50014ddc8620: Installed remote snapshot 2025-07-29T00:56:27.634Z [INFO] server-3c106d9a-436b-9a66-cdbf-06a53095387b: pipelining replication: peer="{Voter server-8b55393c-6d33-120f-d679-50014ddc8620 8b55393c-6d33-120f-d679-50014ddc8620}" 2025-07-29T00:56:28.637Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:27.632422017 +0000 UTC m=+67.321803361 (2 nodes), 1µs from start of poll, 1.65288008s from cluster start. Timeout at 2025-07-29 00:56:28.637836937 +0000 UTC m=+68.327218281, 1.00541492s after stability 2025-07-29T00:56:28.650Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: aborting pipeline replication: peer="{Voter server-cbcdb452-bcb2-ac45-b510-5a7b00833408 cbcdb452-bcb2-ac45-b510-5a7b00833408}" 2025-07-29T00:56:28.650Z [INFO] server-d11f84a7-b314-32b0-8582-d49dbb23383f: aborting pipeline replication: peer="{Voter server-a3bc388f-8670-58d7-669e-41e677e1778c a3bc388f-8670-58d7-669e-41e677e1778c}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2025-07-29T00:56:28.651Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-146ed373-d279-46e3-9448-2c843a804699 Address:146ed373-d279-46e3-9448-2c843a804699} {Suffrage:Voter ID:server-f4e66da7-1469-a5fd-afae-856dfae0aecb Address:f4e66da7-1469-a5fd-afae-856dfae0aecb} {Suffrage:Voter ID:server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 Address:c1dac7f4-006b-ddc6-29a7-297ce386bdc9}]" 2025-07-29T00:56:28.651Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: entering follower state: follower="Node at 146ed373-d279-46e3-9448-2c843a804699 [Follower]" leader-address= leader-id= 2025-07-29T00:56:28.651Z [INFO] server-f4e66da7-1469-a5fd-afae-856dfae0aecb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-146ed373-d279-46e3-9448-2c843a804699 Address:146ed373-d279-46e3-9448-2c843a804699} {Suffrage:Voter ID:server-f4e66da7-1469-a5fd-afae-856dfae0aecb Address:f4e66da7-1469-a5fd-afae-856dfae0aecb} {Suffrage:Voter ID:server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 Address:c1dac7f4-006b-ddc6-29a7-297ce386bdc9}]" 2025-07-29T00:56:28.651Z [INFO] server-f4e66da7-1469-a5fd-afae-856dfae0aecb: entering follower state: follower="Node at f4e66da7-1469-a5fd-afae-856dfae0aecb [Follower]" leader-address= leader-id= 2025-07-29T00:56:28.651Z [INFO] server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-146ed373-d279-46e3-9448-2c843a804699 Address:146ed373-d279-46e3-9448-2c843a804699} {Suffrage:Voter ID:server-f4e66da7-1469-a5fd-afae-856dfae0aecb Address:f4e66da7-1469-a5fd-afae-856dfae0aecb} {Suffrage:Voter ID:server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 Address:c1dac7f4-006b-ddc6-29a7-297ce386bdc9}]" 2025-07-29T00:56:28.651Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:28.651Z [INFO] server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9: entering follower state: follower="Node at c1dac7f4-006b-ddc6-29a7-297ce386bdc9 [Follower]" leader-address= leader-id= 2025-07-29T00:56:29.436Z [WARN] server-146ed373-d279-46e3-9448-2c843a804699: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:29.436Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: entering candidate state: node="Node at 146ed373-d279-46e3-9448-2c843a804699 [Candidate]" term=2 2025-07-29T00:56:29.436Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: election won: term=2 tally=2 2025-07-29T00:56:29.436Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: entering leader state: leader="Node at 146ed373-d279-46e3-9448-2c843a804699 [Leader]" 2025-07-29T00:56:29.436Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: added peer, starting replication: peer=server-f4e66da7-1469-a5fd-afae-856dfae0aecb 2025-07-29T00:56:29.436Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: added peer, starting replication: peer=server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 2025-07-29T00:56:29.436Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: pipelining replication: peer="{Voter server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 c1dac7f4-006b-ddc6-29a7-297ce386bdc9}" 2025-07-29T00:56:29.436Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: pipelining replication: peer="{Voter server-f4e66da7-1469-a5fd-afae-856dfae0aecb f4e66da7-1469-a5fd-afae-856dfae0aecb}" 2025-07-29T00:56:30.442Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:29.436846337 +0000 UTC m=+69.126227681 (1 nodes), 785.52548ms from start of poll, 785.98436ms from cluster start. Timeout at 2025-07-29 00:56:30.442619577 +0000 UTC m=+70.132000921, 1.00577324s after stability 2025-07-29T00:56:30.443Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: starting snapshot up to: index=12 2025-07-29T00:56:30.443Z [INFO] creating new snapshot: path=/tmp/raft1456786153/snapshots/2-12-1753750590443.tmp 2025-07-29T00:56:30.443Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: snapshot complete up to: index=12 2025-07-29T00:56:30.444Z [INFO] creating new snapshot: path=/tmp/raft1456786153/snapshots/2-1013-1753750590444.tmp 2025-07-29T00:56:30.445Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: copied to local snapshot: bytes=71 2025-07-29T00:56:30.445Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: snapshot restore progress: id=2-1013-1753750590444 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:30.445Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: restored user snapshot: index=1013 2025-07-29T00:56:30.445Z [ERROR] server-146ed373-d279-46e3-9448-2c843a804699: failed to get log: index=23 error="log not found" 2025-07-29T00:56:30.445Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: aborting pipeline replication: peer="{Voter server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 c1dac7f4-006b-ddc6-29a7-297ce386bdc9}" 2025-07-29T00:56:30.445Z [ERROR] server-146ed373-d279-46e3-9448-2c843a804699: failed to get log: index=23 error="log not found" 2025-07-29T00:56:30.445Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: aborting pipeline replication: peer="{Voter server-f4e66da7-1469-a5fd-afae-856dfae0aecb f4e66da7-1469-a5fd-afae-856dfae0aecb}" 2025-07-29T00:56:30.452Z [ERROR] server-146ed373-d279-46e3-9448-2c843a804699: failed to get log: index=23 error="log not found" 2025-07-29T00:56:30.453Z [INFO] creating new snapshot: path=/tmp/raft933387236/snapshots/2-1013-1753750590453.tmp 2025-07-29T00:56:30.453Z [INFO] server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:30.453Z [INFO] server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9: copied to local snapshot: bytes=71 2025-07-29T00:56:30.454Z [ERROR] server-146ed373-d279-46e3-9448-2c843a804699: failed to get log: index=23 error="log not found" 2025-07-29T00:56:30.454Z [INFO] server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9: snapshot restore progress: id=2-1013-1753750590453 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:30.454Z [INFO] server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9: Installed remote snapshot 2025-07-29T00:56:30.454Z [WARN] server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9: skipping application of old log: index=22 2025-07-29T00:56:30.454Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: pipelining replication: peer="{Voter server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 c1dac7f4-006b-ddc6-29a7-297ce386bdc9}" 2025-07-29T00:56:30.454Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:30.454Z [INFO] creating new snapshot: path=/tmp/raft3072137057/snapshots/2-1013-1753750590454.tmp 2025-07-29T00:56:30.454Z [INFO] server-f4e66da7-1469-a5fd-afae-856dfae0aecb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:30.455Z [INFO] server-f4e66da7-1469-a5fd-afae-856dfae0aecb: copied to local snapshot: bytes=71 2025-07-29T00:56:30.455Z [INFO] server-f4e66da7-1469-a5fd-afae-856dfae0aecb: snapshot restore progress: id=2-1013-1753750590454 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:30.455Z [INFO] server-f4e66da7-1469-a5fd-afae-856dfae0aecb: Installed remote snapshot 2025-07-29T00:56:30.455Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: pipelining replication: peer="{Voter server-f4e66da7-1469-a5fd-afae-856dfae0aecb f4e66da7-1469-a5fd-afae-856dfae0aecb}" 2025-07-29T00:56:31.459Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:30.454543697 +0000 UTC m=+70.143925041 (2 nodes), 1.2µs from start of poll, 1.80368172s from cluster start. Timeout at 2025-07-29 00:56:31.459903657 +0000 UTC m=+71.149285041, 1.00536s after stability 2025-07-29T00:56:31.471Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: aborting pipeline replication: peer="{Voter server-c1dac7f4-006b-ddc6-29a7-297ce386bdc9 c1dac7f4-006b-ddc6-29a7-297ce386bdc9}" 2025-07-29T00:56:31.471Z [INFO] server-146ed373-d279-46e3-9448-2c843a804699: aborting pipeline replication: peer="{Voter server-f4e66da7-1469-a5fd-afae-856dfae0aecb f4e66da7-1469-a5fd-afae-856dfae0aecb}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2025-07-29T00:56:31.472Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8abc0e6-2afe-888e-cd55-cf3cbb769269 Address:a8abc0e6-2afe-888e-cd55-cf3cbb769269} {Suffrage:Voter ID:server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b Address:180b5d92-3bbd-e03a-9d7b-307b9a97a23b} {Suffrage:Voter ID:server-4366f57d-b06a-dbcc-69e9-7116e88bc091 Address:4366f57d-b06a-dbcc-69e9-7116e88bc091}]" 2025-07-29T00:56:31.472Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: entering follower state: follower="Node at a8abc0e6-2afe-888e-cd55-cf3cbb769269 [Follower]" leader-address= leader-id= 2025-07-29T00:56:31.472Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8abc0e6-2afe-888e-cd55-cf3cbb769269 Address:a8abc0e6-2afe-888e-cd55-cf3cbb769269} {Suffrage:Voter ID:server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b Address:180b5d92-3bbd-e03a-9d7b-307b9a97a23b} {Suffrage:Voter ID:server-4366f57d-b06a-dbcc-69e9-7116e88bc091 Address:4366f57d-b06a-dbcc-69e9-7116e88bc091}]" 2025-07-29T00:56:31.472Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: entering follower state: follower="Node at 180b5d92-3bbd-e03a-9d7b-307b9a97a23b [Follower]" leader-address= leader-id= 2025-07-29T00:56:31.472Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8abc0e6-2afe-888e-cd55-cf3cbb769269 Address:a8abc0e6-2afe-888e-cd55-cf3cbb769269} {Suffrage:Voter ID:server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b Address:180b5d92-3bbd-e03a-9d7b-307b9a97a23b} {Suffrage:Voter ID:server-4366f57d-b06a-dbcc-69e9-7116e88bc091 Address:4366f57d-b06a-dbcc-69e9-7116e88bc091}]" 2025-07-29T00:56:31.473Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:31.473Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: entering follower state: follower="Node at 4366f57d-b06a-dbcc-69e9-7116e88bc091 [Follower]" leader-address= leader-id= 2025-07-29T00:56:32.270Z [WARN] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:32.270Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: entering candidate state: node="Node at a8abc0e6-2afe-888e-cd55-cf3cbb769269 [Candidate]" term=2 2025-07-29T00:56:32.270Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: election won: term=2 tally=2 2025-07-29T00:56:32.270Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: entering leader state: leader="Node at a8abc0e6-2afe-888e-cd55-cf3cbb769269 [Leader]" 2025-07-29T00:56:32.270Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: added peer, starting replication: peer=server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b 2025-07-29T00:56:32.270Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: added peer, starting replication: peer=server-4366f57d-b06a-dbcc-69e9-7116e88bc091 2025-07-29T00:56:32.270Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: pipelining replication: peer="{Voter server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b 180b5d92-3bbd-e03a-9d7b-307b9a97a23b}" 2025-07-29T00:56:32.270Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: pipelining replication: peer="{Voter server-4366f57d-b06a-dbcc-69e9-7116e88bc091 4366f57d-b06a-dbcc-69e9-7116e88bc091}" 2025-07-29T00:56:33.275Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:32.270747417 +0000 UTC m=+71.960128761 (1 nodes), 797.72176ms from start of poll, 798.39696ms from cluster start. Timeout at 2025-07-29 00:56:33.275907977 +0000 UTC m=+72.965289321, 1.00516056s after stability 2025-07-29T00:56:33.276Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: starting snapshot up to: index=12 2025-07-29T00:56:33.276Z [INFO] creating new snapshot: path=/tmp/raft264968425/snapshots/2-12-1753750593276.tmp 2025-07-29T00:56:33.277Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: snapshot complete up to: index=12 2025-07-29T00:56:33.278Z [INFO] creating new snapshot: path=/tmp/raft264968425/snapshots/2-1013-1753750593278.tmp 2025-07-29T00:56:33.278Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: copied to local snapshot: bytes=71 2025-07-29T00:56:33.278Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: snapshot restore progress: id=2-1013-1753750593278 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:33.279Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: removing all old logs from log store 2025-07-29T00:56:33.279Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: compacting logs: from=1 to=22 2025-07-29T00:56:33.279Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: restored user snapshot: index=1013 2025-07-29T00:56:33.279Z [ERROR] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: failed to get log: index=22 error="log not found" 2025-07-29T00:56:33.279Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: aborting pipeline replication: peer="{Voter server-4366f57d-b06a-dbcc-69e9-7116e88bc091 4366f57d-b06a-dbcc-69e9-7116e88bc091}" 2025-07-29T00:56:33.279Z [ERROR] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: failed to get log: index=22 error="log not found" 2025-07-29T00:56:33.279Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: aborting pipeline replication: peer="{Voter server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b 180b5d92-3bbd-e03a-9d7b-307b9a97a23b}" 2025-07-29T00:56:33.285Z [ERROR] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: failed to get log: index=22 error="log not found" 2025-07-29T00:56:33.285Z [INFO] creating new snapshot: path=/tmp/raft3992713923/snapshots/2-1013-1753750593285.tmp 2025-07-29T00:56:33.286Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:33.286Z [ERROR] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: failed to get log: index=22 error="log not found" 2025-07-29T00:56:33.286Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: copied to local snapshot: bytes=71 2025-07-29T00:56:33.286Z [INFO] creating new snapshot: path=/tmp/raft1952085273/snapshots/2-1013-1753750593286.tmp 2025-07-29T00:56:33.286Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: snapshot restore progress: id=2-1013-1753750593285 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:33.286Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: removing all old logs from log store 2025-07-29T00:56:33.286Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: compacting logs: from=1 to=22 2025-07-29T00:56:33.286Z [INFO] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: Installed remote snapshot 2025-07-29T00:56:33.286Z [WARN] server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b: skipping application of old log: index=22 2025-07-29T00:56:33.286Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:33.286Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: pipelining replication: peer="{Voter server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b 180b5d92-3bbd-e03a-9d7b-307b9a97a23b}" 2025-07-29T00:56:33.286Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:33.287Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: copied to local snapshot: bytes=71 2025-07-29T00:56:33.287Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: snapshot restore progress: id=2-1013-1753750593286 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:33.287Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: removing all old logs from log store 2025-07-29T00:56:33.287Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: compacting logs: from=1 to=22 2025-07-29T00:56:33.287Z [INFO] server-4366f57d-b06a-dbcc-69e9-7116e88bc091: Installed remote snapshot 2025-07-29T00:56:33.288Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: pipelining replication: peer="{Voter server-4366f57d-b06a-dbcc-69e9-7116e88bc091 4366f57d-b06a-dbcc-69e9-7116e88bc091}" 2025-07-29T00:56:34.292Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:33.286932977 +0000 UTC m=+72.976314321 (2 nodes), 1.28µs from start of poll, 1.81458252s from cluster start. Timeout at 2025-07-29 00:56:34.292863417 +0000 UTC m=+73.982244761, 1.00593044s after stability 2025-07-29T00:56:34.304Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: aborting pipeline replication: peer="{Voter server-4366f57d-b06a-dbcc-69e9-7116e88bc091 4366f57d-b06a-dbcc-69e9-7116e88bc091}" 2025-07-29T00:56:34.304Z [INFO] server-a8abc0e6-2afe-888e-cd55-cf3cbb769269: aborting pipeline replication: peer="{Voter server-180b5d92-3bbd-e03a-9d7b-307b9a97a23b 180b5d92-3bbd-e03a-9d7b-307b9a97a23b}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2025-07-29T00:56:34.305Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c85dc61-bbc7-e434-43b4-08de379d82ca Address:4c85dc61-bbc7-e434-43b4-08de379d82ca} {Suffrage:Voter ID:server-4dff4cd9-eb99-49bf-593f-6ba270bac680 Address:4dff4cd9-eb99-49bf-593f-6ba270bac680} {Suffrage:Voter ID:server-250dfbcd-0afa-8e76-790b-61c54266c3e9 Address:250dfbcd-0afa-8e76-790b-61c54266c3e9}]" 2025-07-29T00:56:34.305Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: entering follower state: follower="Node at 4c85dc61-bbc7-e434-43b4-08de379d82ca [Follower]" leader-address= leader-id= 2025-07-29T00:56:34.305Z [INFO] server-4dff4cd9-eb99-49bf-593f-6ba270bac680: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c85dc61-bbc7-e434-43b4-08de379d82ca Address:4c85dc61-bbc7-e434-43b4-08de379d82ca} {Suffrage:Voter ID:server-4dff4cd9-eb99-49bf-593f-6ba270bac680 Address:4dff4cd9-eb99-49bf-593f-6ba270bac680} {Suffrage:Voter ID:server-250dfbcd-0afa-8e76-790b-61c54266c3e9 Address:250dfbcd-0afa-8e76-790b-61c54266c3e9}]" 2025-07-29T00:56:34.305Z [INFO] server-4dff4cd9-eb99-49bf-593f-6ba270bac680: entering follower state: follower="Node at 4dff4cd9-eb99-49bf-593f-6ba270bac680 [Follower]" leader-address= leader-id= 2025-07-29T00:56:34.305Z [INFO] server-250dfbcd-0afa-8e76-790b-61c54266c3e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c85dc61-bbc7-e434-43b4-08de379d82ca Address:4c85dc61-bbc7-e434-43b4-08de379d82ca} {Suffrage:Voter ID:server-4dff4cd9-eb99-49bf-593f-6ba270bac680 Address:4dff4cd9-eb99-49bf-593f-6ba270bac680} {Suffrage:Voter ID:server-250dfbcd-0afa-8e76-790b-61c54266c3e9 Address:250dfbcd-0afa-8e76-790b-61c54266c3e9}]" 2025-07-29T00:56:34.305Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:34.305Z [INFO] server-250dfbcd-0afa-8e76-790b-61c54266c3e9: entering follower state: follower="Node at 250dfbcd-0afa-8e76-790b-61c54266c3e9 [Follower]" leader-address= leader-id= 2025-07-29T00:56:34.898Z [WARN] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:34.898Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: entering candidate state: node="Node at 4c85dc61-bbc7-e434-43b4-08de379d82ca [Candidate]" term=2 2025-07-29T00:56:34.898Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: election won: term=2 tally=2 2025-07-29T00:56:34.898Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: entering leader state: leader="Node at 4c85dc61-bbc7-e434-43b4-08de379d82ca [Leader]" 2025-07-29T00:56:34.898Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: added peer, starting replication: peer=server-4dff4cd9-eb99-49bf-593f-6ba270bac680 2025-07-29T00:56:34.898Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: added peer, starting replication: peer=server-250dfbcd-0afa-8e76-790b-61c54266c3e9 2025-07-29T00:56:34.899Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: pipelining replication: peer="{Voter server-4dff4cd9-eb99-49bf-593f-6ba270bac680 4dff4cd9-eb99-49bf-593f-6ba270bac680}" 2025-07-29T00:56:34.899Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: pipelining replication: peer="{Voter server-250dfbcd-0afa-8e76-790b-61c54266c3e9 250dfbcd-0afa-8e76-790b-61c54266c3e9}" 2025-07-29T00:56:35.905Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:34.898946697 +0000 UTC m=+74.588328041 (1 nodes), 593.32712ms from start of poll, 593.81976ms from cluster start. Timeout at 2025-07-29 00:56:35.905185457 +0000 UTC m=+75.594566801, 1.00623876s after stability 2025-07-29T00:56:35.905Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: starting snapshot up to: index=12 2025-07-29T00:56:35.905Z [INFO] creating new snapshot: path=/tmp/raft490214627/snapshots/2-12-1753750595905.tmp 2025-07-29T00:56:35.906Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: snapshot complete up to: index=12 2025-07-29T00:56:35.907Z [INFO] server-7fdc5934-2c84-7c44-8442-b19760381ecb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fdc5934-2c84-7c44-8442-b19760381ecb Address:7fdc5934-2c84-7c44-8442-b19760381ecb} {Suffrage:Voter ID:server-1968d802-8362-a95b-519b-dbf2843fde98 Address:1968d802-8362-a95b-519b-dbf2843fde98} {Suffrage:Voter ID:server-0f511721-08e1-49ec-e275-b2d788e48d73 Address:0f511721-08e1-49ec-e275-b2d788e48d73}]" 2025-07-29T00:56:35.907Z [INFO] server-7fdc5934-2c84-7c44-8442-b19760381ecb: entering follower state: follower="Node at 7fdc5934-2c84-7c44-8442-b19760381ecb [Follower]" leader-address= leader-id= 2025-07-29T00:56:35.907Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fdc5934-2c84-7c44-8442-b19760381ecb Address:7fdc5934-2c84-7c44-8442-b19760381ecb} {Suffrage:Voter ID:server-1968d802-8362-a95b-519b-dbf2843fde98 Address:1968d802-8362-a95b-519b-dbf2843fde98} {Suffrage:Voter ID:server-0f511721-08e1-49ec-e275-b2d788e48d73 Address:0f511721-08e1-49ec-e275-b2d788e48d73}]" 2025-07-29T00:56:35.907Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: entering follower state: follower="Node at 1968d802-8362-a95b-519b-dbf2843fde98 [Follower]" leader-address= leader-id= 2025-07-29T00:56:35.907Z [INFO] server-0f511721-08e1-49ec-e275-b2d788e48d73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fdc5934-2c84-7c44-8442-b19760381ecb Address:7fdc5934-2c84-7c44-8442-b19760381ecb} {Suffrage:Voter ID:server-1968d802-8362-a95b-519b-dbf2843fde98 Address:1968d802-8362-a95b-519b-dbf2843fde98} {Suffrage:Voter ID:server-0f511721-08e1-49ec-e275-b2d788e48d73 Address:0f511721-08e1-49ec-e275-b2d788e48d73}]" 2025-07-29T00:56:35.907Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:35.907Z [INFO] server-0f511721-08e1-49ec-e275-b2d788e48d73: entering follower state: follower="Node at 0f511721-08e1-49ec-e275-b2d788e48d73 [Follower]" leader-address= leader-id= 2025-07-29T00:56:36.741Z [WARN] server-1968d802-8362-a95b-519b-dbf2843fde98: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:36.741Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: entering candidate state: node="Node at 1968d802-8362-a95b-519b-dbf2843fde98 [Candidate]" term=2 2025-07-29T00:56:36.741Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: election won: term=2 tally=2 2025-07-29T00:56:36.741Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: entering leader state: leader="Node at 1968d802-8362-a95b-519b-dbf2843fde98 [Leader]" 2025-07-29T00:56:36.741Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: added peer, starting replication: peer=server-7fdc5934-2c84-7c44-8442-b19760381ecb 2025-07-29T00:56:36.741Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: added peer, starting replication: peer=server-0f511721-08e1-49ec-e275-b2d788e48d73 2025-07-29T00:56:36.741Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: pipelining replication: peer="{Voter server-0f511721-08e1-49ec-e275-b2d788e48d73 0f511721-08e1-49ec-e275-b2d788e48d73}" 2025-07-29T00:56:36.741Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: pipelining replication: peer="{Voter server-7fdc5934-2c84-7c44-8442-b19760381ecb 7fdc5934-2c84-7c44-8442-b19760381ecb}" 2025-07-29T00:56:37.746Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:36.741413497 +0000 UTC m=+76.430794841 (1 nodes), 833.476ms from start of poll, 834.10892ms from cluster start. Timeout at 2025-07-29 00:56:37.746429617 +0000 UTC m=+77.435810961, 1.00501612s after stability 2025-07-29T00:56:37.746Z [INFO] creating new snapshot: path=/tmp/raft3851732482/snapshots/2-1013-1753750597746.tmp 2025-07-29T00:56:37.747Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: copied to local snapshot: bytes=71 2025-07-29T00:56:37.747Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: snapshot restore progress: id=2-1013-1753750597746 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:37.747Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: restored user snapshot: index=1013 2025-07-29T00:56:37.747Z [ERROR] server-1968d802-8362-a95b-519b-dbf2843fde98: failed to get log: index=3 error="log not found" 2025-07-29T00:56:37.747Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: aborting pipeline replication: peer="{Voter server-0f511721-08e1-49ec-e275-b2d788e48d73 0f511721-08e1-49ec-e275-b2d788e48d73}" 2025-07-29T00:56:37.747Z [ERROR] server-1968d802-8362-a95b-519b-dbf2843fde98: failed to get log: index=3 error="log not found" 2025-07-29T00:56:37.747Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: aborting pipeline replication: peer="{Voter server-7fdc5934-2c84-7c44-8442-b19760381ecb 7fdc5934-2c84-7c44-8442-b19760381ecb}" 2025-07-29T00:56:37.753Z [ERROR] server-1968d802-8362-a95b-519b-dbf2843fde98: failed to get log: index=3 error="log not found" 2025-07-29T00:56:37.754Z [INFO] creating new snapshot: path=/tmp/raft3082020791/snapshots/2-1013-1753750597754.tmp 2025-07-29T00:56:37.755Z [INFO] server-0f511721-08e1-49ec-e275-b2d788e48d73: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:37.755Z [INFO] server-0f511721-08e1-49ec-e275-b2d788e48d73: copied to local snapshot: bytes=71 2025-07-29T00:56:37.755Z [ERROR] server-1968d802-8362-a95b-519b-dbf2843fde98: failed to get log: index=3 error="log not found" 2025-07-29T00:56:37.756Z [INFO] server-0f511721-08e1-49ec-e275-b2d788e48d73: snapshot restore progress: id=2-1013-1753750597754 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:37.756Z [INFO] server-0f511721-08e1-49ec-e275-b2d788e48d73: Installed remote snapshot 2025-07-29T00:56:37.756Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: pipelining replication: peer="{Voter server-0f511721-08e1-49ec-e275-b2d788e48d73 0f511721-08e1-49ec-e275-b2d788e48d73}" 2025-07-29T00:56:37.756Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:37.757Z [INFO] creating new snapshot: path=/tmp/raft4002057630/snapshots/2-1013-1753750597757.tmp 2025-07-29T00:56:37.757Z [INFO] server-7fdc5934-2c84-7c44-8442-b19760381ecb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:37.757Z [INFO] server-7fdc5934-2c84-7c44-8442-b19760381ecb: copied to local snapshot: bytes=71 2025-07-29T00:56:37.757Z [INFO] server-7fdc5934-2c84-7c44-8442-b19760381ecb: snapshot restore progress: id=2-1013-1753750597757 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:37.757Z [INFO] server-7fdc5934-2c84-7c44-8442-b19760381ecb: Installed remote snapshot 2025-07-29T00:56:37.757Z [INFO] server-1968d802-8362-a95b-519b-dbf2843fde98: pipelining replication: peer="{Voter server-7fdc5934-2c84-7c44-8442-b19760381ecb 7fdc5934-2c84-7c44-8442-b19760381ecb}" 2025-07-29T00:56:38.762Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:37.756898777 +0000 UTC m=+77.446280121 (2 nodes), 2.28µs from start of poll, 1.8495942s from cluster start. Timeout at 2025-07-29 00:56:38.762545617 +0000 UTC m=+78.451926961, 1.00564684s after stability 2025-07-29T00:56:38.773Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: aborting pipeline replication: peer="{Voter server-250dfbcd-0afa-8e76-790b-61c54266c3e9 250dfbcd-0afa-8e76-790b-61c54266c3e9}" 2025-07-29T00:56:38.773Z [INFO] server-4c85dc61-bbc7-e434-43b4-08de379d82ca: aborting pipeline replication: peer="{Voter server-4dff4cd9-eb99-49bf-593f-6ba270bac680 4dff4cd9-eb99-49bf-593f-6ba270bac680}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2025-07-29T00:56:38.774Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6 Address:a01e87d9-f6ed-1e46-7f7e-9183ed206ca6} {Suffrage:Voter ID:server-9d6dec2a-19b0-5bca-124f-82bd755fff4f Address:9d6dec2a-19b0-5bca-124f-82bd755fff4f} {Suffrage:Voter ID:server-1325ed73-6246-3a9e-a8fb-9cf042014523 Address:1325ed73-6246-3a9e-a8fb-9cf042014523}]" 2025-07-29T00:56:38.774Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: entering follower state: follower="Node at a01e87d9-f6ed-1e46-7f7e-9183ed206ca6 [Follower]" leader-address= leader-id= 2025-07-29T00:56:38.774Z [INFO] server-9d6dec2a-19b0-5bca-124f-82bd755fff4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6 Address:a01e87d9-f6ed-1e46-7f7e-9183ed206ca6} {Suffrage:Voter ID:server-9d6dec2a-19b0-5bca-124f-82bd755fff4f Address:9d6dec2a-19b0-5bca-124f-82bd755fff4f} {Suffrage:Voter ID:server-1325ed73-6246-3a9e-a8fb-9cf042014523 Address:1325ed73-6246-3a9e-a8fb-9cf042014523}]" 2025-07-29T00:56:38.774Z [INFO] server-9d6dec2a-19b0-5bca-124f-82bd755fff4f: entering follower state: follower="Node at 9d6dec2a-19b0-5bca-124f-82bd755fff4f [Follower]" leader-address= leader-id= 2025-07-29T00:56:38.774Z [INFO] server-1325ed73-6246-3a9e-a8fb-9cf042014523: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6 Address:a01e87d9-f6ed-1e46-7f7e-9183ed206ca6} {Suffrage:Voter ID:server-9d6dec2a-19b0-5bca-124f-82bd755fff4f Address:9d6dec2a-19b0-5bca-124f-82bd755fff4f} {Suffrage:Voter ID:server-1325ed73-6246-3a9e-a8fb-9cf042014523 Address:1325ed73-6246-3a9e-a8fb-9cf042014523}]" 2025-07-29T00:56:38.774Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:38.774Z [INFO] server-1325ed73-6246-3a9e-a8fb-9cf042014523: entering follower state: follower="Node at 1325ed73-6246-3a9e-a8fb-9cf042014523 [Follower]" leader-address= leader-id= 2025-07-29T00:56:39.606Z [WARN] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:39.606Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: entering candidate state: node="Node at a01e87d9-f6ed-1e46-7f7e-9183ed206ca6 [Candidate]" term=2 2025-07-29T00:56:39.606Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: election won: term=2 tally=2 2025-07-29T00:56:39.606Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: entering leader state: leader="Node at a01e87d9-f6ed-1e46-7f7e-9183ed206ca6 [Leader]" 2025-07-29T00:56:39.606Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: added peer, starting replication: peer=server-9d6dec2a-19b0-5bca-124f-82bd755fff4f 2025-07-29T00:56:39.606Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: added peer, starting replication: peer=server-1325ed73-6246-3a9e-a8fb-9cf042014523 2025-07-29T00:56:39.606Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: pipelining replication: peer="{Voter server-9d6dec2a-19b0-5bca-124f-82bd755fff4f 9d6dec2a-19b0-5bca-124f-82bd755fff4f}" 2025-07-29T00:56:39.607Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: pipelining replication: peer="{Voter server-1325ed73-6246-3a9e-a8fb-9cf042014523 1325ed73-6246-3a9e-a8fb-9cf042014523}" 2025-07-29T00:56:40.612Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:39.606811337 +0000 UTC m=+79.296192681 (1 nodes), 831.83844ms from start of poll, 832.31064ms from cluster start. Timeout at 2025-07-29 00:56:40.612649417 +0000 UTC m=+80.302030761, 1.00583808s after stability 2025-07-29T00:56:40.613Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: starting snapshot up to: index=12 2025-07-29T00:56:40.613Z [INFO] creating new snapshot: path=/tmp/raft2235053314/snapshots/2-12-1753750600613.tmp 2025-07-29T00:56:40.613Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: snapshot complete up to: index=12 2025-07-29T00:56:40.614Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92cffa37-f6b3-8057-57be-e9b27a9875b1 Address:92cffa37-f6b3-8057-57be-e9b27a9875b1} {Suffrage:Voter ID:server-722a8962-007f-8837-f157-ce69f77cabcf Address:722a8962-007f-8837-f157-ce69f77cabcf} {Suffrage:Voter ID:server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae Address:6c9ceff9-81f0-cefb-f3b3-2f573cad4fae}]" 2025-07-29T00:56:40.615Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: entering follower state: follower="Node at 92cffa37-f6b3-8057-57be-e9b27a9875b1 [Follower]" leader-address= leader-id= 2025-07-29T00:56:40.615Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92cffa37-f6b3-8057-57be-e9b27a9875b1 Address:92cffa37-f6b3-8057-57be-e9b27a9875b1} {Suffrage:Voter ID:server-722a8962-007f-8837-f157-ce69f77cabcf Address:722a8962-007f-8837-f157-ce69f77cabcf} {Suffrage:Voter ID:server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae Address:6c9ceff9-81f0-cefb-f3b3-2f573cad4fae}]" 2025-07-29T00:56:40.615Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: entering follower state: follower="Node at 722a8962-007f-8837-f157-ce69f77cabcf [Follower]" leader-address= leader-id= 2025-07-29T00:56:40.615Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92cffa37-f6b3-8057-57be-e9b27a9875b1 Address:92cffa37-f6b3-8057-57be-e9b27a9875b1} {Suffrage:Voter ID:server-722a8962-007f-8837-f157-ce69f77cabcf Address:722a8962-007f-8837-f157-ce69f77cabcf} {Suffrage:Voter ID:server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae Address:6c9ceff9-81f0-cefb-f3b3-2f573cad4fae}]" 2025-07-29T00:56:40.615Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:40.615Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: entering follower state: follower="Node at 6c9ceff9-81f0-cefb-f3b3-2f573cad4fae [Follower]" leader-address= leader-id= 2025-07-29T00:56:41.402Z [WARN] server-722a8962-007f-8837-f157-ce69f77cabcf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:41.402Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: entering candidate state: node="Node at 722a8962-007f-8837-f157-ce69f77cabcf [Candidate]" term=2 2025-07-29T00:56:41.402Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: election won: term=2 tally=2 2025-07-29T00:56:41.402Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: entering leader state: leader="Node at 722a8962-007f-8837-f157-ce69f77cabcf [Leader]" 2025-07-29T00:56:41.402Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: added peer, starting replication: peer=server-92cffa37-f6b3-8057-57be-e9b27a9875b1 2025-07-29T00:56:41.402Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: added peer, starting replication: peer=server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae 2025-07-29T00:56:41.402Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: pipelining replication: peer="{Voter server-92cffa37-f6b3-8057-57be-e9b27a9875b1 92cffa37-f6b3-8057-57be-e9b27a9875b1}" 2025-07-29T00:56:41.402Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: pipelining replication: peer="{Voter server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae 6c9ceff9-81f0-cefb-f3b3-2f573cad4fae}" 2025-07-29T00:56:42.408Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:41.402903257 +0000 UTC m=+81.092284641 (1 nodes), 787.64416ms from start of poll, 788.11212ms from cluster start. Timeout at 2025-07-29 00:56:42.408844977 +0000 UTC m=+82.098226321, 1.00594168s after stability 2025-07-29T00:56:42.409Z [INFO] creating new snapshot: path=/tmp/raft358628200/snapshots/2-1013-1753750602409.tmp 2025-07-29T00:56:42.409Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: copied to local snapshot: bytes=71 2025-07-29T00:56:42.409Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: snapshot restore progress: id=2-1013-1753750602409 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:42.410Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: removing all old logs from log store 2025-07-29T00:56:42.410Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: compacting logs: from=1 to=2 2025-07-29T00:56:42.410Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: restored user snapshot: index=1013 2025-07-29T00:56:42.410Z [ERROR] server-722a8962-007f-8837-f157-ce69f77cabcf: failed to get log: index=2 error="log not found" 2025-07-29T00:56:42.410Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: aborting pipeline replication: peer="{Voter server-92cffa37-f6b3-8057-57be-e9b27a9875b1 92cffa37-f6b3-8057-57be-e9b27a9875b1}" 2025-07-29T00:56:42.410Z [ERROR] server-722a8962-007f-8837-f157-ce69f77cabcf: failed to get log: index=2 error="log not found" 2025-07-29T00:56:42.410Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: aborting pipeline replication: peer="{Voter server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae 6c9ceff9-81f0-cefb-f3b3-2f573cad4fae}" 2025-07-29T00:56:42.416Z [ERROR] server-722a8962-007f-8837-f157-ce69f77cabcf: failed to get log: index=2 error="log not found" 2025-07-29T00:56:42.416Z [INFO] creating new snapshot: path=/tmp/raft1405392283/snapshots/2-1013-1753750602416.tmp 2025-07-29T00:56:42.417Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:42.417Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: copied to local snapshot: bytes=71 2025-07-29T00:56:42.417Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: snapshot restore progress: id=2-1013-1753750602416 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:42.417Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: removing all old logs from log store 2025-07-29T00:56:42.417Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: compacting logs: from=1 to=2 2025-07-29T00:56:42.417Z [INFO] server-92cffa37-f6b3-8057-57be-e9b27a9875b1: Installed remote snapshot 2025-07-29T00:56:42.417Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: pipelining replication: peer="{Voter server-92cffa37-f6b3-8057-57be-e9b27a9875b1 92cffa37-f6b3-8057-57be-e9b27a9875b1}" 2025-07-29T00:56:42.417Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:42.417Z [ERROR] server-722a8962-007f-8837-f157-ce69f77cabcf: failed to get log: index=2 error="log not found" 2025-07-29T00:56:42.418Z [INFO] creating new snapshot: path=/tmp/raft3288975310/snapshots/2-1013-1753750602418.tmp 2025-07-29T00:56:42.418Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:42.418Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: copied to local snapshot: bytes=71 2025-07-29T00:56:42.419Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: snapshot restore progress: id=2-1013-1753750602418 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:42.419Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: removing all old logs from log store 2025-07-29T00:56:42.419Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: compacting logs: from=1 to=2 2025-07-29T00:56:42.419Z [INFO] server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae: Installed remote snapshot 2025-07-29T00:56:42.419Z [INFO] server-722a8962-007f-8837-f157-ce69f77cabcf: pipelining replication: peer="{Voter server-6c9ceff9-81f0-cefb-f3b3-2f573cad4fae 6c9ceff9-81f0-cefb-f3b3-2f573cad4fae}" 2025-07-29T00:56:43.423Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:42.417854657 +0000 UTC m=+82.107236001 (2 nodes), 1.08µs from start of poll, 1.80306348s from cluster start. Timeout at 2025-07-29 00:56:43.423084577 +0000 UTC m=+83.112465921, 1.00522992s after stability 2025-07-29T00:56:43.437Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: aborting pipeline replication: peer="{Voter server-1325ed73-6246-3a9e-a8fb-9cf042014523 1325ed73-6246-3a9e-a8fb-9cf042014523}" 2025-07-29T00:56:43.437Z [INFO] server-a01e87d9-f6ed-1e46-7f7e-9183ed206ca6: aborting pipeline replication: peer="{Voter server-9d6dec2a-19b0-5bca-124f-82bd755fff4f 9d6dec2a-19b0-5bca-124f-82bd755fff4f}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2025-07-29T00:56:43.438Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e84705f-37a0-d48e-1779-0f2085eb5419 Address:9e84705f-37a0-d48e-1779-0f2085eb5419} {Suffrage:Voter ID:server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 Address:3743d6f8-f12a-ebd5-cd12-625a760fbdb8} {Suffrage:Voter ID:server-dd41e4e2-8601-9660-b90e-400acce989ee Address:dd41e4e2-8601-9660-b90e-400acce989ee}]" 2025-07-29T00:56:43.438Z [INFO] server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e84705f-37a0-d48e-1779-0f2085eb5419 Address:9e84705f-37a0-d48e-1779-0f2085eb5419} {Suffrage:Voter ID:server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 Address:3743d6f8-f12a-ebd5-cd12-625a760fbdb8} {Suffrage:Voter ID:server-dd41e4e2-8601-9660-b90e-400acce989ee Address:dd41e4e2-8601-9660-b90e-400acce989ee}]" 2025-07-29T00:56:43.438Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: entering follower state: follower="Node at 9e84705f-37a0-d48e-1779-0f2085eb5419 [Follower]" leader-address= leader-id= 2025-07-29T00:56:43.438Z [INFO] server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8: entering follower state: follower="Node at 3743d6f8-f12a-ebd5-cd12-625a760fbdb8 [Follower]" leader-address= leader-id= 2025-07-29T00:56:43.438Z [INFO] server-dd41e4e2-8601-9660-b90e-400acce989ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e84705f-37a0-d48e-1779-0f2085eb5419 Address:9e84705f-37a0-d48e-1779-0f2085eb5419} {Suffrage:Voter ID:server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 Address:3743d6f8-f12a-ebd5-cd12-625a760fbdb8} {Suffrage:Voter ID:server-dd41e4e2-8601-9660-b90e-400acce989ee Address:dd41e4e2-8601-9660-b90e-400acce989ee}]" 2025-07-29T00:56:43.438Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:43.438Z [INFO] server-dd41e4e2-8601-9660-b90e-400acce989ee: entering follower state: follower="Node at dd41e4e2-8601-9660-b90e-400acce989ee [Follower]" leader-address= leader-id= 2025-07-29T00:56:44.187Z [WARN] server-9e84705f-37a0-d48e-1779-0f2085eb5419: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:44.187Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: entering candidate state: node="Node at 9e84705f-37a0-d48e-1779-0f2085eb5419 [Candidate]" term=2 2025-07-29T00:56:44.187Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: election won: term=2 tally=2 2025-07-29T00:56:44.187Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: entering leader state: leader="Node at 9e84705f-37a0-d48e-1779-0f2085eb5419 [Leader]" 2025-07-29T00:56:44.187Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: added peer, starting replication: peer=server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 2025-07-29T00:56:44.187Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: added peer, starting replication: peer=server-dd41e4e2-8601-9660-b90e-400acce989ee 2025-07-29T00:56:44.188Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: pipelining replication: peer="{Voter server-dd41e4e2-8601-9660-b90e-400acce989ee dd41e4e2-8601-9660-b90e-400acce989ee}" 2025-07-29T00:56:44.188Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: pipelining replication: peer="{Voter server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 3743d6f8-f12a-ebd5-cd12-625a760fbdb8}" 2025-07-29T00:56:45.193Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:44.187991217 +0000 UTC m=+83.877372521 (1 nodes), 749.54204ms from start of poll, 750.03076ms from cluster start. Timeout at 2025-07-29 00:56:45.193415177 +0000 UTC m=+84.882796481, 1.00542396s after stability 2025-07-29T00:56:45.193Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: starting snapshot up to: index=12 2025-07-29T00:56:45.193Z [INFO] creating new snapshot: path=/tmp/raft977837739/snapshots/2-12-1753750605193.tmp 2025-07-29T00:56:45.194Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: snapshot complete up to: index=12 2025-07-29T00:56:45.194Z [INFO] creating new snapshot: path=/tmp/raft977837739/snapshots/2-10013-1753750605194.tmp 2025-07-29T00:56:45.194Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: copied to local snapshot: bytes=71 2025-07-29T00:56:45.195Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: snapshot restore progress: id=2-10013-1753750605194 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:45.195Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: restored user snapshot: index=10013 2025-07-29T00:56:45.195Z [ERROR] server-9e84705f-37a0-d48e-1779-0f2085eb5419: failed to get log: index=23 error="log not found" 2025-07-29T00:56:45.195Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: aborting pipeline replication: peer="{Voter server-dd41e4e2-8601-9660-b90e-400acce989ee dd41e4e2-8601-9660-b90e-400acce989ee}" 2025-07-29T00:56:45.195Z [ERROR] server-9e84705f-37a0-d48e-1779-0f2085eb5419: failed to get log: index=23 error="log not found" 2025-07-29T00:56:45.195Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: aborting pipeline replication: peer="{Voter server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 3743d6f8-f12a-ebd5-cd12-625a760fbdb8}" 2025-07-29T00:56:45.202Z [ERROR] server-9e84705f-37a0-d48e-1779-0f2085eb5419: failed to get log: index=23 error="log not found" 2025-07-29T00:56:45.203Z [ERROR] server-9e84705f-37a0-d48e-1779-0f2085eb5419: failed to get log: index=23 error="log not found" 2025-07-29T00:56:45.203Z [INFO] creating new snapshot: path=/tmp/raft418461898/snapshots/2-10013-1753750605203.tmp 2025-07-29T00:56:45.203Z [INFO] server-dd41e4e2-8601-9660-b90e-400acce989ee: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:45.203Z [INFO] server-dd41e4e2-8601-9660-b90e-400acce989ee: copied to local snapshot: bytes=71 2025-07-29T00:56:45.203Z [INFO] server-dd41e4e2-8601-9660-b90e-400acce989ee: snapshot restore progress: id=2-10013-1753750605203 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:45.203Z [INFO] server-dd41e4e2-8601-9660-b90e-400acce989ee: Installed remote snapshot 2025-07-29T00:56:45.203Z [WARN] server-dd41e4e2-8601-9660-b90e-400acce989ee: skipping application of old log: index=22 2025-07-29T00:56:45.203Z [INFO] creating new snapshot: path=/tmp/raft3117392512/snapshots/2-10013-1753750605203.tmp 2025-07-29T00:56:45.203Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: pipelining replication: peer="{Voter server-dd41e4e2-8601-9660-b90e-400acce989ee dd41e4e2-8601-9660-b90e-400acce989ee}" 2025-07-29T00:56:45.203Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:45.203Z [INFO] server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:45.204Z [INFO] server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8: copied to local snapshot: bytes=71 2025-07-29T00:56:45.204Z [INFO] server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8: snapshot restore progress: id=2-10013-1753750605203 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:45.204Z [INFO] server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8: Installed remote snapshot 2025-07-29T00:56:45.204Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: pipelining replication: peer="{Voter server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 3743d6f8-f12a-ebd5-cd12-625a760fbdb8}" 2025-07-29T00:56:46.209Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:45.203846617 +0000 UTC m=+84.893227921 (2 nodes), 640ns from start of poll, 1.76588616s from cluster start. Timeout at 2025-07-29 00:56:46.209073377 +0000 UTC m=+85.898454681, 1.00522676s after stability 2025-07-29T00:56:46.220Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: aborting pipeline replication: peer="{Voter server-dd41e4e2-8601-9660-b90e-400acce989ee dd41e4e2-8601-9660-b90e-400acce989ee}" 2025-07-29T00:56:46.220Z [INFO] server-9e84705f-37a0-d48e-1779-0f2085eb5419: aborting pipeline replication: peer="{Voter server-3743d6f8-f12a-ebd5-cd12-625a760fbdb8 3743d6f8-f12a-ebd5-cd12-625a760fbdb8}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2025-07-29T00:56:46.221Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 Address:8aef87e9-8bba-85e5-c1e7-71eade44a9c3} {Suffrage:Voter ID:server-0cb9c9b0-8939-e457-7a94-99096f174eac Address:0cb9c9b0-8939-e457-7a94-99096f174eac} {Suffrage:Voter ID:server-84ac5953-c1e8-b02b-aef0-1e499f945aaa Address:84ac5953-c1e8-b02b-aef0-1e499f945aaa}]" 2025-07-29T00:56:46.221Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 Address:8aef87e9-8bba-85e5-c1e7-71eade44a9c3} {Suffrage:Voter ID:server-0cb9c9b0-8939-e457-7a94-99096f174eac Address:0cb9c9b0-8939-e457-7a94-99096f174eac} {Suffrage:Voter ID:server-84ac5953-c1e8-b02b-aef0-1e499f945aaa Address:84ac5953-c1e8-b02b-aef0-1e499f945aaa}]" 2025-07-29T00:56:46.221Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 Address:8aef87e9-8bba-85e5-c1e7-71eade44a9c3} {Suffrage:Voter ID:server-0cb9c9b0-8939-e457-7a94-99096f174eac Address:0cb9c9b0-8939-e457-7a94-99096f174eac} {Suffrage:Voter ID:server-84ac5953-c1e8-b02b-aef0-1e499f945aaa Address:84ac5953-c1e8-b02b-aef0-1e499f945aaa}]" 2025-07-29T00:56:46.221Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:46.221Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: entering follower state: follower="Node at 8aef87e9-8bba-85e5-c1e7-71eade44a9c3 [Follower]" leader-address= leader-id= 2025-07-29T00:56:46.221Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: entering follower state: follower="Node at 0cb9c9b0-8939-e457-7a94-99096f174eac [Follower]" leader-address= leader-id= 2025-07-29T00:56:46.221Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: entering follower state: follower="Node at 84ac5953-c1e8-b02b-aef0-1e499f945aaa [Follower]" leader-address= leader-id= 2025-07-29T00:56:46.735Z [WARN] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:46.735Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: entering candidate state: node="Node at 84ac5953-c1e8-b02b-aef0-1e499f945aaa [Candidate]" term=2 2025-07-29T00:56:46.735Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: election won: term=2 tally=2 2025-07-29T00:56:46.735Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: entering leader state: leader="Node at 84ac5953-c1e8-b02b-aef0-1e499f945aaa [Leader]" 2025-07-29T00:56:46.735Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: added peer, starting replication: peer=server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 2025-07-29T00:56:46.735Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: added peer, starting replication: peer=server-0cb9c9b0-8939-e457-7a94-99096f174eac 2025-07-29T00:56:46.735Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: pipelining replication: peer="{Voter server-0cb9c9b0-8939-e457-7a94-99096f174eac 0cb9c9b0-8939-e457-7a94-99096f174eac}" 2025-07-29T00:56:46.735Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: pipelining replication: peer="{Voter server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 8aef87e9-8bba-85e5-c1e7-71eade44a9c3}" 2025-07-29T00:56:47.741Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:46.735643097 +0000 UTC m=+86.425024361 (1 nodes), 514.12764ms from start of poll, 514.37052ms from cluster start. Timeout at 2025-07-29 00:56:47.741264857 +0000 UTC m=+87.430646201, 1.00562184s after stability 2025-07-29T00:56:47.742Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: starting snapshot up to: index=12 2025-07-29T00:56:47.742Z [INFO] creating new snapshot: path=/tmp/raft1485780794/snapshots/2-12-1753750607742.tmp 2025-07-29T00:56:47.742Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: snapshot complete up to: index=12 2025-07-29T00:56:47.743Z [INFO] creating new snapshot: path=/tmp/raft1485780794/snapshots/2-10013-1753750607743.tmp 2025-07-29T00:56:47.743Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: copied to local snapshot: bytes=71 2025-07-29T00:56:47.744Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: snapshot restore progress: id=2-10013-1753750607743 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:47.744Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: removing all old logs from log store 2025-07-29T00:56:47.744Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: compacting logs: from=1 to=22 2025-07-29T00:56:47.744Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: restored user snapshot: index=10013 2025-07-29T00:56:47.744Z [ERROR] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: failed to get log: index=22 error="log not found" 2025-07-29T00:56:47.744Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: aborting pipeline replication: peer="{Voter server-0cb9c9b0-8939-e457-7a94-99096f174eac 0cb9c9b0-8939-e457-7a94-99096f174eac}" 2025-07-29T00:56:47.744Z [ERROR] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: failed to get log: index=22 error="log not found" 2025-07-29T00:56:47.744Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: aborting pipeline replication: peer="{Voter server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 8aef87e9-8bba-85e5-c1e7-71eade44a9c3}" 2025-07-29T00:56:47.751Z [ERROR] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: failed to get log: index=22 error="log not found" 2025-07-29T00:56:47.751Z [INFO] creating new snapshot: path=/tmp/raft514673753/snapshots/2-10013-1753750607751.tmp 2025-07-29T00:56:47.751Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:47.752Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: copied to local snapshot: bytes=71 2025-07-29T00:56:47.752Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: snapshot restore progress: id=2-10013-1753750607751 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:47.752Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: removing all old logs from log store 2025-07-29T00:56:47.752Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: compacting logs: from=1 to=22 2025-07-29T00:56:47.752Z [INFO] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: Installed remote snapshot 2025-07-29T00:56:47.752Z [WARN] server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3: skipping application of old log: index=22 2025-07-29T00:56:47.752Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: pipelining replication: peer="{Voter server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 8aef87e9-8bba-85e5-c1e7-71eade44a9c3}" 2025-07-29T00:56:47.752Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:47.753Z [ERROR] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: failed to get log: index=22 error="log not found" 2025-07-29T00:56:47.754Z [INFO] creating new snapshot: path=/tmp/raft1631152900/snapshots/2-10013-1753750607754.tmp 2025-07-29T00:56:47.754Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:47.754Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: copied to local snapshot: bytes=71 2025-07-29T00:56:47.754Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: snapshot restore progress: id=2-10013-1753750607754 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:47.755Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: removing all old logs from log store 2025-07-29T00:56:47.755Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: compacting logs: from=1 to=22 2025-07-29T00:56:47.755Z [INFO] server-0cb9c9b0-8939-e457-7a94-99096f174eac: Installed remote snapshot 2025-07-29T00:56:47.755Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: pipelining replication: peer="{Voter server-0cb9c9b0-8939-e457-7a94-99096f174eac 0cb9c9b0-8939-e457-7a94-99096f174eac}" 2025-07-29T00:56:48.758Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:47.752510537 +0000 UTC m=+87.441891881 (2 nodes), 1.44µs from start of poll, 1.53123804s from cluster start. Timeout at 2025-07-29 00:56:48.758010417 +0000 UTC m=+88.447391761, 1.00549988s after stability 2025-07-29T00:56:48.769Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: aborting pipeline replication: peer="{Voter server-0cb9c9b0-8939-e457-7a94-99096f174eac 0cb9c9b0-8939-e457-7a94-99096f174eac}" 2025-07-29T00:56:48.769Z [INFO] server-84ac5953-c1e8-b02b-aef0-1e499f945aaa: aborting pipeline replication: peer="{Voter server-8aef87e9-8bba-85e5-c1e7-71eade44a9c3 8aef87e9-8bba-85e5-c1e7-71eade44a9c3}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2025-07-29T00:56:48.770Z [INFO] server-ad6f682e-daa8-8f82-5d52-87f65e431182: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad6f682e-daa8-8f82-5d52-87f65e431182 Address:ad6f682e-daa8-8f82-5d52-87f65e431182} {Suffrage:Voter ID:server-6e5c6378-e11c-052e-0f18-135af66d44f1 Address:6e5c6378-e11c-052e-0f18-135af66d44f1} {Suffrage:Voter ID:server-1cae3ca9-2c86-806f-c103-a817e83fb92d Address:1cae3ca9-2c86-806f-c103-a817e83fb92d}]" 2025-07-29T00:56:48.770Z [INFO] server-ad6f682e-daa8-8f82-5d52-87f65e431182: entering follower state: follower="Node at ad6f682e-daa8-8f82-5d52-87f65e431182 [Follower]" leader-address= leader-id= 2025-07-29T00:56:48.770Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad6f682e-daa8-8f82-5d52-87f65e431182 Address:ad6f682e-daa8-8f82-5d52-87f65e431182} {Suffrage:Voter ID:server-6e5c6378-e11c-052e-0f18-135af66d44f1 Address:6e5c6378-e11c-052e-0f18-135af66d44f1} {Suffrage:Voter ID:server-1cae3ca9-2c86-806f-c103-a817e83fb92d Address:1cae3ca9-2c86-806f-c103-a817e83fb92d}]" 2025-07-29T00:56:48.770Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: entering follower state: follower="Node at 6e5c6378-e11c-052e-0f18-135af66d44f1 [Follower]" leader-address= leader-id= 2025-07-29T00:56:48.770Z [INFO] server-1cae3ca9-2c86-806f-c103-a817e83fb92d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad6f682e-daa8-8f82-5d52-87f65e431182 Address:ad6f682e-daa8-8f82-5d52-87f65e431182} {Suffrage:Voter ID:server-6e5c6378-e11c-052e-0f18-135af66d44f1 Address:6e5c6378-e11c-052e-0f18-135af66d44f1} {Suffrage:Voter ID:server-1cae3ca9-2c86-806f-c103-a817e83fb92d Address:1cae3ca9-2c86-806f-c103-a817e83fb92d}]" 2025-07-29T00:56:48.770Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:48.770Z [INFO] server-1cae3ca9-2c86-806f-c103-a817e83fb92d: entering follower state: follower="Node at 1cae3ca9-2c86-806f-c103-a817e83fb92d [Follower]" leader-address= leader-id= 2025-07-29T00:56:49.437Z [WARN] server-6e5c6378-e11c-052e-0f18-135af66d44f1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:49.437Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: entering candidate state: node="Node at 6e5c6378-e11c-052e-0f18-135af66d44f1 [Candidate]" term=2 2025-07-29T00:56:49.437Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: election won: term=2 tally=2 2025-07-29T00:56:49.438Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: entering leader state: leader="Node at 6e5c6378-e11c-052e-0f18-135af66d44f1 [Leader]" 2025-07-29T00:56:49.438Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: added peer, starting replication: peer=server-ad6f682e-daa8-8f82-5d52-87f65e431182 2025-07-29T00:56:49.438Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: added peer, starting replication: peer=server-1cae3ca9-2c86-806f-c103-a817e83fb92d 2025-07-29T00:56:49.438Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: pipelining replication: peer="{Voter server-1cae3ca9-2c86-806f-c103-a817e83fb92d 1cae3ca9-2c86-806f-c103-a817e83fb92d}" 2025-07-29T00:56:49.438Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: pipelining replication: peer="{Voter server-ad6f682e-daa8-8f82-5d52-87f65e431182 ad6f682e-daa8-8f82-5d52-87f65e431182}" 2025-07-29T00:56:50.444Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:49.438053257 +0000 UTC m=+89.127434601 (1 nodes), 667.30064ms from start of poll, 667.76144ms from cluster start. Timeout at 2025-07-29 00:56:50.444061577 +0000 UTC m=+90.133442921, 1.00600832s after stability 2025-07-29T00:56:50.444Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: starting snapshot up to: index=12 2025-07-29T00:56:50.444Z [INFO] creating new snapshot: path=/tmp/raft2231767988/snapshots/2-12-1753750610444.tmp 2025-07-29T00:56:50.445Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: snapshot complete up to: index=12 2025-07-29T00:56:50.446Z [INFO] server-70b785c8-4995-6714-7ee9-c1f320dfaf5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70b785c8-4995-6714-7ee9-c1f320dfaf5e Address:70b785c8-4995-6714-7ee9-c1f320dfaf5e} {Suffrage:Voter ID:server-365a410c-de38-b2a1-a021-efe494cee44c Address:365a410c-de38-b2a1-a021-efe494cee44c} {Suffrage:Voter ID:server-5b6ce571-588e-8830-3251-aec5785e447e Address:5b6ce571-588e-8830-3251-aec5785e447e}]" 2025-07-29T00:56:50.446Z [INFO] server-70b785c8-4995-6714-7ee9-c1f320dfaf5e: entering follower state: follower="Node at 70b785c8-4995-6714-7ee9-c1f320dfaf5e [Follower]" leader-address= leader-id= 2025-07-29T00:56:50.446Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70b785c8-4995-6714-7ee9-c1f320dfaf5e Address:70b785c8-4995-6714-7ee9-c1f320dfaf5e} {Suffrage:Voter ID:server-365a410c-de38-b2a1-a021-efe494cee44c Address:365a410c-de38-b2a1-a021-efe494cee44c} {Suffrage:Voter ID:server-5b6ce571-588e-8830-3251-aec5785e447e Address:5b6ce571-588e-8830-3251-aec5785e447e}]" 2025-07-29T00:56:50.446Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: entering follower state: follower="Node at 365a410c-de38-b2a1-a021-efe494cee44c [Follower]" leader-address= leader-id= 2025-07-29T00:56:50.446Z [INFO] server-5b6ce571-588e-8830-3251-aec5785e447e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70b785c8-4995-6714-7ee9-c1f320dfaf5e Address:70b785c8-4995-6714-7ee9-c1f320dfaf5e} {Suffrage:Voter ID:server-365a410c-de38-b2a1-a021-efe494cee44c Address:365a410c-de38-b2a1-a021-efe494cee44c} {Suffrage:Voter ID:server-5b6ce571-588e-8830-3251-aec5785e447e Address:5b6ce571-588e-8830-3251-aec5785e447e}]" 2025-07-29T00:56:50.446Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:50.446Z [INFO] server-5b6ce571-588e-8830-3251-aec5785e447e: entering follower state: follower="Node at 5b6ce571-588e-8830-3251-aec5785e447e [Follower]" leader-address= leader-id= 2025-07-29T00:56:50.957Z [WARN] server-365a410c-de38-b2a1-a021-efe494cee44c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:50.957Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: entering candidate state: node="Node at 365a410c-de38-b2a1-a021-efe494cee44c [Candidate]" term=2 2025-07-29T00:56:50.957Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: election won: term=2 tally=2 2025-07-29T00:56:50.957Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: entering leader state: leader="Node at 365a410c-de38-b2a1-a021-efe494cee44c [Leader]" 2025-07-29T00:56:50.957Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: added peer, starting replication: peer=server-70b785c8-4995-6714-7ee9-c1f320dfaf5e 2025-07-29T00:56:50.957Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: added peer, starting replication: peer=server-5b6ce571-588e-8830-3251-aec5785e447e 2025-07-29T00:56:50.958Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: pipelining replication: peer="{Voter server-5b6ce571-588e-8830-3251-aec5785e447e 5b6ce571-588e-8830-3251-aec5785e447e}" 2025-07-29T00:56:50.958Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: pipelining replication: peer="{Voter server-70b785c8-4995-6714-7ee9-c1f320dfaf5e 70b785c8-4995-6714-7ee9-c1f320dfaf5e}" 2025-07-29T00:56:51.963Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:50.957888417 +0000 UTC m=+90.647269761 (1 nodes), 511.23476ms from start of poll, 511.73196ms from cluster start. Timeout at 2025-07-29 00:56:51.963096017 +0000 UTC m=+91.652477361, 1.0052076s after stability 2025-07-29T00:56:51.963Z [INFO] creating new snapshot: path=/tmp/raft2608203870/snapshots/2-10013-1753750611963.tmp 2025-07-29T00:56:51.963Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: copied to local snapshot: bytes=71 2025-07-29T00:56:51.964Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: snapshot restore progress: id=2-10013-1753750611963 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:51.964Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: restored user snapshot: index=10013 2025-07-29T00:56:51.964Z [ERROR] server-365a410c-de38-b2a1-a021-efe494cee44c: failed to get log: index=3 error="log not found" 2025-07-29T00:56:51.964Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: aborting pipeline replication: peer="{Voter server-70b785c8-4995-6714-7ee9-c1f320dfaf5e 70b785c8-4995-6714-7ee9-c1f320dfaf5e}" 2025-07-29T00:56:51.964Z [ERROR] server-365a410c-de38-b2a1-a021-efe494cee44c: failed to get log: index=3 error="log not found" 2025-07-29T00:56:51.964Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: aborting pipeline replication: peer="{Voter server-5b6ce571-588e-8830-3251-aec5785e447e 5b6ce571-588e-8830-3251-aec5785e447e}" 2025-07-29T00:56:51.972Z [ERROR] server-365a410c-de38-b2a1-a021-efe494cee44c: failed to get log: index=3 error="log not found" 2025-07-29T00:56:51.972Z [INFO] creating new snapshot: path=/tmp/raft3238078059/snapshots/2-10013-1753750611972.tmp 2025-07-29T00:56:51.972Z [INFO] server-5b6ce571-588e-8830-3251-aec5785e447e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:51.973Z [INFO] server-5b6ce571-588e-8830-3251-aec5785e447e: copied to local snapshot: bytes=71 2025-07-29T00:56:51.973Z [INFO] server-5b6ce571-588e-8830-3251-aec5785e447e: snapshot restore progress: id=2-10013-1753750611972 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:51.973Z [ERROR] server-365a410c-de38-b2a1-a021-efe494cee44c: failed to get log: index=3 error="log not found" 2025-07-29T00:56:51.973Z [INFO] server-5b6ce571-588e-8830-3251-aec5785e447e: Installed remote snapshot 2025-07-29T00:56:51.974Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: pipelining replication: peer="{Voter server-5b6ce571-588e-8830-3251-aec5785e447e 5b6ce571-588e-8830-3251-aec5785e447e}" 2025-07-29T00:56:51.974Z [INFO] creating new snapshot: path=/tmp/raft2655915225/snapshots/2-10013-1753750611974.tmp 2025-07-29T00:56:51.974Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:51.974Z [INFO] server-70b785c8-4995-6714-7ee9-c1f320dfaf5e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:51.974Z [INFO] server-70b785c8-4995-6714-7ee9-c1f320dfaf5e: copied to local snapshot: bytes=71 2025-07-29T00:56:51.975Z [INFO] server-70b785c8-4995-6714-7ee9-c1f320dfaf5e: snapshot restore progress: id=2-10013-1753750611974 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:51.975Z [INFO] server-70b785c8-4995-6714-7ee9-c1f320dfaf5e: Installed remote snapshot 2025-07-29T00:56:51.976Z [INFO] server-365a410c-de38-b2a1-a021-efe494cee44c: pipelining replication: peer="{Voter server-70b785c8-4995-6714-7ee9-c1f320dfaf5e 70b785c8-4995-6714-7ee9-c1f320dfaf5e}" 2025-07-29T00:56:52.979Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:51.974650377 +0000 UTC m=+91.664031721 (2 nodes), 2.64µs from start of poll, 1.52849392s from cluster start. Timeout at 2025-07-29 00:56:52.979940977 +0000 UTC m=+92.669322321, 1.0052906s after stability 2025-07-29T00:56:52.992Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: aborting pipeline replication: peer="{Voter server-ad6f682e-daa8-8f82-5d52-87f65e431182 ad6f682e-daa8-8f82-5d52-87f65e431182}" 2025-07-29T00:56:52.992Z [INFO] server-6e5c6378-e11c-052e-0f18-135af66d44f1: aborting pipeline replication: peer="{Voter server-1cae3ca9-2c86-806f-c103-a817e83fb92d 1cae3ca9-2c86-806f-c103-a817e83fb92d}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2025-07-29T00:56:52.993Z [INFO] server-9d61675e-97c5-db1c-879a-a250905ad6cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d61675e-97c5-db1c-879a-a250905ad6cc Address:9d61675e-97c5-db1c-879a-a250905ad6cc} {Suffrage:Voter ID:server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f Address:b4f29b31-7b1f-9430-1ffa-97a0ad5b601f} {Suffrage:Voter ID:server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f Address:9d21acd1-f4e8-0f4b-e737-1e03691a1d0f}]" 2025-07-29T00:56:52.993Z [INFO] server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d61675e-97c5-db1c-879a-a250905ad6cc Address:9d61675e-97c5-db1c-879a-a250905ad6cc} {Suffrage:Voter ID:server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f Address:b4f29b31-7b1f-9430-1ffa-97a0ad5b601f} {Suffrage:Voter ID:server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f Address:9d21acd1-f4e8-0f4b-e737-1e03691a1d0f}]" 2025-07-29T00:56:52.993Z [INFO] server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f: entering follower state: follower="Node at b4f29b31-7b1f-9430-1ffa-97a0ad5b601f [Follower]" leader-address= leader-id= 2025-07-29T00:56:52.993Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d61675e-97c5-db1c-879a-a250905ad6cc Address:9d61675e-97c5-db1c-879a-a250905ad6cc} {Suffrage:Voter ID:server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f Address:b4f29b31-7b1f-9430-1ffa-97a0ad5b601f} {Suffrage:Voter ID:server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f Address:9d21acd1-f4e8-0f4b-e737-1e03691a1d0f}]" 2025-07-29T00:56:52.993Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:52.993Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: entering follower state: follower="Node at 9d21acd1-f4e8-0f4b-e737-1e03691a1d0f [Follower]" leader-address= leader-id= 2025-07-29T00:56:52.993Z [INFO] server-9d61675e-97c5-db1c-879a-a250905ad6cc: entering follower state: follower="Node at 9d61675e-97c5-db1c-879a-a250905ad6cc [Follower]" leader-address= leader-id= 2025-07-29T00:56:53.555Z [WARN] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:53.555Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: entering candidate state: node="Node at 9d21acd1-f4e8-0f4b-e737-1e03691a1d0f [Candidate]" term=2 2025-07-29T00:56:53.555Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: election won: term=2 tally=2 2025-07-29T00:56:53.555Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: entering leader state: leader="Node at 9d21acd1-f4e8-0f4b-e737-1e03691a1d0f [Leader]" 2025-07-29T00:56:53.555Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: added peer, starting replication: peer=server-9d61675e-97c5-db1c-879a-a250905ad6cc 2025-07-29T00:56:53.555Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: added peer, starting replication: peer=server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f 2025-07-29T00:56:53.555Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: pipelining replication: peer="{Voter server-9d61675e-97c5-db1c-879a-a250905ad6cc 9d61675e-97c5-db1c-879a-a250905ad6cc}" 2025-07-29T00:56:53.555Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: pipelining replication: peer="{Voter server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f b4f29b31-7b1f-9430-1ffa-97a0ad5b601f}" 2025-07-29T00:56:54.561Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:53.555529057 +0000 UTC m=+93.244910401 (1 nodes), 561.61128ms from start of poll, 562.09644ms from cluster start. Timeout at 2025-07-29 00:56:54.561181257 +0000 UTC m=+94.250562601, 1.0056522s after stability 2025-07-29T00:56:54.561Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: starting snapshot up to: index=12 2025-07-29T00:56:54.562Z [INFO] creating new snapshot: path=/tmp/raft1970288918/snapshots/2-12-1753750614562.tmp 2025-07-29T00:56:54.562Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: snapshot complete up to: index=12 2025-07-29T00:56:54.563Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae0b6161-22d9-6009-7e2a-df9c88934623 Address:ae0b6161-22d9-6009-7e2a-df9c88934623} {Suffrage:Voter ID:server-e99efa7a-5600-1b4c-6679-b9b2e3d08355 Address:e99efa7a-5600-1b4c-6679-b9b2e3d08355} {Suffrage:Voter ID:server-348333a0-6b1d-4cab-ff64-8a3d13723735 Address:348333a0-6b1d-4cab-ff64-8a3d13723735}]" 2025-07-29T00:56:54.563Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: entering follower state: follower="Node at ae0b6161-22d9-6009-7e2a-df9c88934623 [Follower]" leader-address= leader-id= 2025-07-29T00:56:54.563Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae0b6161-22d9-6009-7e2a-df9c88934623 Address:ae0b6161-22d9-6009-7e2a-df9c88934623} {Suffrage:Voter ID:server-e99efa7a-5600-1b4c-6679-b9b2e3d08355 Address:e99efa7a-5600-1b4c-6679-b9b2e3d08355} {Suffrage:Voter ID:server-348333a0-6b1d-4cab-ff64-8a3d13723735 Address:348333a0-6b1d-4cab-ff64-8a3d13723735}]" 2025-07-29T00:56:54.563Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: entering follower state: follower="Node at e99efa7a-5600-1b4c-6679-b9b2e3d08355 [Follower]" leader-address= leader-id= 2025-07-29T00:56:54.563Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae0b6161-22d9-6009-7e2a-df9c88934623 Address:ae0b6161-22d9-6009-7e2a-df9c88934623} {Suffrage:Voter ID:server-e99efa7a-5600-1b4c-6679-b9b2e3d08355 Address:e99efa7a-5600-1b4c-6679-b9b2e3d08355} {Suffrage:Voter ID:server-348333a0-6b1d-4cab-ff64-8a3d13723735 Address:348333a0-6b1d-4cab-ff64-8a3d13723735}]" 2025-07-29T00:56:54.563Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:54.563Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: entering follower state: follower="Node at 348333a0-6b1d-4cab-ff64-8a3d13723735 [Follower]" leader-address= leader-id= 2025-07-29T00:56:55.156Z [WARN] server-ae0b6161-22d9-6009-7e2a-df9c88934623: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:55.156Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: entering candidate state: node="Node at ae0b6161-22d9-6009-7e2a-df9c88934623 [Candidate]" term=2 2025-07-29T00:56:55.156Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: election won: term=2 tally=2 2025-07-29T00:56:55.156Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: entering leader state: leader="Node at ae0b6161-22d9-6009-7e2a-df9c88934623 [Leader]" 2025-07-29T00:56:55.156Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: added peer, starting replication: peer=server-e99efa7a-5600-1b4c-6679-b9b2e3d08355 2025-07-29T00:56:55.156Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: added peer, starting replication: peer=server-348333a0-6b1d-4cab-ff64-8a3d13723735 2025-07-29T00:56:55.156Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: pipelining replication: peer="{Voter server-348333a0-6b1d-4cab-ff64-8a3d13723735 348333a0-6b1d-4cab-ff64-8a3d13723735}" 2025-07-29T00:56:55.156Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: pipelining replication: peer="{Voter server-e99efa7a-5600-1b4c-6679-b9b2e3d08355 e99efa7a-5600-1b4c-6679-b9b2e3d08355}" 2025-07-29T00:56:56.162Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:55.156656417 +0000 UTC m=+94.846037761 (1 nodes), 592.73792ms from start of poll, 593.19648ms from cluster start. Timeout at 2025-07-29 00:56:56.162192697 +0000 UTC m=+95.851574041, 1.00553628s after stability 2025-07-29T00:56:56.162Z [INFO] creating new snapshot: path=/tmp/raft3946818392/snapshots/2-10013-1753750616162.tmp 2025-07-29T00:56:56.163Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: copied to local snapshot: bytes=71 2025-07-29T00:56:56.163Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: snapshot restore progress: id=2-10013-1753750616162 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:56.163Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: removing all old logs from log store 2025-07-29T00:56:56.163Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: compacting logs: from=1 to=2 2025-07-29T00:56:56.163Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: restored user snapshot: index=10013 2025-07-29T00:56:56.163Z [ERROR] server-ae0b6161-22d9-6009-7e2a-df9c88934623: failed to get log: index=2 error="log not found" 2025-07-29T00:56:56.163Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: aborting pipeline replication: peer="{Voter server-348333a0-6b1d-4cab-ff64-8a3d13723735 348333a0-6b1d-4cab-ff64-8a3d13723735}" 2025-07-29T00:56:56.163Z [ERROR] server-ae0b6161-22d9-6009-7e2a-df9c88934623: failed to get log: index=2 error="log not found" 2025-07-29T00:56:56.163Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: aborting pipeline replication: peer="{Voter server-e99efa7a-5600-1b4c-6679-b9b2e3d08355 e99efa7a-5600-1b4c-6679-b9b2e3d08355}" 2025-07-29T00:56:56.172Z [ERROR] server-ae0b6161-22d9-6009-7e2a-df9c88934623: failed to get log: index=2 error="log not found" 2025-07-29T00:56:56.172Z [ERROR] server-ae0b6161-22d9-6009-7e2a-df9c88934623: failed to get log: index=2 error="log not found" 2025-07-29T00:56:56.172Z [INFO] creating new snapshot: path=/tmp/raft1404169397/snapshots/2-10013-1753750616172.tmp 2025-07-29T00:56:56.172Z [INFO] creating new snapshot: path=/tmp/raft1563604996/snapshots/2-10013-1753750616172.tmp 2025-07-29T00:56:56.173Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:56.173Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:56.173Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: copied to local snapshot: bytes=71 2025-07-29T00:56:56.173Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: copied to local snapshot: bytes=71 2025-07-29T00:56:56.173Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: snapshot restore progress: id=2-10013-1753750616172 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:56.173Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: removing all old logs from log store 2025-07-29T00:56:56.173Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: compacting logs: from=1 to=2 2025-07-29T00:56:56.173Z [INFO] server-348333a0-6b1d-4cab-ff64-8a3d13723735: Installed remote snapshot 2025-07-29T00:56:56.173Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: snapshot restore progress: id=2-10013-1753750616172 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-07-29T00:56:56.173Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: removing all old logs from log store 2025-07-29T00:56:56.173Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: compacting logs: from=1 to=2 2025-07-29T00:56:56.173Z [INFO] server-e99efa7a-5600-1b4c-6679-b9b2e3d08355: Installed remote snapshot 2025-07-29T00:56:56.173Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: pipelining replication: peer="{Voter server-348333a0-6b1d-4cab-ff64-8a3d13723735 348333a0-6b1d-4cab-ff64-8a3d13723735}" 2025-07-29T00:56:56.173Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:56.173Z [INFO] server-ae0b6161-22d9-6009-7e2a-df9c88934623: pipelining replication: peer="{Voter server-e99efa7a-5600-1b4c-6679-b9b2e3d08355 e99efa7a-5600-1b4c-6679-b9b2e3d08355}" 2025-07-29T00:56:57.179Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:56.173950817 +0000 UTC m=+95.863332161 (2 nodes), 1.08µs from start of poll, 1.61049088s from cluster start. Timeout at 2025-07-29 00:56:57.179482697 +0000 UTC m=+96.868864041, 1.00553188s after stability 2025-07-29T00:56:57.191Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: aborting pipeline replication: peer="{Voter server-9d61675e-97c5-db1c-879a-a250905ad6cc 9d61675e-97c5-db1c-879a-a250905ad6cc}" 2025-07-29T00:56:57.191Z [INFO] server-9d21acd1-f4e8-0f4b-e737-1e03691a1d0f: aborting pipeline replication: peer="{Voter server-b4f29b31-7b1f-9430-1ffa-97a0ad5b601f b4f29b31-7b1f-9430-1ffa-97a0ad5b601f}" --- PASS: TestRaft_UserRestore (83.80s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.75s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.75s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.27s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.37s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.56s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.61s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.19s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.12s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.66s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.71s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.08s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.25s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.69s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.61s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.27s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.37s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.82s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.83s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.47s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.66s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.78s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.55s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.22s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.20s) === RUN TestRaft_SendSnapshotFollower 2025-07-29T00:56:57.192Z [INFO] server-77171696-11b7-4b84-11a1-eb760d71e118: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77171696-11b7-4b84-11a1-eb760d71e118 Address:77171696-11b7-4b84-11a1-eb760d71e118} {Suffrage:Voter ID:server-67f7577b-128a-7070-f9f9-c31340c8916c Address:67f7577b-128a-7070-f9f9-c31340c8916c} {Suffrage:Voter ID:server-6d158eab-a45e-e722-53b5-845c7173385f Address:6d158eab-a45e-e722-53b5-845c7173385f}]" 2025-07-29T00:56:57.192Z [INFO] server-77171696-11b7-4b84-11a1-eb760d71e118: entering follower state: follower="Node at 77171696-11b7-4b84-11a1-eb760d71e118 [Follower]" leader-address= leader-id= 2025-07-29T00:56:57.192Z [INFO] server-67f7577b-128a-7070-f9f9-c31340c8916c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77171696-11b7-4b84-11a1-eb760d71e118 Address:77171696-11b7-4b84-11a1-eb760d71e118} {Suffrage:Voter ID:server-67f7577b-128a-7070-f9f9-c31340c8916c Address:67f7577b-128a-7070-f9f9-c31340c8916c} {Suffrage:Voter ID:server-6d158eab-a45e-e722-53b5-845c7173385f Address:6d158eab-a45e-e722-53b5-845c7173385f}]" 2025-07-29T00:56:57.192Z [INFO] server-67f7577b-128a-7070-f9f9-c31340c8916c: entering follower state: follower="Node at 67f7577b-128a-7070-f9f9-c31340c8916c [Follower]" leader-address= leader-id= 2025-07-29T00:56:57.192Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77171696-11b7-4b84-11a1-eb760d71e118 Address:77171696-11b7-4b84-11a1-eb760d71e118} {Suffrage:Voter ID:server-67f7577b-128a-7070-f9f9-c31340c8916c Address:67f7577b-128a-7070-f9f9-c31340c8916c} {Suffrage:Voter ID:server-6d158eab-a45e-e722-53b5-845c7173385f Address:6d158eab-a45e-e722-53b5-845c7173385f}]" 2025-07-29T00:56:57.192Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:57.192Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: entering follower state: follower="Node at 6d158eab-a45e-e722-53b5-845c7173385f [Follower]" leader-address= leader-id= 2025-07-29T00:56:57.246Z [WARN] server-6d158eab-a45e-e722-53b5-845c7173385f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:57.246Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: entering candidate state: node="Node at 6d158eab-a45e-e722-53b5-845c7173385f [Candidate]" term=2 2025-07-29T00:56:57.246Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: election won: term=2 tally=2 2025-07-29T00:56:57.246Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: entering leader state: leader="Node at 6d158eab-a45e-e722-53b5-845c7173385f [Leader]" 2025-07-29T00:56:57.246Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: added peer, starting replication: peer=server-77171696-11b7-4b84-11a1-eb760d71e118 2025-07-29T00:56:57.246Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: added peer, starting replication: peer=server-67f7577b-128a-7070-f9f9-c31340c8916c 2025-07-29T00:56:57.246Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: pipelining replication: peer="{Voter server-77171696-11b7-4b84-11a1-eb760d71e118 77171696-11b7-4b84-11a1-eb760d71e118}" 2025-07-29T00:56:57.246Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: pipelining replication: peer="{Voter server-67f7577b-128a-7070-f9f9-c31340c8916c 67f7577b-128a-7070-f9f9-c31340c8916c}" 2025-07-29T00:56:57.352Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:57.246447497 +0000 UTC m=+96.935828841 (2 nodes), 54.03732ms from start of poll, 54.48012ms from cluster start. Timeout at 2025-07-29 00:56:57.352061017 +0000 UTC m=+97.041442361, 105.61352ms after stability 2025-07-29T00:56:57.352Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:57.457Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:57.352191337 +0000 UTC m=+97.041572721 (1 nodes), 2.76µs from start of poll, 160.224ms from cluster start. Timeout at 2025-07-29 00:56:57.457780817 +0000 UTC m=+97.147162161, 105.58944ms after stability 2025-07-29T00:56:57.457Z [ERROR] server-6d158eab-a45e-e722-53b5-845c7173385f: failed to pipeline appendEntries: peer="{Voter server-77171696-11b7-4b84-11a1-eb760d71e118 77171696-11b7-4b84-11a1-eb760d71e118}" error="append pipeline closed" 2025-07-29T00:56:57.457Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: aborting pipeline replication: peer="{Voter server-77171696-11b7-4b84-11a1-eb760d71e118 77171696-11b7-4b84-11a1-eb760d71e118}" 2025-07-29T00:56:57.458Z [ERROR] server-6d158eab-a45e-e722-53b5-845c7173385f: failed to appendEntries to: peer="{Voter server-77171696-11b7-4b84-11a1-eb760d71e118 77171696-11b7-4b84-11a1-eb760d71e118}" error="failed to connect to peer: 77171696-11b7-4b84-11a1-eb760d71e118" raft_test.go:1627: [INFO] Finished apply without behind follower 2025-07-29T00:56:57.463Z [ERROR] server-77171696-11b7-4b84-11a1-eb760d71e118: failed to take snapshot: error="nothing new to snapshot" 2025-07-29T00:56:57.463Z [INFO] server-67f7577b-128a-7070-f9f9-c31340c8916c: starting snapshot up to: index=99 2025-07-29T00:56:57.463Z [INFO] creating new snapshot: path=/tmp/raft3528490907/snapshots/2-99-1753750617463.tmp 2025-07-29T00:56:57.463Z [INFO] server-67f7577b-128a-7070-f9f9-c31340c8916c: compacting logs: from=1 to=92 2025-07-29T00:56:57.463Z [INFO] server-67f7577b-128a-7070-f9f9-c31340c8916c: snapshot complete up to: index=99 2025-07-29T00:56:57.464Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: starting snapshot up to: index=102 2025-07-29T00:56:57.464Z [INFO] creating new snapshot: path=/tmp/raft612033847/snapshots/2-102-1753750617464.tmp 2025-07-29T00:56:57.464Z [ERROR] server-6d158eab-a45e-e722-53b5-845c7173385f: failed to heartbeat to: peer=77171696-11b7-4b84-11a1-eb760d71e118 backoff time=10ms error="failed to connect to peer: 77171696-11b7-4b84-11a1-eb760d71e118" 2025-07-29T00:56:57.465Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: compacting logs: from=1 to=92 2025-07-29T00:56:57.465Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: snapshot complete up to: index=102 2025-07-29T00:56:57.469Z [ERROR] server-6d158eab-a45e-e722-53b5-845c7173385f: failed to get log: index=2 error="log not found" 2025-07-29T00:56:57.469Z [INFO] creating new snapshot: path=/tmp/raft3248488597/snapshots/2-102-1753750617469.tmp 2025-07-29T00:56:57.469Z [INFO] server-77171696-11b7-4b84-11a1-eb760d71e118: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2025-07-29T00:56:57.469Z [INFO] server-77171696-11b7-4b84-11a1-eb760d71e118: copied to local snapshot: bytes=693 2025-07-29T00:56:57.469Z [INFO] server-77171696-11b7-4b84-11a1-eb760d71e118: snapshot restore progress: id=2-102-1753750617469 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-07-29T00:56:57.469Z [INFO] server-77171696-11b7-4b84-11a1-eb760d71e118: Installed remote snapshot 2025-07-29T00:56:57.469Z [WARN] server-77171696-11b7-4b84-11a1-eb760d71e118: skipping application of old log: index=102 2025-07-29T00:56:57.469Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: pipelining replication: peer="{Voter server-77171696-11b7-4b84-11a1-eb760d71e118 77171696-11b7-4b84-11a1-eb760d71e118}" 2025-07-29T00:56:57.476Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: aborting pipeline replication: peer="{Voter server-77171696-11b7-4b84-11a1-eb760d71e118 77171696-11b7-4b84-11a1-eb760d71e118}" 2025-07-29T00:56:57.476Z [INFO] server-6d158eab-a45e-e722-53b5-845c7173385f: aborting pipeline replication: peer="{Voter server-67f7577b-128a-7070-f9f9-c31340c8916c 67f7577b-128a-7070-f9f9-c31340c8916c}" 2025-07-29T00:56:57.977Z [ERROR] server-6d158eab-a45e-e722-53b5-845c7173385f: failed to heartbeat to: peer=67f7577b-128a-7070-f9f9-c31340c8916c backoff time=10ms error="command timed out" --- PASS: TestRaft_SendSnapshotFollower (0.79s) === RUN TestRaft_SendSnapshotAndLogsFollower 2025-07-29T00:56:57.978Z [INFO] server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 Address:d8e18bf5-ab90-627c-7f53-9b54e8bd0454} {Suffrage:Voter ID:server-31333703-de41-003d-1bbb-e57ee68d6737 Address:31333703-de41-003d-1bbb-e57ee68d6737} {Suffrage:Voter ID:server-2cb9c805-2e14-8b9d-4803-d256e67e7d69 Address:2cb9c805-2e14-8b9d-4803-d256e67e7d69}]" 2025-07-29T00:56:57.978Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 Address:d8e18bf5-ab90-627c-7f53-9b54e8bd0454} {Suffrage:Voter ID:server-31333703-de41-003d-1bbb-e57ee68d6737 Address:31333703-de41-003d-1bbb-e57ee68d6737} {Suffrage:Voter ID:server-2cb9c805-2e14-8b9d-4803-d256e67e7d69 Address:2cb9c805-2e14-8b9d-4803-d256e67e7d69}]" 2025-07-29T00:56:57.978Z [INFO] server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454: entering follower state: follower="Node at d8e18bf5-ab90-627c-7f53-9b54e8bd0454 [Follower]" leader-address= leader-id= 2025-07-29T00:56:57.978Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: entering follower state: follower="Node at 31333703-de41-003d-1bbb-e57ee68d6737 [Follower]" leader-address= leader-id= 2025-07-29T00:56:57.978Z [INFO] server-2cb9c805-2e14-8b9d-4803-d256e67e7d69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 Address:d8e18bf5-ab90-627c-7f53-9b54e8bd0454} {Suffrage:Voter ID:server-31333703-de41-003d-1bbb-e57ee68d6737 Address:31333703-de41-003d-1bbb-e57ee68d6737} {Suffrage:Voter ID:server-2cb9c805-2e14-8b9d-4803-d256e67e7d69 Address:2cb9c805-2e14-8b9d-4803-d256e67e7d69}]" 2025-07-29T00:56:57.978Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:57.978Z [INFO] server-2cb9c805-2e14-8b9d-4803-d256e67e7d69: entering follower state: follower="Node at 2cb9c805-2e14-8b9d-4803-d256e67e7d69 [Follower]" leader-address= leader-id= 2025-07-29T00:56:58.037Z [WARN] server-31333703-de41-003d-1bbb-e57ee68d6737: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:58.037Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: entering candidate state: node="Node at 31333703-de41-003d-1bbb-e57ee68d6737 [Candidate]" term=2 2025-07-29T00:56:58.037Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: election won: term=2 tally=2 2025-07-29T00:56:58.037Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: entering leader state: leader="Node at 31333703-de41-003d-1bbb-e57ee68d6737 [Leader]" 2025-07-29T00:56:58.037Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: added peer, starting replication: peer=server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 2025-07-29T00:56:58.037Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: added peer, starting replication: peer=server-2cb9c805-2e14-8b9d-4803-d256e67e7d69 2025-07-29T00:56:58.037Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: pipelining replication: peer="{Voter server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 d8e18bf5-ab90-627c-7f53-9b54e8bd0454}" 2025-07-29T00:56:58.037Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: pipelining replication: peer="{Voter server-2cb9c805-2e14-8b9d-4803-d256e67e7d69 2cb9c805-2e14-8b9d-4803-d256e67e7d69}" 2025-07-29T00:56:58.142Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:58.037622337 +0000 UTC m=+97.727003681 (2 nodes), 58.7548ms from start of poll, 59.2832ms from cluster start. Timeout at 2025-07-29 00:56:58.142744137 +0000 UTC m=+97.832125481, 105.1218ms after stability 2025-07-29T00:56:58.142Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:58.248Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:58.142903057 +0000 UTC m=+97.832284401 (1 nodes), 2.68µs from start of poll, 164.56392ms from cluster start. Timeout at 2025-07-29 00:56:58.248306337 +0000 UTC m=+97.937687681, 105.40328ms after stability 2025-07-29T00:56:58.248Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to pipeline appendEntries: peer="{Voter server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 d8e18bf5-ab90-627c-7f53-9b54e8bd0454}" error="append pipeline closed" 2025-07-29T00:56:58.248Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: aborting pipeline replication: peer="{Voter server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 d8e18bf5-ab90-627c-7f53-9b54e8bd0454}" 2025-07-29T00:56:58.248Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to appendEntries to: peer="{Voter server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 d8e18bf5-ab90-627c-7f53-9b54e8bd0454}" error="failed to connect to peer: d8e18bf5-ab90-627c-7f53-9b54e8bd0454" 2025-07-29T00:56:58.255Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to heartbeat to: peer=d8e18bf5-ab90-627c-7f53-9b54e8bd0454 backoff time=10ms error="failed to connect to peer: d8e18bf5-ab90-627c-7f53-9b54e8bd0454" raft_test.go:1669: [INFO] Finished apply without behind follower 2025-07-29T00:56:58.258Z [ERROR] server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454: failed to take snapshot: error="nothing new to snapshot" 2025-07-29T00:56:58.258Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: starting snapshot up to: index=102 2025-07-29T00:56:58.258Z [INFO] creating new snapshot: path=/tmp/raft1604222156/snapshots/2-102-1753750618258.tmp 2025-07-29T00:56:58.258Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: compacting logs: from=1 to=92 2025-07-29T00:56:58.258Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: snapshot complete up to: index=102 2025-07-29T00:56:58.258Z [INFO] server-2cb9c805-2e14-8b9d-4803-d256e67e7d69: starting snapshot up to: index=95 2025-07-29T00:56:58.258Z [INFO] creating new snapshot: path=/tmp/raft1418374476/snapshots/2-95-1753750618258.tmp 2025-07-29T00:56:58.258Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to get log: index=2 error="log not found" 2025-07-29T00:56:58.259Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to install snapshot: id=2-102-1753750618258 error="failed to connect to peer: d8e18bf5-ab90-627c-7f53-9b54e8bd0454" 2025-07-29T00:56:58.259Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to send snapshot to: peer="{Voter server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 d8e18bf5-ab90-627c-7f53-9b54e8bd0454}" error="failed to connect to peer: d8e18bf5-ab90-627c-7f53-9b54e8bd0454" 2025-07-29T00:56:58.259Z [INFO] server-2cb9c805-2e14-8b9d-4803-d256e67e7d69: compacting logs: from=1 to=92 2025-07-29T00:56:58.259Z [INFO] server-2cb9c805-2e14-8b9d-4803-d256e67e7d69: snapshot complete up to: index=95 raft_test.go:1690: [INFO] Finished apply without behind follower 2025-07-29T00:56:58.270Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to get log: index=2 error="log not found" 2025-07-29T00:56:58.270Z [INFO] creating new snapshot: path=/tmp/raft503717426/snapshots/2-102-1753750618270.tmp 2025-07-29T00:56:58.271Z [INFO] server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2025-07-29T00:56:58.271Z [INFO] server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454: copied to local snapshot: bytes=693 2025-07-29T00:56:58.271Z [INFO] server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454: snapshot restore progress: id=2-102-1753750618270 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-07-29T00:56:58.271Z [INFO] server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454: Installed remote snapshot 2025-07-29T00:56:58.272Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: pipelining replication: peer="{Voter server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 d8e18bf5-ab90-627c-7f53-9b54e8bd0454}" 2025-07-29T00:56:58.274Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: aborting pipeline replication: peer="{Voter server-2cb9c805-2e14-8b9d-4803-d256e67e7d69 2cb9c805-2e14-8b9d-4803-d256e67e7d69}" 2025-07-29T00:56:58.274Z [INFO] server-31333703-de41-003d-1bbb-e57ee68d6737: aborting pipeline replication: peer="{Voter server-d8e18bf5-ab90-627c-7f53-9b54e8bd0454 d8e18bf5-ab90-627c-7f53-9b54e8bd0454}" 2025-07-29T00:56:58.775Z [ERROR] server-31333703-de41-003d-1bbb-e57ee68d6737: failed to heartbeat to: peer=2cb9c805-2e14-8b9d-4803-d256e67e7d69 backoff time=10ms error="command timed out" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.80s) === RUN TestRaft_ReJoinFollower 2025-07-29T00:56:58.776Z [INFO] server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d Address:2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d} {Suffrage:Voter ID:server-b96aafc9-22b0-066e-97ce-b7ca23edee61 Address:b96aafc9-22b0-066e-97ce-b7ca23edee61} {Suffrage:Voter ID:server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 Address:256fdf97-6e73-3a6a-b5d1-44a5337a37d9}]" 2025-07-29T00:56:58.776Z [INFO] server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d: entering follower state: follower="Node at 2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d [Follower]" leader-address= leader-id= 2025-07-29T00:56:58.776Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d Address:2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d} {Suffrage:Voter ID:server-b96aafc9-22b0-066e-97ce-b7ca23edee61 Address:b96aafc9-22b0-066e-97ce-b7ca23edee61} {Suffrage:Voter ID:server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 Address:256fdf97-6e73-3a6a-b5d1-44a5337a37d9}]" 2025-07-29T00:56:58.776Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: entering follower state: follower="Node at b96aafc9-22b0-066e-97ce-b7ca23edee61 [Follower]" leader-address= leader-id= 2025-07-29T00:56:58.776Z [INFO] server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d Address:2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d} {Suffrage:Voter ID:server-b96aafc9-22b0-066e-97ce-b7ca23edee61 Address:b96aafc9-22b0-066e-97ce-b7ca23edee61} {Suffrage:Voter ID:server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 Address:256fdf97-6e73-3a6a-b5d1-44a5337a37d9}]" 2025-07-29T00:56:58.776Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:58.777Z [INFO] server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9: entering follower state: follower="Node at 256fdf97-6e73-3a6a-b5d1-44a5337a37d9 [Follower]" leader-address= leader-id= 2025-07-29T00:56:58.851Z [WARN] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:58.851Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: entering candidate state: node="Node at b96aafc9-22b0-066e-97ce-b7ca23edee61 [Candidate]" term=2 2025-07-29T00:56:58.851Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: election won: term=2 tally=2 2025-07-29T00:56:58.851Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: entering leader state: leader="Node at b96aafc9-22b0-066e-97ce-b7ca23edee61 [Leader]" 2025-07-29T00:56:58.851Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: added peer, starting replication: peer=server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d 2025-07-29T00:56:58.851Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: added peer, starting replication: peer=server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 2025-07-29T00:56:58.851Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: pipelining replication: peer="{Voter server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 256fdf97-6e73-3a6a-b5d1-44a5337a37d9}" 2025-07-29T00:56:58.851Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: pipelining replication: peer="{Voter server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d 2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d}" 2025-07-29T00:56:58.957Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:58.851718497 +0000 UTC m=+98.541099841 (1 nodes), 74.722ms from start of poll, 75.21252ms from cluster start. Timeout at 2025-07-29 00:56:58.957852377 +0000 UTC m=+98.647233721, 106.13388ms after stability 2025-07-29T00:56:58.962Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:59.068Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:58.963026017 +0000 UTC m=+98.652407361 (2 nodes), 3.56µs from start of poll, 186.52004ms from cluster start. Timeout at 2025-07-29 00:56:59.068528897 +0000 UTC m=+98.757910241, 105.50288ms after stability 2025-07-29T00:56:59.068Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: updating configuration: command=RemoveServer server-id=server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d server-addr= servers="[{Suffrage:Voter ID:server-b96aafc9-22b0-066e-97ce-b7ca23edee61 Address:b96aafc9-22b0-066e-97ce-b7ca23edee61} {Suffrage:Voter ID:server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 Address:256fdf97-6e73-3a6a-b5d1-44a5337a37d9}]" 2025-07-29T00:56:59.068Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: removed peer, stopping replication: peer=server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d last-index=3 2025-07-29T00:56:59.068Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: aborting pipeline replication: peer="{Voter server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d 2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d}" 2025-07-29T00:56:59.156Z [WARN] server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-07-29T00:56:59.174Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: updating configuration: command=AddVoter server-id=server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d server-addr=2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d servers="[{Suffrage:Voter ID:server-b96aafc9-22b0-066e-97ce-b7ca23edee61 Address:b96aafc9-22b0-066e-97ce-b7ca23edee61} {Suffrage:Voter ID:server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 Address:256fdf97-6e73-3a6a-b5d1-44a5337a37d9} {Suffrage:Voter ID:server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d Address:2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d}]" 2025-07-29T00:56:59.175Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: added peer, starting replication: peer=server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d 2025-07-29T00:56:59.175Z [WARN] server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2025-07-29T00:56:59.175Z [WARN] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: appendEntries rejected, sending older logs: peer="{Voter server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d 2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d}" next=4 2025-07-29T00:56:59.175Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:59.175Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: pipelining replication: peer="{Voter server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d 2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d}" 2025-07-29T00:56:59.280Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:59.175333937 +0000 UTC m=+98.864715281 (1 nodes), 960ns from start of poll, 398.82796ms from cluster start. Timeout at 2025-07-29 00:56:59.280337057 +0000 UTC m=+98.969718401, 105.00312ms after stability 2025-07-29T00:56:59.280Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: aborting pipeline replication: peer="{Voter server-256fdf97-6e73-3a6a-b5d1-44a5337a37d9 256fdf97-6e73-3a6a-b5d1-44a5337a37d9}" 2025-07-29T00:56:59.280Z [INFO] server-b96aafc9-22b0-066e-97ce-b7ca23edee61: aborting pipeline replication: peer="{Voter server-2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d 2b6d53ff-70ad-e4c3-1d14-8846e4d83b8d}" --- PASS: TestRaft_ReJoinFollower (0.50s) === RUN TestRaft_LeaderLeaseExpire 2025-07-29T00:56:59.281Z [INFO] server-f16314bd-263c-8018-a155-dc89f2ca36ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f16314bd-263c-8018-a155-dc89f2ca36ac Address:f16314bd-263c-8018-a155-dc89f2ca36ac} {Suffrage:Voter ID:server-993206b6-88c8-4100-0790-14aca470cf16 Address:993206b6-88c8-4100-0790-14aca470cf16}]" 2025-07-29T00:56:59.281Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f16314bd-263c-8018-a155-dc89f2ca36ac Address:f16314bd-263c-8018-a155-dc89f2ca36ac} {Suffrage:Voter ID:server-993206b6-88c8-4100-0790-14aca470cf16 Address:993206b6-88c8-4100-0790-14aca470cf16}]" 2025-07-29T00:56:59.281Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:59.281Z [INFO] server-f16314bd-263c-8018-a155-dc89f2ca36ac: entering follower state: follower="Node at f16314bd-263c-8018-a155-dc89f2ca36ac [Follower]" leader-address= leader-id= 2025-07-29T00:56:59.281Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: entering follower state: follower="Node at 993206b6-88c8-4100-0790-14aca470cf16 [Follower]" leader-address= leader-id= 2025-07-29T00:56:59.364Z [WARN] server-993206b6-88c8-4100-0790-14aca470cf16: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:59.364Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: entering candidate state: node="Node at 993206b6-88c8-4100-0790-14aca470cf16 [Candidate]" term=2 2025-07-29T00:56:59.364Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: election won: term=2 tally=2 2025-07-29T00:56:59.364Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: entering leader state: leader="Node at 993206b6-88c8-4100-0790-14aca470cf16 [Leader]" 2025-07-29T00:56:59.364Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: added peer, starting replication: peer=server-f16314bd-263c-8018-a155-dc89f2ca36ac 2025-07-29T00:56:59.364Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: pipelining replication: peer="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" 2025-07-29T00:56:59.470Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:59.364753977 +0000 UTC m=+99.054135361 (1 nodes), 83.2824ms from start of poll, 83.63392ms from cluster start. Timeout at 2025-07-29 00:56:59.470720257 +0000 UTC m=+99.160101601, 105.96624ms after stability 2025-07-29T00:56:59.476Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:56:59.581Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:56:59.476247817 +0000 UTC m=+99.165629161 (1 nodes), 3.16µs from start of poll, 195.12772ms from cluster start. Timeout at 2025-07-29 00:56:59.581303337 +0000 UTC m=+99.270684681, 105.05552ms after stability raft_test.go:1799: [INFO] Disconnecting Node at f16314bd-263c-8018-a155-dc89f2ca36ac [Follower] 2025-07-29T00:56:59.583Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to heartbeat to: peer=f16314bd-263c-8018-a155-dc89f2ca36ac backoff time=10ms error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" 2025-07-29T00:56:59.587Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to pipeline appendEntries: peer="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" error="append pipeline closed" 2025-07-29T00:56:59.587Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: aborting pipeline replication: peer="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" 2025-07-29T00:56:59.593Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to appendEntries to: peer="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" 2025-07-29T00:56:59.602Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to heartbeat to: peer=f16314bd-263c-8018-a155-dc89f2ca36ac backoff time=10ms error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" 2025-07-29T00:56:59.611Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to appendEntries to: peer="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" 2025-07-29T00:56:59.621Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to heartbeat to: peer=f16314bd-263c-8018-a155-dc89f2ca36ac backoff time=10ms error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" 2025-07-29T00:56:59.628Z [WARN] server-993206b6-88c8-4100-0790-14aca470cf16: failed to contact: server-id=server-f16314bd-263c-8018-a155-dc89f2ca36ac time=50.57032ms 2025-07-29T00:56:59.628Z [WARN] server-993206b6-88c8-4100-0790-14aca470cf16: failed to contact quorum of nodes, stepping down 2025-07-29T00:56:59.628Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: entering follower state: follower="Node at 993206b6-88c8-4100-0790-14aca470cf16 [Follower]" leader-address= leader-id= 2025-07-29T00:56:59.628Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:59.630Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to appendEntries to: peer="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" 2025-07-29T00:56:59.642Z [WARN] server-f16314bd-263c-8018-a155-dc89f2ca36ac: heartbeat timeout reached, starting election: last-leader-addr=993206b6-88c8-4100-0790-14aca470cf16 last-leader-id=server-993206b6-88c8-4100-0790-14aca470cf16 2025-07-29T00:56:59.642Z [INFO] server-f16314bd-263c-8018-a155-dc89f2ca36ac: entering candidate state: node="Node at f16314bd-263c-8018-a155-dc89f2ca36ac [Candidate]" term=3 2025-07-29T00:56:59.642Z [ERROR] server-f16314bd-263c-8018-a155-dc89f2ca36ac: failed to make requestVote RPC: target="{Voter server-993206b6-88c8-4100-0790-14aca470cf16 993206b6-88c8-4100-0790-14aca470cf16}" error="failed to connect to peer: 993206b6-88c8-4100-0790-14aca470cf16" term=3 2025-07-29T00:56:59.679Z [WARN] server-993206b6-88c8-4100-0790-14aca470cf16: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:59.679Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: entering candidate state: node="Node at 993206b6-88c8-4100-0790-14aca470cf16 [Candidate]" term=3 2025-07-29T00:56:59.679Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to make requestVote RPC: target="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" term=3 2025-07-29T00:56:59.729Z [WARN] server-f16314bd-263c-8018-a155-dc89f2ca36ac: Election timeout reached, restarting election 2025-07-29T00:56:59.729Z [INFO] server-f16314bd-263c-8018-a155-dc89f2ca36ac: entering candidate state: node="Node at f16314bd-263c-8018-a155-dc89f2ca36ac [Candidate]" term=4 2025-07-29T00:56:59.729Z [ERROR] server-f16314bd-263c-8018-a155-dc89f2ca36ac: failed to make requestVote RPC: target="{Voter server-993206b6-88c8-4100-0790-14aca470cf16 993206b6-88c8-4100-0790-14aca470cf16}" error="failed to connect to peer: 993206b6-88c8-4100-0790-14aca470cf16" term=4 2025-07-29T00:56:59.771Z [WARN] server-993206b6-88c8-4100-0790-14aca470cf16: Election timeout reached, restarting election 2025-07-29T00:56:59.771Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: entering candidate state: node="Node at 993206b6-88c8-4100-0790-14aca470cf16 [Candidate]" term=4 2025-07-29T00:56:59.771Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to make requestVote RPC: target="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" term=4 2025-07-29T00:56:59.785Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:59.679683657 +0000 UTC m=+99.369065001 (0 nodes), 51.44396ms from start of poll, 398.56356ms from cluster start. Timeout at 2025-07-29 00:56:59.785692337 +0000 UTC m=+99.475073681, 106.00868ms after stability 2025-07-29T00:56:59.823Z [WARN] server-f16314bd-263c-8018-a155-dc89f2ca36ac: Election timeout reached, restarting election 2025-07-29T00:56:59.823Z [INFO] server-f16314bd-263c-8018-a155-dc89f2ca36ac: entering candidate state: node="Node at f16314bd-263c-8018-a155-dc89f2ca36ac [Candidate]" term=5 2025-07-29T00:56:59.823Z [ERROR] server-f16314bd-263c-8018-a155-dc89f2ca36ac: failed to make requestVote RPC: target="{Voter server-993206b6-88c8-4100-0790-14aca470cf16 993206b6-88c8-4100-0790-14aca470cf16}" error="failed to connect to peer: 993206b6-88c8-4100-0790-14aca470cf16" term=5 2025-07-29T00:56:59.845Z [WARN] server-993206b6-88c8-4100-0790-14aca470cf16: Election timeout reached, restarting election 2025-07-29T00:56:59.845Z [INFO] server-993206b6-88c8-4100-0790-14aca470cf16: entering candidate state: node="Node at 993206b6-88c8-4100-0790-14aca470cf16 [Candidate]" term=5 2025-07-29T00:56:59.846Z [ERROR] server-993206b6-88c8-4100-0790-14aca470cf16: failed to make requestVote RPC: target="{Voter server-f16314bd-263c-8018-a155-dc89f2ca36ac f16314bd-263c-8018-a155-dc89f2ca36ac}" error="failed to connect to peer: f16314bd-263c-8018-a155-dc89f2ca36ac" term=5 2025-07-29T00:56:59.880Z [WARN] server-f16314bd-263c-8018-a155-dc89f2ca36ac: Election timeout reached, restarting election 2025-07-29T00:56:59.880Z [INFO] server-f16314bd-263c-8018-a155-dc89f2ca36ac: entering candidate state: node="Node at f16314bd-263c-8018-a155-dc89f2ca36ac [Candidate]" term=6 2025-07-29T00:56:59.880Z [ERROR] server-f16314bd-263c-8018-a155-dc89f2ca36ac: failed to make requestVote RPC: target="{Voter server-993206b6-88c8-4100-0790-14aca470cf16 993206b6-88c8-4100-0790-14aca470cf16}" error="failed to connect to peer: 993206b6-88c8-4100-0790-14aca470cf16" term=6 --- PASS: TestRaft_LeaderLeaseExpire (0.61s) === RUN TestRaft_Barrier 2025-07-29T00:56:59.891Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a4bb9c-b171-32d0-8a06-840bd22d687f Address:16a4bb9c-b171-32d0-8a06-840bd22d687f} {Suffrage:Voter ID:server-0adf5c42-5484-aca3-3ca3-3ae508d35de4 Address:0adf5c42-5484-aca3-3ca3-3ae508d35de4} {Suffrage:Voter ID:server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625 Address:d2254012-7aa0-995a-d4c2-0cfa0b1f3625}]" 2025-07-29T00:56:59.891Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: entering follower state: follower="Node at 16a4bb9c-b171-32d0-8a06-840bd22d687f [Follower]" leader-address= leader-id= 2025-07-29T00:56:59.891Z [INFO] server-0adf5c42-5484-aca3-3ca3-3ae508d35de4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a4bb9c-b171-32d0-8a06-840bd22d687f Address:16a4bb9c-b171-32d0-8a06-840bd22d687f} {Suffrage:Voter ID:server-0adf5c42-5484-aca3-3ca3-3ae508d35de4 Address:0adf5c42-5484-aca3-3ca3-3ae508d35de4} {Suffrage:Voter ID:server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625 Address:d2254012-7aa0-995a-d4c2-0cfa0b1f3625}]" 2025-07-29T00:56:59.891Z [INFO] server-0adf5c42-5484-aca3-3ca3-3ae508d35de4: entering follower state: follower="Node at 0adf5c42-5484-aca3-3ca3-3ae508d35de4 [Follower]" leader-address= leader-id= 2025-07-29T00:56:59.892Z [INFO] server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a4bb9c-b171-32d0-8a06-840bd22d687f Address:16a4bb9c-b171-32d0-8a06-840bd22d687f} {Suffrage:Voter ID:server-0adf5c42-5484-aca3-3ca3-3ae508d35de4 Address:0adf5c42-5484-aca3-3ca3-3ae508d35de4} {Suffrage:Voter ID:server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625 Address:d2254012-7aa0-995a-d4c2-0cfa0b1f3625}]" 2025-07-29T00:56:59.892Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:56:59.892Z [INFO] server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625: entering follower state: follower="Node at d2254012-7aa0-995a-d4c2-0cfa0b1f3625 [Follower]" leader-address= leader-id= 2025-07-29T00:56:59.945Z [WARN] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:56:59.946Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: entering candidate state: node="Node at 16a4bb9c-b171-32d0-8a06-840bd22d687f [Candidate]" term=2 2025-07-29T00:56:59.946Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: election won: term=2 tally=2 2025-07-29T00:56:59.946Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: entering leader state: leader="Node at 16a4bb9c-b171-32d0-8a06-840bd22d687f [Leader]" 2025-07-29T00:56:59.946Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: added peer, starting replication: peer=server-0adf5c42-5484-aca3-3ca3-3ae508d35de4 2025-07-29T00:56:59.946Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: added peer, starting replication: peer=server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625 2025-07-29T00:56:59.946Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: pipelining replication: peer="{Voter server-0adf5c42-5484-aca3-3ca3-3ae508d35de4 0adf5c42-5484-aca3-3ca3-3ae508d35de4}" 2025-07-29T00:56:59.946Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: pipelining replication: peer="{Voter server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625 d2254012-7aa0-995a-d4c2-0cfa0b1f3625}" 2025-07-29T00:57:00.051Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:56:59.946182657 +0000 UTC m=+99.635564001 (1 nodes), 53.97308ms from start of poll, 54.6166ms from cluster start. Timeout at 2025-07-29 00:57:00.051256337 +0000 UTC m=+99.740637681, 105.07368ms after stability 2025-07-29T00:57:00.072Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: aborting pipeline replication: peer="{Voter server-d2254012-7aa0-995a-d4c2-0cfa0b1f3625 d2254012-7aa0-995a-d4c2-0cfa0b1f3625}" 2025-07-29T00:57:00.072Z [INFO] server-16a4bb9c-b171-32d0-8a06-840bd22d687f: aborting pipeline replication: peer="{Voter server-0adf5c42-5484-aca3-3ca3-3ae508d35de4 0adf5c42-5484-aca3-3ca3-3ae508d35de4}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader 2025-07-29T00:57:00.072Z [INFO] server-3d1dc424-7e71-d321-00e4-5402bb566d8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d1dc424-7e71-d321-00e4-5402bb566d8b Address:3d1dc424-7e71-d321-00e4-5402bb566d8b} {Suffrage:Voter ID:server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d Address:bb689f4b-42c6-52d6-1ce6-5c0408284f9d} {Suffrage:Voter ID:server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b Address:0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b}]" 2025-07-29T00:57:00.073Z [INFO] server-3d1dc424-7e71-d321-00e4-5402bb566d8b: entering follower state: follower="Node at 3d1dc424-7e71-d321-00e4-5402bb566d8b [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.073Z [INFO] server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d1dc424-7e71-d321-00e4-5402bb566d8b Address:3d1dc424-7e71-d321-00e4-5402bb566d8b} {Suffrage:Voter ID:server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d Address:bb689f4b-42c6-52d6-1ce6-5c0408284f9d} {Suffrage:Voter ID:server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b Address:0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b}]" 2025-07-29T00:57:00.073Z [INFO] server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d: entering follower state: follower="Node at bb689f4b-42c6-52d6-1ce6-5c0408284f9d [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.073Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d1dc424-7e71-d321-00e4-5402bb566d8b Address:3d1dc424-7e71-d321-00e4-5402bb566d8b} {Suffrage:Voter ID:server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d Address:bb689f4b-42c6-52d6-1ce6-5c0408284f9d} {Suffrage:Voter ID:server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b Address:0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b}]" 2025-07-29T00:57:00.073Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:00.073Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: entering follower state: follower="Node at 0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.150Z [WARN] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:00.150Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: entering candidate state: node="Node at 0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b [Candidate]" term=2 2025-07-29T00:57:00.151Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: election won: term=2 tally=2 2025-07-29T00:57:00.151Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: entering leader state: leader="Node at 0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b [Leader]" 2025-07-29T00:57:00.151Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: added peer, starting replication: peer=server-3d1dc424-7e71-d321-00e4-5402bb566d8b 2025-07-29T00:57:00.151Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: added peer, starting replication: peer=server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d 2025-07-29T00:57:00.151Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: pipelining replication: peer="{Voter server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d bb689f4b-42c6-52d6-1ce6-5c0408284f9d}" 2025-07-29T00:57:00.151Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: pipelining replication: peer="{Voter server-3d1dc424-7e71-d321-00e4-5402bb566d8b 3d1dc424-7e71-d321-00e4-5402bb566d8b}" 2025-07-29T00:57:00.256Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:00.151245977 +0000 UTC m=+99.840627321 (1 nodes), 77.9974ms from start of poll, 78.4522ms from cluster start. Timeout at 2025-07-29 00:57:00.256469057 +0000 UTC m=+99.945850401, 105.22308ms after stability 2025-07-29T00:57:00.256Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: aborting pipeline replication: peer="{Voter server-bb689f4b-42c6-52d6-1ce6-5c0408284f9d bb689f4b-42c6-52d6-1ce6-5c0408284f9d}" 2025-07-29T00:57:00.256Z [INFO] server-0ed2e5bb-c8cd-aa28-0920-c7ce5d58af5b: aborting pipeline replication: peer="{Voter server-3d1dc424-7e71-d321-00e4-5402bb566d8b 3d1dc424-7e71-d321-00e4-5402bb566d8b}" --- PASS: TestRaft_VerifyLeader (0.18s) === RUN TestRaft_VerifyLeader_Single 2025-07-29T00:57:00.257Z [INFO] server-89b1f2cf-6624-5e44-828a-23ca44364e28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89b1f2cf-6624-5e44-828a-23ca44364e28 Address:89b1f2cf-6624-5e44-828a-23ca44364e28}]" 2025-07-29T00:57:00.257Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:00.257Z [INFO] server-89b1f2cf-6624-5e44-828a-23ca44364e28: entering follower state: follower="Node at 89b1f2cf-6624-5e44-828a-23ca44364e28 [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.342Z [WARN] server-89b1f2cf-6624-5e44-828a-23ca44364e28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:00.342Z [INFO] server-89b1f2cf-6624-5e44-828a-23ca44364e28: entering candidate state: node="Node at 89b1f2cf-6624-5e44-828a-23ca44364e28 [Candidate]" term=2 2025-07-29T00:57:00.342Z [INFO] server-89b1f2cf-6624-5e44-828a-23ca44364e28: election won: term=2 tally=1 2025-07-29T00:57:00.342Z [INFO] server-89b1f2cf-6624-5e44-828a-23ca44364e28: entering leader state: leader="Node at 89b1f2cf-6624-5e44-828a-23ca44364e28 [Leader]" 2025-07-29T00:57:00.447Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:00.342246897 +0000 UTC m=+100.031628241 (1 nodes), 84.69644ms from start of poll, 84.91956ms from cluster start. Timeout at 2025-07-29 00:57:00.447561737 +0000 UTC m=+100.136943081, 105.31484ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.19s) === RUN TestRaft_VerifyLeader_Fail 2025-07-29T00:57:00.448Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d4a78f4-2f8b-0114-27f4-729b3272ba51 Address:1d4a78f4-2f8b-0114-27f4-729b3272ba51} {Suffrage:Voter ID:server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51 Address:b4497a9f-0200-a2cb-7b1a-0b14a73b1f51}]" 2025-07-29T00:57:00.448Z [INFO] server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d4a78f4-2f8b-0114-27f4-729b3272ba51 Address:1d4a78f4-2f8b-0114-27f4-729b3272ba51} {Suffrage:Voter ID:server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51 Address:b4497a9f-0200-a2cb-7b1a-0b14a73b1f51}]" 2025-07-29T00:57:00.448Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:00.448Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: entering follower state: follower="Node at 1d4a78f4-2f8b-0114-27f4-729b3272ba51 [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.448Z [INFO] server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51: entering follower state: follower="Node at b4497a9f-0200-a2cb-7b1a-0b14a73b1f51 [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.532Z [WARN] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:00.532Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: entering candidate state: node="Node at 1d4a78f4-2f8b-0114-27f4-729b3272ba51 [Candidate]" term=2 2025-07-29T00:57:00.532Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: election won: term=2 tally=2 2025-07-29T00:57:00.532Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: entering leader state: leader="Node at 1d4a78f4-2f8b-0114-27f4-729b3272ba51 [Leader]" 2025-07-29T00:57:00.533Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: added peer, starting replication: peer=server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51 2025-07-29T00:57:00.533Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: pipelining replication: peer="{Voter server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51 b4497a9f-0200-a2cb-7b1a-0b14a73b1f51}" 2025-07-29T00:57:00.638Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:00.533028617 +0000 UTC m=+100.222409961 (1 nodes), 84.41788ms from start of poll, 84.76344ms from cluster start. Timeout at 2025-07-29 00:57:00.638460337 +0000 UTC m=+100.327841681, 105.43172ms after stability 2025-07-29T00:57:00.638Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:00.743Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:00.638568497 +0000 UTC m=+100.327949841 (1 nodes), 2.52µs from start of poll, 190.30332ms from cluster start. Timeout at 2025-07-29 00:57:00.743584457 +0000 UTC m=+100.432965801, 105.01596ms after stability 2025-07-29T00:57:00.747Z [ERROR] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: peer has newer term, stopping replication: peer="{Voter server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51 b4497a9f-0200-a2cb-7b1a-0b14a73b1f51}" 2025-07-29T00:57:00.747Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: aborting pipeline replication: peer="{Voter server-b4497a9f-0200-a2cb-7b1a-0b14a73b1f51 b4497a9f-0200-a2cb-7b1a-0b14a73b1f51}" 2025-07-29T00:57:00.747Z [INFO] server-1d4a78f4-2f8b-0114-27f4-729b3272ba51: entering follower state: follower="Node at 1d4a78f4-2f8b-0114-27f4-729b3272ba51 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.30s) === RUN TestRaft_VerifyLeader_PartialConnect 2025-07-29T00:57:00.748Z [INFO] server-725e5ca0-3831-60f9-efd1-26afcbaed2a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725e5ca0-3831-60f9-efd1-26afcbaed2a2 Address:725e5ca0-3831-60f9-efd1-26afcbaed2a2} {Suffrage:Voter ID:server-7d9014e5-d4c4-7107-81c1-c9af0cc02719 Address:7d9014e5-d4c4-7107-81c1-c9af0cc02719} {Suffrage:Voter ID:server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0 Address:d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0}]" 2025-07-29T00:57:00.748Z [INFO] server-725e5ca0-3831-60f9-efd1-26afcbaed2a2: entering follower state: follower="Node at 725e5ca0-3831-60f9-efd1-26afcbaed2a2 [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.748Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725e5ca0-3831-60f9-efd1-26afcbaed2a2 Address:725e5ca0-3831-60f9-efd1-26afcbaed2a2} {Suffrage:Voter ID:server-7d9014e5-d4c4-7107-81c1-c9af0cc02719 Address:7d9014e5-d4c4-7107-81c1-c9af0cc02719} {Suffrage:Voter ID:server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0 Address:d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0}]" 2025-07-29T00:57:00.748Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: entering follower state: follower="Node at 7d9014e5-d4c4-7107-81c1-c9af0cc02719 [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.748Z [INFO] server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725e5ca0-3831-60f9-efd1-26afcbaed2a2 Address:725e5ca0-3831-60f9-efd1-26afcbaed2a2} {Suffrage:Voter ID:server-7d9014e5-d4c4-7107-81c1-c9af0cc02719 Address:7d9014e5-d4c4-7107-81c1-c9af0cc02719} {Suffrage:Voter ID:server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0 Address:d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0}]" 2025-07-29T00:57:00.748Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:00.748Z [INFO] server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0: entering follower state: follower="Node at d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0 [Follower]" leader-address= leader-id= 2025-07-29T00:57:00.801Z [WARN] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:00.801Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: entering candidate state: node="Node at 7d9014e5-d4c4-7107-81c1-c9af0cc02719 [Candidate]" term=2 2025-07-29T00:57:00.801Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: election won: term=2 tally=2 2025-07-29T00:57:00.801Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: entering leader state: leader="Node at 7d9014e5-d4c4-7107-81c1-c9af0cc02719 [Leader]" 2025-07-29T00:57:00.801Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: added peer, starting replication: peer=server-725e5ca0-3831-60f9-efd1-26afcbaed2a2 2025-07-29T00:57:00.801Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: added peer, starting replication: peer=server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0 2025-07-29T00:57:00.801Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: pipelining replication: peer="{Voter server-725e5ca0-3831-60f9-efd1-26afcbaed2a2 725e5ca0-3831-60f9-efd1-26afcbaed2a2}" 2025-07-29T00:57:00.801Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: pipelining replication: peer="{Voter server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0 d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0}" 2025-07-29T00:57:00.907Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:00.801808057 +0000 UTC m=+100.491189401 (1 nodes), 53.0936ms from start of poll, 53.54756ms from cluster start. Timeout at 2025-07-29 00:57:00.907181217 +0000 UTC m=+100.596562561, 105.37316ms after stability 2025-07-29T00:57:00.912Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:01.017Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:00.912367897 +0000 UTC m=+100.601749241 (2 nodes), 3.76µs from start of poll, 164.1074ms from cluster start. Timeout at 2025-07-29 00:57:01.017392017 +0000 UTC m=+100.706773361, 105.02412ms after stability raft_test.go:1968: [INFO] Disconnecting Node at 725e5ca0-3831-60f9-efd1-26afcbaed2a2 [Follower] 2025-07-29T00:57:01.017Z [ERROR] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: failed to heartbeat to: peer=725e5ca0-3831-60f9-efd1-26afcbaed2a2 backoff time=10ms error="failed to connect to peer: 725e5ca0-3831-60f9-efd1-26afcbaed2a2" 2025-07-29T00:57:01.017Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: aborting pipeline replication: peer="{Voter server-d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0 d43c52fc-d7ce-d2df-3a0e-10d16d17e6c0}" 2025-07-29T00:57:01.017Z [INFO] server-7d9014e5-d4c4-7107-81c1-c9af0cc02719: aborting pipeline replication: peer="{Voter server-725e5ca0-3831-60f9-efd1-26afcbaed2a2 725e5ca0-3831-60f9-efd1-26afcbaed2a2}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) === RUN TestRaft_NotifyCh 2025-07-29T00:57:01.018Z [INFO] server-b99c8416-058c-aa92-58c0-2a48e095894d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b99c8416-058c-aa92-58c0-2a48e095894d Address:b99c8416-058c-aa92-58c0-2a48e095894d}]" 2025-07-29T00:57:01.018Z [INFO] server-b99c8416-058c-aa92-58c0-2a48e095894d: entering follower state: follower="Node at b99c8416-058c-aa92-58c0-2a48e095894d [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.088Z [WARN] server-b99c8416-058c-aa92-58c0-2a48e095894d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:01.088Z [INFO] server-b99c8416-058c-aa92-58c0-2a48e095894d: entering candidate state: node="Node at b99c8416-058c-aa92-58c0-2a48e095894d [Candidate]" term=2 2025-07-29T00:57:01.088Z [INFO] server-b99c8416-058c-aa92-58c0-2a48e095894d: election won: term=2 tally=1 2025-07-29T00:57:01.088Z [INFO] server-b99c8416-058c-aa92-58c0-2a48e095894d: entering leader state: leader="Node at b99c8416-058c-aa92-58c0-2a48e095894d [Leader]" --- PASS: TestRaft_NotifyCh (0.07s) === RUN TestRaft_AppendEntry 2025-07-29T00:57:01.089Z [INFO] server-b6bee4f7-fba4-3b92-4f35-874f8388b12f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6bee4f7-fba4-3b92-4f35-874f8388b12f Address:b6bee4f7-fba4-3b92-4f35-874f8388b12f} {Suffrage:Voter ID:server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1 Address:c7fe76a6-d5d2-4bbe-6d09-422636289fa1} {Suffrage:Voter ID:server-e40663ea-56e6-9e3a-082c-304bdf0f3175 Address:e40663ea-56e6-9e3a-082c-304bdf0f3175}]" 2025-07-29T00:57:01.089Z [INFO] server-b6bee4f7-fba4-3b92-4f35-874f8388b12f: entering follower state: follower="Node at b6bee4f7-fba4-3b92-4f35-874f8388b12f [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.089Z [INFO] server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6bee4f7-fba4-3b92-4f35-874f8388b12f Address:b6bee4f7-fba4-3b92-4f35-874f8388b12f} {Suffrage:Voter ID:server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1 Address:c7fe76a6-d5d2-4bbe-6d09-422636289fa1} {Suffrage:Voter ID:server-e40663ea-56e6-9e3a-082c-304bdf0f3175 Address:e40663ea-56e6-9e3a-082c-304bdf0f3175}]" 2025-07-29T00:57:01.089Z [INFO] server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1: entering follower state: follower="Node at c7fe76a6-d5d2-4bbe-6d09-422636289fa1 [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.089Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6bee4f7-fba4-3b92-4f35-874f8388b12f Address:b6bee4f7-fba4-3b92-4f35-874f8388b12f} {Suffrage:Voter ID:server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1 Address:c7fe76a6-d5d2-4bbe-6d09-422636289fa1} {Suffrage:Voter ID:server-e40663ea-56e6-9e3a-082c-304bdf0f3175 Address:e40663ea-56e6-9e3a-082c-304bdf0f3175}]" 2025-07-29T00:57:01.089Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:01.089Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: entering follower state: follower="Node at e40663ea-56e6-9e3a-082c-304bdf0f3175 [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.156Z [WARN] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:01.156Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: entering candidate state: node="Node at e40663ea-56e6-9e3a-082c-304bdf0f3175 [Candidate]" term=2 2025-07-29T00:57:01.156Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: election won: term=2 tally=2 2025-07-29T00:57:01.156Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: entering leader state: leader="Node at e40663ea-56e6-9e3a-082c-304bdf0f3175 [Leader]" 2025-07-29T00:57:01.156Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: added peer, starting replication: peer=server-b6bee4f7-fba4-3b92-4f35-874f8388b12f 2025-07-29T00:57:01.156Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: added peer, starting replication: peer=server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1 2025-07-29T00:57:01.157Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: pipelining replication: peer="{Voter server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1 c7fe76a6-d5d2-4bbe-6d09-422636289fa1}" 2025-07-29T00:57:01.157Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: pipelining replication: peer="{Voter server-b6bee4f7-fba4-3b92-4f35-874f8388b12f b6bee4f7-fba4-3b92-4f35-874f8388b12f}" 2025-07-29T00:57:01.268Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:01.157267057 +0000 UTC m=+100.846648401 (2 nodes), 67.51164ms from start of poll, 68.07916ms from cluster start. Timeout at 2025-07-29 00:57:01.263887297 +0000 UTC m=+100.953268601, 106.6202ms after stability 2025-07-29T00:57:01.268Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:01.379Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:01.268368617 +0000 UTC m=+100.957749921 (1 nodes), 1.72µs from start of poll, 179.18068ms from cluster start. Timeout at 2025-07-29 00:57:01.379870617 +0000 UTC m=+101.069251921, 111.502ms after stability 2025-07-29T00:57:01.379Z [WARN] server-b6bee4f7-fba4-3b92-4f35-874f8388b12f: clearing log suffix: from=1 to=2 2025-07-29T00:57:01.379Z [WARN] server-b6bee4f7-fba4-3b92-4f35-874f8388b12f: skipping application of old log: index=1 2025-07-29T00:57:01.379Z [WARN] server-b6bee4f7-fba4-3b92-4f35-874f8388b12f: skipping application of old log: index=1 2025-07-29T00:57:01.380Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: aborting pipeline replication: peer="{Voter server-c7fe76a6-d5d2-4bbe-6d09-422636289fa1 c7fe76a6-d5d2-4bbe-6d09-422636289fa1}" 2025-07-29T00:57:01.380Z [INFO] server-e40663ea-56e6-9e3a-082c-304bdf0f3175: aborting pipeline replication: peer="{Voter server-b6bee4f7-fba4-3b92-4f35-874f8388b12f b6bee4f7-fba4-3b92-4f35-874f8388b12f}" --- PASS: TestRaft_AppendEntry (0.29s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2025-07-29T00:57:01.380Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59dc3677-8dca-89f6-7d93-2b248c404ceb Address:59dc3677-8dca-89f6-7d93-2b248c404ceb} {Suffrage:Voter ID:server-0582d741-4ebb-0e0b-bb30-72b9e745afb7 Address:0582d741-4ebb-0e0b-bb30-72b9e745afb7} {Suffrage:Voter ID:server-f6abe266-7618-61a2-cc24-2ea630f700c9 Address:f6abe266-7618-61a2-cc24-2ea630f700c9}]" 2025-07-29T00:57:01.380Z [INFO] server-0582d741-4ebb-0e0b-bb30-72b9e745afb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59dc3677-8dca-89f6-7d93-2b248c404ceb Address:59dc3677-8dca-89f6-7d93-2b248c404ceb} {Suffrage:Voter ID:server-0582d741-4ebb-0e0b-bb30-72b9e745afb7 Address:0582d741-4ebb-0e0b-bb30-72b9e745afb7} {Suffrage:Voter ID:server-f6abe266-7618-61a2-cc24-2ea630f700c9 Address:f6abe266-7618-61a2-cc24-2ea630f700c9}]" 2025-07-29T00:57:01.380Z [INFO] server-f6abe266-7618-61a2-cc24-2ea630f700c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59dc3677-8dca-89f6-7d93-2b248c404ceb Address:59dc3677-8dca-89f6-7d93-2b248c404ceb} {Suffrage:Voter ID:server-0582d741-4ebb-0e0b-bb30-72b9e745afb7 Address:0582d741-4ebb-0e0b-bb30-72b9e745afb7} {Suffrage:Voter ID:server-f6abe266-7618-61a2-cc24-2ea630f700c9 Address:f6abe266-7618-61a2-cc24-2ea630f700c9}]" 2025-07-29T00:57:01.380Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:01.380Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: entering follower state: follower="Node at 59dc3677-8dca-89f6-7d93-2b248c404ceb [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.380Z [INFO] server-0582d741-4ebb-0e0b-bb30-72b9e745afb7: entering follower state: follower="Node at 0582d741-4ebb-0e0b-bb30-72b9e745afb7 [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.380Z [INFO] server-f6abe266-7618-61a2-cc24-2ea630f700c9: entering follower state: follower="Node at f6abe266-7618-61a2-cc24-2ea630f700c9 [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.471Z [WARN] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:01.471Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: entering candidate state: node="Node at 59dc3677-8dca-89f6-7d93-2b248c404ceb [Candidate]" term=2 2025-07-29T00:57:01.471Z [WARN] server-f6abe266-7618-61a2-cc24-2ea630f700c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:01.471Z [INFO] server-f6abe266-7618-61a2-cc24-2ea630f700c9: entering candidate state: node="Node at f6abe266-7618-61a2-cc24-2ea630f700c9 [Candidate]" term=2 2025-07-29T00:57:01.471Z [INFO] server-f6abe266-7618-61a2-cc24-2ea630f700c9: duplicate requestVote for same term: term=2 2025-07-29T00:57:01.472Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: duplicate requestVote for same term: term=2 2025-07-29T00:57:01.472Z [INFO] server-0582d741-4ebb-0e0b-bb30-72b9e745afb7: duplicate requestVote for same term: term=2 2025-07-29T00:57:01.472Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: election won: term=2 tally=2 2025-07-29T00:57:01.472Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: entering leader state: leader="Node at 59dc3677-8dca-89f6-7d93-2b248c404ceb [Leader]" 2025-07-29T00:57:01.472Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: added peer, starting replication: peer=server-0582d741-4ebb-0e0b-bb30-72b9e745afb7 2025-07-29T00:57:01.472Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: added peer, starting replication: peer=server-f6abe266-7618-61a2-cc24-2ea630f700c9 2025-07-29T00:57:01.472Z [INFO] server-f6abe266-7618-61a2-cc24-2ea630f700c9: entering follower state: follower="Node at f6abe266-7618-61a2-cc24-2ea630f700c9 [Follower]" leader-address=59dc3677-8dca-89f6-7d93-2b248c404ceb leader-id=server-59dc3677-8dca-89f6-7d93-2b248c404ceb 2025-07-29T00:57:01.472Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: pipelining replication: peer="{Voter server-f6abe266-7618-61a2-cc24-2ea630f700c9 f6abe266-7618-61a2-cc24-2ea630f700c9}" 2025-07-29T00:57:01.472Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: pipelining replication: peer="{Voter server-0582d741-4ebb-0e0b-bb30-72b9e745afb7 0582d741-4ebb-0e0b-bb30-72b9e745afb7}" 2025-07-29T00:57:01.579Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:01.472118257 +0000 UTC m=+101.161499561 (2 nodes), 91.51364ms from start of poll, 91.72296ms from cluster start. Timeout at 2025-07-29 00:57:01.579787697 +0000 UTC m=+101.269169001, 107.66944ms after stability 2025-07-29T00:57:01.579Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:01.687Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:01.579857297 +0000 UTC m=+101.269238601 (1 nodes), 1.76µs from start of poll, 199.462ms from cluster start. Timeout at 2025-07-29 00:57:01.687458937 +0000 UTC m=+101.376840241, 107.60164ms after stability 2025-07-29T00:57:01.687Z [WARN] server-f6abe266-7618-61a2-cc24-2ea630f700c9: rejecting vote request since we have a leader: from=0582d741-4ebb-0e0b-bb30-72b9e745afb7 leader=59dc3677-8dca-89f6-7d93-2b248c404ceb leader-id=server-59dc3677-8dca-89f6-7d93-2b248c404ceb 2025-07-29T00:57:01.687Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: aborting pipeline replication: peer="{Voter server-f6abe266-7618-61a2-cc24-2ea630f700c9 f6abe266-7618-61a2-cc24-2ea630f700c9}" 2025-07-29T00:57:01.687Z [INFO] server-59dc3677-8dca-89f6-7d93-2b248c404ceb: aborting pipeline replication: peer="{Voter server-0582d741-4ebb-0e0b-bb30-72b9e745afb7 0582d741-4ebb-0e0b-bb30-72b9e745afb7}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.31s) === RUN TestRaft_ProtocolVersion_RejectRPC 2025-07-29T00:57:01.688Z [INFO] server-d32c1345-51d8-307d-7fbe-ebd2063546e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d32c1345-51d8-307d-7fbe-ebd2063546e4 Address:d32c1345-51d8-307d-7fbe-ebd2063546e4} {Suffrage:Voter ID:server-e1eb75a8-42b0-786b-cad7-21720c4cca9d Address:e1eb75a8-42b0-786b-cad7-21720c4cca9d} {Suffrage:Voter ID:server-8300e53a-cacf-196e-6348-d2b46d54404b Address:8300e53a-cacf-196e-6348-d2b46d54404b}]" 2025-07-29T00:57:01.688Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d32c1345-51d8-307d-7fbe-ebd2063546e4 Address:d32c1345-51d8-307d-7fbe-ebd2063546e4} {Suffrage:Voter ID:server-e1eb75a8-42b0-786b-cad7-21720c4cca9d Address:e1eb75a8-42b0-786b-cad7-21720c4cca9d} {Suffrage:Voter ID:server-8300e53a-cacf-196e-6348-d2b46d54404b Address:8300e53a-cacf-196e-6348-d2b46d54404b}]" 2025-07-29T00:57:01.688Z [INFO] server-8300e53a-cacf-196e-6348-d2b46d54404b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d32c1345-51d8-307d-7fbe-ebd2063546e4 Address:d32c1345-51d8-307d-7fbe-ebd2063546e4} {Suffrage:Voter ID:server-e1eb75a8-42b0-786b-cad7-21720c4cca9d Address:e1eb75a8-42b0-786b-cad7-21720c4cca9d} {Suffrage:Voter ID:server-8300e53a-cacf-196e-6348-d2b46d54404b Address:8300e53a-cacf-196e-6348-d2b46d54404b}]" 2025-07-29T00:57:01.688Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:01.688Z [INFO] server-d32c1345-51d8-307d-7fbe-ebd2063546e4: entering follower state: follower="Node at d32c1345-51d8-307d-7fbe-ebd2063546e4 [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.688Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: entering follower state: follower="Node at e1eb75a8-42b0-786b-cad7-21720c4cca9d [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.688Z [INFO] server-8300e53a-cacf-196e-6348-d2b46d54404b: entering follower state: follower="Node at 8300e53a-cacf-196e-6348-d2b46d54404b [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.771Z [WARN] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:01.771Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: entering candidate state: node="Node at e1eb75a8-42b0-786b-cad7-21720c4cca9d [Candidate]" term=2 2025-07-29T00:57:01.771Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: election won: term=2 tally=2 2025-07-29T00:57:01.771Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: entering leader state: leader="Node at e1eb75a8-42b0-786b-cad7-21720c4cca9d [Leader]" 2025-07-29T00:57:01.771Z [WARN] server-d32c1345-51d8-307d-7fbe-ebd2063546e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:01.771Z [INFO] server-d32c1345-51d8-307d-7fbe-ebd2063546e4: entering candidate state: node="Node at d32c1345-51d8-307d-7fbe-ebd2063546e4 [Candidate]" term=2 2025-07-29T00:57:01.771Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: added peer, starting replication: peer=server-d32c1345-51d8-307d-7fbe-ebd2063546e4 2025-07-29T00:57:01.771Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: added peer, starting replication: peer=server-8300e53a-cacf-196e-6348-d2b46d54404b 2025-07-29T00:57:01.771Z [INFO] server-d32c1345-51d8-307d-7fbe-ebd2063546e4: duplicate requestVote for same term: term=2 2025-07-29T00:57:01.771Z [INFO] server-8300e53a-cacf-196e-6348-d2b46d54404b: duplicate requestVote for same term: term=2 2025-07-29T00:57:01.771Z [WARN] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: rejecting vote request since we have a leader: from=d32c1345-51d8-307d-7fbe-ebd2063546e4 leader=e1eb75a8-42b0-786b-cad7-21720c4cca9d leader-id=server-e1eb75a8-42b0-786b-cad7-21720c4cca9d 2025-07-29T00:57:01.771Z [INFO] server-d32c1345-51d8-307d-7fbe-ebd2063546e4: entering follower state: follower="Node at d32c1345-51d8-307d-7fbe-ebd2063546e4 [Follower]" leader-address=e1eb75a8-42b0-786b-cad7-21720c4cca9d leader-id=server-e1eb75a8-42b0-786b-cad7-21720c4cca9d 2025-07-29T00:57:01.771Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: pipelining replication: peer="{Voter server-8300e53a-cacf-196e-6348-d2b46d54404b 8300e53a-cacf-196e-6348-d2b46d54404b}" 2025-07-29T00:57:01.783Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: pipelining replication: peer="{Voter server-d32c1345-51d8-307d-7fbe-ebd2063546e4 d32c1345-51d8-307d-7fbe-ebd2063546e4}" 2025-07-29T00:57:01.879Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:01.771719337 +0000 UTC m=+101.461100641 (2 nodes), 83.5026ms from start of poll, 83.70168ms from cluster start. Timeout at 2025-07-29 00:57:01.879619337 +0000 UTC m=+101.569000641, 107.9ms after stability 2025-07-29T00:57:01.879Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:01.987Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:01.879709497 +0000 UTC m=+101.569090801 (1 nodes), 2.2µs from start of poll, 191.69184ms from cluster start. Timeout at 2025-07-29 00:57:01.987469017 +0000 UTC m=+101.676850281, 107.75948ms after stability 2025-07-29T00:57:01.987Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: aborting pipeline replication: peer="{Voter server-8300e53a-cacf-196e-6348-d2b46d54404b 8300e53a-cacf-196e-6348-d2b46d54404b}" 2025-07-29T00:57:01.987Z [INFO] server-e1eb75a8-42b0-786b-cad7-21720c4cca9d: aborting pipeline replication: peer="{Voter server-d32c1345-51d8-307d-7fbe-ebd2063546e4 d32c1345-51d8-307d-7fbe-ebd2063546e4}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.30s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2025-07-29T00:57:01.988Z [INFO] server-605f11a4-e44d-8f94-baa3-4887a1962884: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-605f11a4-e44d-8f94-baa3-4887a1962884 Address:605f11a4-e44d-8f94-baa3-4887a1962884} {Suffrage:Voter ID:server-ab5424e4-1e09-5365-81ad-a00cc18701df Address:ab5424e4-1e09-5365-81ad-a00cc18701df} {Suffrage:Voter ID:server-7b521558-a8f9-c00b-6db4-efe9a043de8d Address:7b521558-a8f9-c00b-6db4-efe9a043de8d}]" 2025-07-29T00:57:01.988Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-605f11a4-e44d-8f94-baa3-4887a1962884 Address:605f11a4-e44d-8f94-baa3-4887a1962884} {Suffrage:Voter ID:server-ab5424e4-1e09-5365-81ad-a00cc18701df Address:ab5424e4-1e09-5365-81ad-a00cc18701df} {Suffrage:Voter ID:server-7b521558-a8f9-c00b-6db4-efe9a043de8d Address:7b521558-a8f9-c00b-6db4-efe9a043de8d}]" 2025-07-29T00:57:01.988Z [INFO] server-7b521558-a8f9-c00b-6db4-efe9a043de8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-605f11a4-e44d-8f94-baa3-4887a1962884 Address:605f11a4-e44d-8f94-baa3-4887a1962884} {Suffrage:Voter ID:server-ab5424e4-1e09-5365-81ad-a00cc18701df Address:ab5424e4-1e09-5365-81ad-a00cc18701df} {Suffrage:Voter ID:server-7b521558-a8f9-c00b-6db4-efe9a043de8d Address:7b521558-a8f9-c00b-6db4-efe9a043de8d}]" 2025-07-29T00:57:01.988Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:01.988Z [INFO] server-605f11a4-e44d-8f94-baa3-4887a1962884: entering follower state: follower="Node at 605f11a4-e44d-8f94-baa3-4887a1962884 [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.988Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: entering follower state: follower="Node at ab5424e4-1e09-5365-81ad-a00cc18701df [Follower]" leader-address= leader-id= 2025-07-29T00:57:01.988Z [INFO] server-7b521558-a8f9-c00b-6db4-efe9a043de8d: entering follower state: follower="Node at 7b521558-a8f9-c00b-6db4-efe9a043de8d [Follower]" leader-address= leader-id= 2025-07-29T00:57:02.071Z [WARN] server-ab5424e4-1e09-5365-81ad-a00cc18701df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:02.071Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: entering candidate state: node="Node at ab5424e4-1e09-5365-81ad-a00cc18701df [Candidate]" term=2 2025-07-29T00:57:02.071Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: election won: term=2 tally=2 2025-07-29T00:57:02.071Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: entering leader state: leader="Node at ab5424e4-1e09-5365-81ad-a00cc18701df [Leader]" 2025-07-29T00:57:02.071Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: added peer, starting replication: peer=server-605f11a4-e44d-8f94-baa3-4887a1962884 2025-07-29T00:57:02.071Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: added peer, starting replication: peer=server-7b521558-a8f9-c00b-6db4-efe9a043de8d 2025-07-29T00:57:02.071Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: pipelining replication: peer="{Voter server-7b521558-a8f9-c00b-6db4-efe9a043de8d 7b521558-a8f9-c00b-6db4-efe9a043de8d}" 2025-07-29T00:57:02.071Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: pipelining replication: peer="{Voter server-605f11a4-e44d-8f94-baa3-4887a1962884 605f11a4-e44d-8f94-baa3-4887a1962884}" 2025-07-29T00:57:02.179Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:02.071816737 +0000 UTC m=+101.761198041 (1 nodes), 83.41696ms from start of poll, 83.61764ms from cluster start. Timeout at 2025-07-29 00:57:02.179835457 +0000 UTC m=+101.869216721, 108.01868ms after stability 2025-07-29T00:57:02.179Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:02.287Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:02.179916857 +0000 UTC m=+101.869298161 (1 nodes), 1.16µs from start of poll, 191.71776ms from cluster start. Timeout at 2025-07-29 00:57:02.287498497 +0000 UTC m=+101.976879801, 107.58164ms after stability 2025-07-29T00:57:02.287Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:02.395Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:02.287703297 +0000 UTC m=+101.977084601 (1 nodes), 560ns from start of poll, 299.5042ms from cluster start. Timeout at 2025-07-29 00:57:02.395688737 +0000 UTC m=+102.085070001, 107.9854ms after stability 2025-07-29T00:57:02.395Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:02.503Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:02.395905177 +0000 UTC m=+102.085286441 (1 nodes), 960ns from start of poll, 407.70604ms from cluster start. Timeout at 2025-07-29 00:57:02.503620137 +0000 UTC m=+102.193001441, 107.715ms after stability 2025-07-29T00:57:02.503Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:02.623Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:02.503777417 +0000 UTC m=+102.193158721 (1 nodes), 920ns from start of poll, 515.57832ms from cluster start. Timeout at 2025-07-29 00:57:02.623444777 +0000 UTC m=+102.312826081, 119.66736ms after stability 2025-07-29T00:57:02.640Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:02.747Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:02.640105057 +0000 UTC m=+102.329486321 (1 nodes), 600ns from start of poll, 651.90592ms from cluster start. Timeout at 2025-07-29 00:57:02.747454457 +0000 UTC m=+102.436835761, 107.34944ms after stability 2025-07-29T00:57:02.855Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:02.963Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:02.855585657 +0000 UTC m=+102.544966961 (1 nodes), 1µs from start of poll, 867.38656ms from cluster start. Timeout at 2025-07-29 00:57:02.963453057 +0000 UTC m=+102.652834361, 107.8674ms after stability 2025-07-29T00:57:02.963Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: aborting pipeline replication: peer="{Voter server-7b521558-a8f9-c00b-6db4-efe9a043de8d 7b521558-a8f9-c00b-6db4-efe9a043de8d}" 2025-07-29T00:57:02.963Z [INFO] server-ab5424e4-1e09-5365-81ad-a00cc18701df: aborting pipeline replication: peer="{Voter server-605f11a4-e44d-8f94-baa3-4887a1962884 605f11a4-e44d-8f94-baa3-4887a1962884}" 2025-07-29T00:57:03.463Z [ERROR] server-ab5424e4-1e09-5365-81ad-a00cc18701df: failed to heartbeat to: peer=605f11a4-e44d-8f94-baa3-4887a1962884 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeaderID_Propagated (1.48s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2025-07-29T00:57:03.464Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f Address:e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f} {Suffrage:Voter ID:server-73b33f9d-4b8c-4719-8e45-b789ed141855 Address:73b33f9d-4b8c-4719-8e45-b789ed141855} {Suffrage:Voter ID:server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 Address:f10edfd9-d3f7-d201-c5df-4f3559d5f7a0}]" 2025-07-29T00:57:03.464Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f Address:e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f} {Suffrage:Voter ID:server-73b33f9d-4b8c-4719-8e45-b789ed141855 Address:73b33f9d-4b8c-4719-8e45-b789ed141855} {Suffrage:Voter ID:server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 Address:f10edfd9-d3f7-d201-c5df-4f3559d5f7a0}]" 2025-07-29T00:57:03.464Z [INFO] server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f Address:e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f} {Suffrage:Voter ID:server-73b33f9d-4b8c-4719-8e45-b789ed141855 Address:73b33f9d-4b8c-4719-8e45-b789ed141855} {Suffrage:Voter ID:server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 Address:f10edfd9-d3f7-d201-c5df-4f3559d5f7a0}]" 2025-07-29T00:57:03.464Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:03.464Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: entering follower state: follower="Node at e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f [Follower]" leader-address= leader-id= 2025-07-29T00:57:03.464Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: entering follower state: follower="Node at 73b33f9d-4b8c-4719-8e45-b789ed141855 [Follower]" leader-address= leader-id= 2025-07-29T00:57:03.464Z [INFO] server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0: entering follower state: follower="Node at f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 [Follower]" leader-address= leader-id= 2025-07-29T00:57:03.515Z [WARN] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:03.515Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: entering candidate state: node="Node at e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f [Candidate]" term=2 2025-07-29T00:57:03.515Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: election won: term=2 tally=2 2025-07-29T00:57:03.515Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: entering leader state: leader="Node at e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f [Leader]" 2025-07-29T00:57:03.515Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: added peer, starting replication: peer=server-73b33f9d-4b8c-4719-8e45-b789ed141855 2025-07-29T00:57:03.515Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: added peer, starting replication: peer=server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 2025-07-29T00:57:03.515Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: pipelining replication: peer="{Voter server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 f10edfd9-d3f7-d201-c5df-4f3559d5f7a0}" 2025-07-29T00:57:03.515Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: pipelining replication: peer="{Voter server-73b33f9d-4b8c-4719-8e45-b789ed141855 73b33f9d-4b8c-4719-8e45-b789ed141855}" 2025-07-29T00:57:03.628Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:03.515994177 +0000 UTC m=+103.205375481 (1 nodes), 51.67068ms from start of poll, 51.88404ms from cluster start. Timeout at 2025-07-29 00:57:03.628250857 +0000 UTC m=+103.317632161, 112.25668ms after stability 2025-07-29T00:57:03.628Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:03.735Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:03.628348457 +0000 UTC m=+103.317729721 (1 nodes), 1.48µs from start of poll, 164.23828ms from cluster start. Timeout at 2025-07-29 00:57:03.735875057 +0000 UTC m=+103.425256361, 107.52664ms after stability 2025-07-29T00:57:03.736Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: entering candidate state: node="Node at 73b33f9d-4b8c-4719-8e45-b789ed141855 [Candidate]" term=3 2025-07-29T00:57:03.736Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: election won: term=3 tally=2 2025-07-29T00:57:03.736Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: entering leader state: leader="Node at 73b33f9d-4b8c-4719-8e45-b789ed141855 [Leader]" 2025-07-29T00:57:03.736Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: added peer, starting replication: peer=server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f 2025-07-29T00:57:03.736Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: added peer, starting replication: peer=server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 2025-07-29T00:57:03.736Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: pipelining replication: peer="{Voter server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 f10edfd9-d3f7-d201-c5df-4f3559d5f7a0}" 2025-07-29T00:57:03.736Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:03.736Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: entering follower state: follower="Node at e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f [Follower]" leader-address= leader-id= 2025-07-29T00:57:03.736Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: aborting pipeline replication: peer="{Voter server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 f10edfd9-d3f7-d201-c5df-4f3559d5f7a0}" 2025-07-29T00:57:03.736Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: pipelining replication: peer="{Voter server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f}" 2025-07-29T00:57:03.736Z [INFO] server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f: aborting pipeline replication: peer="{Voter server-73b33f9d-4b8c-4719-8e45-b789ed141855 73b33f9d-4b8c-4719-8e45-b789ed141855}" 2025-07-29T00:57:03.843Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:03.736578617 +0000 UTC m=+103.425959921 (1 nodes), 112.16µs from start of poll, 272.46848ms from cluster start. Timeout at 2025-07-29 00:57:03.843708857 +0000 UTC m=+103.533090161, 107.13024ms after stability 2025-07-29T00:57:03.843Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: aborting pipeline replication: peer="{Voter server-f10edfd9-d3f7-d201-c5df-4f3559d5f7a0 f10edfd9-d3f7-d201-c5df-4f3559d5f7a0}" 2025-07-29T00:57:03.843Z [INFO] server-73b33f9d-4b8c-4719-8e45-b789ed141855: aborting pipeline replication: peer="{Voter server-e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f e9e9df7f-eee5-5ba8-ac2c-b9440c0c760f}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode 2025-07-29T00:57:03.847Z [INFO] server-6b3b358d-0048-4db4-1839-8dfd4f799e60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b3b358d-0048-4db4-1839-8dfd4f799e60 Address:6b3b358d-0048-4db4-1839-8dfd4f799e60}]" 2025-07-29T00:57:03.847Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:03.847Z [INFO] server-6b3b358d-0048-4db4-1839-8dfd4f799e60: entering follower state: follower="Node at 6b3b358d-0048-4db4-1839-8dfd4f799e60 [Follower]" leader-address= leader-id= 2025-07-29T00:57:03.904Z [WARN] server-6b3b358d-0048-4db4-1839-8dfd4f799e60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:03.904Z [INFO] server-6b3b358d-0048-4db4-1839-8dfd4f799e60: entering candidate state: node="Node at 6b3b358d-0048-4db4-1839-8dfd4f799e60 [Candidate]" term=2 2025-07-29T00:57:03.904Z [INFO] server-6b3b358d-0048-4db4-1839-8dfd4f799e60: election won: term=2 tally=1 2025-07-29T00:57:03.904Z [INFO] server-6b3b358d-0048-4db4-1839-8dfd4f799e60: entering leader state: leader="Node at 6b3b358d-0048-4db4-1839-8dfd4f799e60 [Leader]" 2025-07-29T00:57:04.011Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:03.904927537 +0000 UTC m=+103.594308801 (1 nodes), 57.02764ms from start of poll, 57.19532ms from cluster start. Timeout at 2025-07-29 00:57:04.011727977 +0000 UTC m=+103.701109241, 106.80044ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.16s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2025-07-29T00:57:04.012Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 Address:1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2} {Suffrage:Voter ID:server-9c98114a-e0bc-5e02-f853-d48415f36b2c Address:9c98114a-e0bc-5e02-f853-d48415f36b2c} {Suffrage:Voter ID:server-20498e01-6027-2369-b7e4-f917cadf2c96 Address:20498e01-6027-2369-b7e4-f917cadf2c96} {Suffrage:Voter ID:server-aeff0cf7-cc86-71b9-9838-fe41166eff9c Address:aeff0cf7-cc86-71b9-9838-fe41166eff9c} {Suffrage:Voter ID:server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 Address:6d1379b6-3c41-8fdd-3905-47b9f34c28d7} {Suffrage:Voter ID:server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 Address:fd77a36a-040a-acbd-f46a-21cf16fd2f20} {Suffrage:Voter ID:server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b Address:e94b58b0-69f8-ee8a-ab71-b9190a902a5b}]" 2025-07-29T00:57:04.012Z [INFO] server-9c98114a-e0bc-5e02-f853-d48415f36b2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 Address:1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2} {Suffrage:Voter ID:server-9c98114a-e0bc-5e02-f853-d48415f36b2c Address:9c98114a-e0bc-5e02-f853-d48415f36b2c} {Suffrage:Voter ID:server-20498e01-6027-2369-b7e4-f917cadf2c96 Address:20498e01-6027-2369-b7e4-f917cadf2c96} {Suffrage:Voter ID:server-aeff0cf7-cc86-71b9-9838-fe41166eff9c Address:aeff0cf7-cc86-71b9-9838-fe41166eff9c} {Suffrage:Voter ID:server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 Address:6d1379b6-3c41-8fdd-3905-47b9f34c28d7} {Suffrage:Voter ID:server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 Address:fd77a36a-040a-acbd-f46a-21cf16fd2f20} {Suffrage:Voter ID:server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b Address:e94b58b0-69f8-ee8a-ab71-b9190a902a5b}]" 2025-07-29T00:57:04.012Z [INFO] server-20498e01-6027-2369-b7e4-f917cadf2c96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 Address:1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2} {Suffrage:Voter ID:server-9c98114a-e0bc-5e02-f853-d48415f36b2c Address:9c98114a-e0bc-5e02-f853-d48415f36b2c} {Suffrage:Voter ID:server-20498e01-6027-2369-b7e4-f917cadf2c96 Address:20498e01-6027-2369-b7e4-f917cadf2c96} {Suffrage:Voter ID:server-aeff0cf7-cc86-71b9-9838-fe41166eff9c Address:aeff0cf7-cc86-71b9-9838-fe41166eff9c} {Suffrage:Voter ID:server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 Address:6d1379b6-3c41-8fdd-3905-47b9f34c28d7} {Suffrage:Voter ID:server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 Address:fd77a36a-040a-acbd-f46a-21cf16fd2f20} {Suffrage:Voter ID:server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b Address:e94b58b0-69f8-ee8a-ab71-b9190a902a5b}]" 2025-07-29T00:57:04.012Z [INFO] server-aeff0cf7-cc86-71b9-9838-fe41166eff9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 Address:1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2} {Suffrage:Voter ID:server-9c98114a-e0bc-5e02-f853-d48415f36b2c Address:9c98114a-e0bc-5e02-f853-d48415f36b2c} {Suffrage:Voter ID:server-20498e01-6027-2369-b7e4-f917cadf2c96 Address:20498e01-6027-2369-b7e4-f917cadf2c96} {Suffrage:Voter ID:server-aeff0cf7-cc86-71b9-9838-fe41166eff9c Address:aeff0cf7-cc86-71b9-9838-fe41166eff9c} {Suffrage:Voter ID:server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 Address:6d1379b6-3c41-8fdd-3905-47b9f34c28d7} {Suffrage:Voter ID:server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 Address:fd77a36a-040a-acbd-f46a-21cf16fd2f20} {Suffrage:Voter ID:server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b Address:e94b58b0-69f8-ee8a-ab71-b9190a902a5b}]" 2025-07-29T00:57:04.012Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 Address:1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2} {Suffrage:Voter ID:server-9c98114a-e0bc-5e02-f853-d48415f36b2c Address:9c98114a-e0bc-5e02-f853-d48415f36b2c} {Suffrage:Voter ID:server-20498e01-6027-2369-b7e4-f917cadf2c96 Address:20498e01-6027-2369-b7e4-f917cadf2c96} {Suffrage:Voter ID:server-aeff0cf7-cc86-71b9-9838-fe41166eff9c Address:aeff0cf7-cc86-71b9-9838-fe41166eff9c} {Suffrage:Voter ID:server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 Address:6d1379b6-3c41-8fdd-3905-47b9f34c28d7} {Suffrage:Voter ID:server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 Address:fd77a36a-040a-acbd-f46a-21cf16fd2f20} {Suffrage:Voter ID:server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b Address:e94b58b0-69f8-ee8a-ab71-b9190a902a5b}]" 2025-07-29T00:57:04.012Z [INFO] server-fd77a36a-040a-acbd-f46a-21cf16fd2f20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 Address:1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2} {Suffrage:Voter ID:server-9c98114a-e0bc-5e02-f853-d48415f36b2c Address:9c98114a-e0bc-5e02-f853-d48415f36b2c} {Suffrage:Voter ID:server-20498e01-6027-2369-b7e4-f917cadf2c96 Address:20498e01-6027-2369-b7e4-f917cadf2c96} {Suffrage:Voter ID:server-aeff0cf7-cc86-71b9-9838-fe41166eff9c Address:aeff0cf7-cc86-71b9-9838-fe41166eff9c} {Suffrage:Voter ID:server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 Address:6d1379b6-3c41-8fdd-3905-47b9f34c28d7} {Suffrage:Voter ID:server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 Address:fd77a36a-040a-acbd-f46a-21cf16fd2f20} {Suffrage:Voter ID:server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b Address:e94b58b0-69f8-ee8a-ab71-b9190a902a5b}]" 2025-07-29T00:57:04.012Z [INFO] server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 Address:1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2} {Suffrage:Voter ID:server-9c98114a-e0bc-5e02-f853-d48415f36b2c Address:9c98114a-e0bc-5e02-f853-d48415f36b2c} {Suffrage:Voter ID:server-20498e01-6027-2369-b7e4-f917cadf2c96 Address:20498e01-6027-2369-b7e4-f917cadf2c96} {Suffrage:Voter ID:server-aeff0cf7-cc86-71b9-9838-fe41166eff9c Address:aeff0cf7-cc86-71b9-9838-fe41166eff9c} {Suffrage:Voter ID:server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 Address:6d1379b6-3c41-8fdd-3905-47b9f34c28d7} {Suffrage:Voter ID:server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 Address:fd77a36a-040a-acbd-f46a-21cf16fd2f20} {Suffrage:Voter ID:server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b Address:e94b58b0-69f8-ee8a-ab71-b9190a902a5b}]" 2025-07-29T00:57:04.012Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:04.012Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: entering follower state: follower="Node at 1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.013Z [INFO] server-9c98114a-e0bc-5e02-f853-d48415f36b2c: entering follower state: follower="Node at 9c98114a-e0bc-5e02-f853-d48415f36b2c [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.013Z [INFO] server-20498e01-6027-2369-b7e4-f917cadf2c96: entering follower state: follower="Node at 20498e01-6027-2369-b7e4-f917cadf2c96 [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.013Z [INFO] server-aeff0cf7-cc86-71b9-9838-fe41166eff9c: entering follower state: follower="Node at aeff0cf7-cc86-71b9-9838-fe41166eff9c [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.013Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: entering follower state: follower="Node at 6d1379b6-3c41-8fdd-3905-47b9f34c28d7 [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.013Z [INFO] server-fd77a36a-040a-acbd-f46a-21cf16fd2f20: entering follower state: follower="Node at fd77a36a-040a-acbd-f46a-21cf16fd2f20 [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.013Z [INFO] server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b: entering follower state: follower="Node at e94b58b0-69f8-ee8a-ab71-b9190a902a5b [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.067Z [WARN] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:04.067Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: entering candidate state: node="Node at 6d1379b6-3c41-8fdd-3905-47b9f34c28d7 [Candidate]" term=2 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: election won: term=2 tally=4 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: entering leader state: leader="Node at 6d1379b6-3c41-8fdd-3905-47b9f34c28d7 [Leader]" 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: added peer, starting replication: peer=server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: added peer, starting replication: peer=server-9c98114a-e0bc-5e02-f853-d48415f36b2c 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: added peer, starting replication: peer=server-20498e01-6027-2369-b7e4-f917cadf2c96 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: added peer, starting replication: peer=server-aeff0cf7-cc86-71b9-9838-fe41166eff9c 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: added peer, starting replication: peer=server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: added peer, starting replication: peer=server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: pipelining replication: peer="{Voter server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b e94b58b0-69f8-ee8a-ab71-b9190a902a5b}" 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: pipelining replication: peer="{Voter server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2}" 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: pipelining replication: peer="{Voter server-9c98114a-e0bc-5e02-f853-d48415f36b2c 9c98114a-e0bc-5e02-f853-d48415f36b2c}" 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: pipelining replication: peer="{Voter server-20498e01-6027-2369-b7e4-f917cadf2c96 20498e01-6027-2369-b7e4-f917cadf2c96}" 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: pipelining replication: peer="{Voter server-aeff0cf7-cc86-71b9-9838-fe41166eff9c aeff0cf7-cc86-71b9-9838-fe41166eff9c}" 2025-07-29T00:57:04.068Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: pipelining replication: peer="{Voter server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 fd77a36a-040a-acbd-f46a-21cf16fd2f20}" 2025-07-29T00:57:04.179Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:04.068405897 +0000 UTC m=+103.757787201 (1 nodes), 55.54652ms from start of poll, 56.02664ms from cluster start. Timeout at 2025-07-29 00:57:04.179795097 +0000 UTC m=+103.869176361, 111.38916ms after stability 2025-07-29T00:57:04.179Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:04.292Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:04.179878017 +0000 UTC m=+103.869259321 (6 nodes), 2.52µs from start of poll, 167.49876ms from cluster start. Timeout at 2025-07-29 00:57:04.292664777 +0000 UTC m=+103.982046081, 112.78676ms after stability 2025-07-29T00:57:04.292Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:04.399Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:04.292728377 +0000 UTC m=+103.982109641 (1 nodes), 1.4µs from start of poll, 280.34908ms from cluster start. Timeout at 2025-07-29 00:57:04.399457057 +0000 UTC m=+104.088838361, 106.72872ms after stability 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: entering candidate state: node="Node at 1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 [Candidate]" term=3 2025-07-29T00:57:04.399Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: election won: term=3 tally=4 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: entering leader state: leader="Node at 1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 [Leader]" 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: added peer, starting replication: peer=server-9c98114a-e0bc-5e02-f853-d48415f36b2c 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: added peer, starting replication: peer=server-20498e01-6027-2369-b7e4-f917cadf2c96 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: added peer, starting replication: peer=server-aeff0cf7-cc86-71b9-9838-fe41166eff9c 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: added peer, starting replication: peer=server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: added peer, starting replication: peer=server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 2025-07-29T00:57:04.399Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: added peer, starting replication: peer=server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b 2025-07-29T00:57:04.400Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: pipelining replication: peer="{Voter server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b e94b58b0-69f8-ee8a-ab71-b9190a902a5b}" 2025-07-29T00:57:04.400Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: pipelining replication: peer="{Voter server-9c98114a-e0bc-5e02-f853-d48415f36b2c 9c98114a-e0bc-5e02-f853-d48415f36b2c}" 2025-07-29T00:57:04.400Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: pipelining replication: peer="{Voter server-20498e01-6027-2369-b7e4-f917cadf2c96 20498e01-6027-2369-b7e4-f917cadf2c96}" 2025-07-29T00:57:04.400Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: pipelining replication: peer="{Voter server-aeff0cf7-cc86-71b9-9838-fe41166eff9c aeff0cf7-cc86-71b9-9838-fe41166eff9c}" 2025-07-29T00:57:04.400Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: entering follower state: follower="Node at 6d1379b6-3c41-8fdd-3905-47b9f34c28d7 [Follower]" leader-address=1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 leader-id=server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 2025-07-29T00:57:04.400Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: aborting pipeline replication: peer="{Voter server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b e94b58b0-69f8-ee8a-ab71-b9190a902a5b}" 2025-07-29T00:57:04.400Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: pipelining replication: peer="{Voter server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 fd77a36a-040a-acbd-f46a-21cf16fd2f20}" 2025-07-29T00:57:04.400Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: pipelining replication: peer="{Voter server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 6d1379b6-3c41-8fdd-3905-47b9f34c28d7}" 2025-07-29T00:57:04.400Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: aborting pipeline replication: peer="{Voter server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2}" 2025-07-29T00:57:04.400Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: aborting pipeline replication: peer="{Voter server-9c98114a-e0bc-5e02-f853-d48415f36b2c 9c98114a-e0bc-5e02-f853-d48415f36b2c}" 2025-07-29T00:57:04.400Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: aborting pipeline replication: peer="{Voter server-20498e01-6027-2369-b7e4-f917cadf2c96 20498e01-6027-2369-b7e4-f917cadf2c96}" 2025-07-29T00:57:04.400Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: aborting pipeline replication: peer="{Voter server-aeff0cf7-cc86-71b9-9838-fe41166eff9c aeff0cf7-cc86-71b9-9838-fe41166eff9c}" 2025-07-29T00:57:04.400Z [INFO] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: aborting pipeline replication: peer="{Voter server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 fd77a36a-040a-acbd-f46a-21cf16fd2f20}" 2025-07-29T00:57:04.400Z [WARN] server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7: rejecting vote request since our last term is greater: candidate=1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 last-term=3 last-candidate-term=2 2025-07-29T00:57:04.400Z [WARN] server-fd77a36a-040a-acbd-f46a-21cf16fd2f20: rejecting vote request since our last term is greater: candidate=1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2 last-term=3 last-candidate-term=2 2025-07-29T00:57:04.532Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:04.400408977 +0000 UTC m=+104.089790281 (1 nodes), 568.04µs from start of poll, 388.02972ms from cluster start. Timeout at 2025-07-29 00:57:04.532034737 +0000 UTC m=+104.221416041, 131.62576ms after stability 2025-07-29T00:57:04.532Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: aborting pipeline replication: peer="{Voter server-fd77a36a-040a-acbd-f46a-21cf16fd2f20 fd77a36a-040a-acbd-f46a-21cf16fd2f20}" 2025-07-29T00:57:04.532Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: aborting pipeline replication: peer="{Voter server-e94b58b0-69f8-ee8a-ab71-b9190a902a5b e94b58b0-69f8-ee8a-ab71-b9190a902a5b}" 2025-07-29T00:57:04.532Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: aborting pipeline replication: peer="{Voter server-9c98114a-e0bc-5e02-f853-d48415f36b2c 9c98114a-e0bc-5e02-f853-d48415f36b2c}" 2025-07-29T00:57:04.532Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: aborting pipeline replication: peer="{Voter server-20498e01-6027-2369-b7e4-f917cadf2c96 20498e01-6027-2369-b7e4-f917cadf2c96}" 2025-07-29T00:57:04.532Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: aborting pipeline replication: peer="{Voter server-aeff0cf7-cc86-71b9-9838-fe41166eff9c aeff0cf7-cc86-71b9-9838-fe41166eff9c}" 2025-07-29T00:57:04.532Z [INFO] server-1d162dcd-eab3-d1e9-05cb-5ac7cc6b9dd2: aborting pipeline replication: peer="{Voter server-6d1379b6-3c41-8fdd-3905-47b9f34c28d7 6d1379b6-3c41-8fdd-3905-47b9f34c28d7}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.52s) === RUN TestRaft_LeadershipTransferToInvalidID 2025-07-29T00:57:04.532Z [INFO] server-725d5296-9613-3806-cb65-cb30dae6079e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725d5296-9613-3806-cb65-cb30dae6079e Address:725d5296-9613-3806-cb65-cb30dae6079e} {Suffrage:Voter ID:server-de543213-8213-883d-7136-d7e5217b92aa Address:de543213-8213-883d-7136-d7e5217b92aa} {Suffrage:Voter ID:server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 Address:2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46}]" 2025-07-29T00:57:04.532Z [INFO] server-de543213-8213-883d-7136-d7e5217b92aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725d5296-9613-3806-cb65-cb30dae6079e Address:725d5296-9613-3806-cb65-cb30dae6079e} {Suffrage:Voter ID:server-de543213-8213-883d-7136-d7e5217b92aa Address:de543213-8213-883d-7136-d7e5217b92aa} {Suffrage:Voter ID:server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 Address:2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46}]" 2025-07-29T00:57:04.532Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725d5296-9613-3806-cb65-cb30dae6079e Address:725d5296-9613-3806-cb65-cb30dae6079e} {Suffrage:Voter ID:server-de543213-8213-883d-7136-d7e5217b92aa Address:de543213-8213-883d-7136-d7e5217b92aa} {Suffrage:Voter ID:server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 Address:2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46}]" 2025-07-29T00:57:04.532Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:04.532Z [INFO] server-725d5296-9613-3806-cb65-cb30dae6079e: entering follower state: follower="Node at 725d5296-9613-3806-cb65-cb30dae6079e [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.532Z [INFO] server-de543213-8213-883d-7136-d7e5217b92aa: entering follower state: follower="Node at de543213-8213-883d-7136-d7e5217b92aa [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.532Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: entering follower state: follower="Node at 2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.603Z [WARN] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:04.603Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: entering candidate state: node="Node at 2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 [Candidate]" term=2 2025-07-29T00:57:04.603Z [WARN] server-725d5296-9613-3806-cb65-cb30dae6079e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:04.603Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: election won: term=2 tally=2 2025-07-29T00:57:04.603Z [INFO] server-725d5296-9613-3806-cb65-cb30dae6079e: entering candidate state: node="Node at 725d5296-9613-3806-cb65-cb30dae6079e [Candidate]" term=2 2025-07-29T00:57:04.603Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: entering leader state: leader="Node at 2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 [Leader]" 2025-07-29T00:57:04.603Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: added peer, starting replication: peer=server-725d5296-9613-3806-cb65-cb30dae6079e 2025-07-29T00:57:04.603Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: added peer, starting replication: peer=server-de543213-8213-883d-7136-d7e5217b92aa 2025-07-29T00:57:04.603Z [INFO] server-de543213-8213-883d-7136-d7e5217b92aa: duplicate requestVote for same term: term=2 2025-07-29T00:57:04.603Z [WARN] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: rejecting vote request since we have a leader: from=725d5296-9613-3806-cb65-cb30dae6079e leader=2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 leader-id=server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 2025-07-29T00:57:04.603Z [INFO] server-725d5296-9613-3806-cb65-cb30dae6079e: entering follower state: follower="Node at 725d5296-9613-3806-cb65-cb30dae6079e [Follower]" leader-address=2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 leader-id=server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46 2025-07-29T00:57:04.603Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: pipelining replication: peer="{Voter server-725d5296-9613-3806-cb65-cb30dae6079e 725d5296-9613-3806-cb65-cb30dae6079e}" 2025-07-29T00:57:04.603Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: pipelining replication: peer="{Voter server-de543213-8213-883d-7136-d7e5217b92aa de543213-8213-883d-7136-d7e5217b92aa}" 2025-07-29T00:57:04.607Z [INFO] server-725d5296-9613-3806-cb65-cb30dae6079e: duplicate requestVote for same term: term=2 2025-07-29T00:57:04.715Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:04.607722177 +0000 UTC m=+104.297103481 (1 nodes), 74.86336ms from start of poll, 75.0618ms from cluster start. Timeout at 2025-07-29 00:57:04.715770017 +0000 UTC m=+104.405151321, 108.04784ms after stability 2025-07-29T00:57:04.715Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: aborting pipeline replication: peer="{Voter server-de543213-8213-883d-7136-d7e5217b92aa de543213-8213-883d-7136-d7e5217b92aa}" 2025-07-29T00:57:04.715Z [INFO] server-2c4d3d9f-6b05-4ccc-12fa-adb4b104fe46: aborting pipeline replication: peer="{Voter server-725d5296-9613-3806-cb65-cb30dae6079e 725d5296-9613-3806-cb65-cb30dae6079e}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2025-07-29T00:57:04.716Z [INFO] server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8 Address:9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8} {Suffrage:Voter ID:server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3 Address:96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3} {Suffrage:Voter ID:server-8b8a3882-4b05-d745-4e2d-ea5189efec9c Address:8b8a3882-4b05-d745-4e2d-ea5189efec9c}]" 2025-07-29T00:57:04.716Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8 Address:9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8} {Suffrage:Voter ID:server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3 Address:96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3} {Suffrage:Voter ID:server-8b8a3882-4b05-d745-4e2d-ea5189efec9c Address:8b8a3882-4b05-d745-4e2d-ea5189efec9c}]" 2025-07-29T00:57:04.716Z [INFO] server-8b8a3882-4b05-d745-4e2d-ea5189efec9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8 Address:9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8} {Suffrage:Voter ID:server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3 Address:96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3} {Suffrage:Voter ID:server-8b8a3882-4b05-d745-4e2d-ea5189efec9c Address:8b8a3882-4b05-d745-4e2d-ea5189efec9c}]" 2025-07-29T00:57:04.716Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:04.716Z [INFO] server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8: entering follower state: follower="Node at 9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8 [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.716Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: entering follower state: follower="Node at 96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3 [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.716Z [INFO] server-8b8a3882-4b05-d745-4e2d-ea5189efec9c: entering follower state: follower="Node at 8b8a3882-4b05-d745-4e2d-ea5189efec9c [Follower]" leader-address= leader-id= 2025-07-29T00:57:04.795Z [WARN] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:04.795Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: entering candidate state: node="Node at 96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3 [Candidate]" term=2 2025-07-29T00:57:04.795Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: election won: term=2 tally=2 2025-07-29T00:57:04.795Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: entering leader state: leader="Node at 96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3 [Leader]" 2025-07-29T00:57:04.795Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: added peer, starting replication: peer=server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8 2025-07-29T00:57:04.795Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: added peer, starting replication: peer=server-8b8a3882-4b05-d745-4e2d-ea5189efec9c 2025-07-29T00:57:04.795Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: pipelining replication: peer="{Voter server-8b8a3882-4b05-d745-4e2d-ea5189efec9c 8b8a3882-4b05-d745-4e2d-ea5189efec9c}" 2025-07-29T00:57:04.795Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: pipelining replication: peer="{Voter server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8 9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8}" 2025-07-29T00:57:04.903Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:04.796003577 +0000 UTC m=+104.485384841 (2 nodes), 79.54576ms from start of poll, 79.74576ms from cluster start. Timeout at 2025-07-29 00:57:04.903578577 +0000 UTC m=+104.592959881, 107.57504ms after stability 2025-07-29T00:57:04.903Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:05.011Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:04.903673137 +0000 UTC m=+104.593054401 (1 nodes), 920ns from start of poll, 187.41532ms from cluster start. Timeout at 2025-07-29 00:57:05.011705257 +0000 UTC m=+104.701086561, 108.03216ms after stability 2025-07-29T00:57:05.011Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: aborting pipeline replication: peer="{Voter server-8b8a3882-4b05-d745-4e2d-ea5189efec9c 8b8a3882-4b05-d745-4e2d-ea5189efec9c}" 2025-07-29T00:57:05.011Z [INFO] server-96af8bc1-a6ca-2bde-912e-ddc2c8b31cd3: aborting pipeline replication: peer="{Voter server-9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8 9bfb2dfa-cf66-1eda-93c6-b7ee20d875e8}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.30s) === RUN TestRaft_LeadershipTransferToBehindServer 2025-07-29T00:57:05.012Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d Address:74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d} {Suffrage:Voter ID:server-db21b492-43c9-0abd-0590-ffb83f366880 Address:db21b492-43c9-0abd-0590-ffb83f366880} {Suffrage:Voter ID:server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8 Address:1d7c5a64-df8b-e686-618e-fdcb9fb524c8}]" 2025-07-29T00:57:05.012Z [INFO] server-db21b492-43c9-0abd-0590-ffb83f366880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d Address:74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d} {Suffrage:Voter ID:server-db21b492-43c9-0abd-0590-ffb83f366880 Address:db21b492-43c9-0abd-0590-ffb83f366880} {Suffrage:Voter ID:server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8 Address:1d7c5a64-df8b-e686-618e-fdcb9fb524c8}]" 2025-07-29T00:57:05.012Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d Address:74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d} {Suffrage:Voter ID:server-db21b492-43c9-0abd-0590-ffb83f366880 Address:db21b492-43c9-0abd-0590-ffb83f366880} {Suffrage:Voter ID:server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8 Address:1d7c5a64-df8b-e686-618e-fdcb9fb524c8}]" 2025-07-29T00:57:05.012Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:05.012Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: entering follower state: follower="Node at 74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.012Z [INFO] server-db21b492-43c9-0abd-0590-ffb83f366880: entering follower state: follower="Node at db21b492-43c9-0abd-0590-ffb83f366880 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.012Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: entering follower state: follower="Node at 1d7c5a64-df8b-e686-618e-fdcb9fb524c8 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.091Z [WARN] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:05.092Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: entering candidate state: node="Node at 1d7c5a64-df8b-e686-618e-fdcb9fb524c8 [Candidate]" term=2 2025-07-29T00:57:05.092Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: election won: term=2 tally=2 2025-07-29T00:57:05.092Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: entering leader state: leader="Node at 1d7c5a64-df8b-e686-618e-fdcb9fb524c8 [Leader]" 2025-07-29T00:57:05.092Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: added peer, starting replication: peer=server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d 2025-07-29T00:57:05.092Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: added peer, starting replication: peer=server-db21b492-43c9-0abd-0590-ffb83f366880 2025-07-29T00:57:05.092Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: pipelining replication: peer="{Voter server-db21b492-43c9-0abd-0590-ffb83f366880 db21b492-43c9-0abd-0590-ffb83f366880}" 2025-07-29T00:57:05.092Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: pipelining replication: peer="{Voter server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d 74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d}" 2025-07-29T00:57:05.199Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:05.092278617 +0000 UTC m=+104.781659921 (1 nodes), 79.8892ms from start of poll, 80.07616ms from cluster start. Timeout at 2025-07-29 00:57:05.199458977 +0000 UTC m=+104.888840281, 107.18036ms after stability 2025-07-29T00:57:05.199Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:05.307Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:05.199550897 +0000 UTC m=+104.888932201 (2 nodes), 2.24µs from start of poll, 187.34844ms from cluster start. Timeout at 2025-07-29 00:57:05.307798857 +0000 UTC m=+104.997180161, 108.24796ms after stability 2025-07-29T00:57:05.357Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: entering candidate state: node="Node at 74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d [Candidate]" term=3 2025-07-29T00:57:05.358Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: entering follower state: follower="Node at 1d7c5a64-df8b-e686-618e-fdcb9fb524c8 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.358Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: aborting pipeline replication: peer="{Voter server-db21b492-43c9-0abd-0590-ffb83f366880 db21b492-43c9-0abd-0590-ffb83f366880}" 2025-07-29T00:57:05.358Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: election won: term=3 tally=2 2025-07-29T00:57:05.358Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: entering leader state: leader="Node at 74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d [Leader]" 2025-07-29T00:57:05.358Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: added peer, starting replication: peer=server-db21b492-43c9-0abd-0590-ffb83f366880 2025-07-29T00:57:05.358Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: added peer, starting replication: peer=server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8 2025-07-29T00:57:05.358Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: pipelining replication: peer="{Voter server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8 1d7c5a64-df8b-e686-618e-fdcb9fb524c8}" 2025-07-29T00:57:05.358Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:05.358Z [INFO] server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8: aborting pipeline replication: peer="{Voter server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d 74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d}" 2025-07-29T00:57:05.358Z [WARN] server-db21b492-43c9-0abd-0590-ffb83f366880: failed to get previous log: previous-index=1002 last-index=744 error="log not found" 2025-07-29T00:57:05.358Z [WARN] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: appendEntries rejected, sending older logs: peer="{Voter server-db21b492-43c9-0abd-0590-ffb83f366880 db21b492-43c9-0abd-0590-ffb83f366880}" next=745 2025-07-29T00:57:05.359Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: pipelining replication: peer="{Voter server-db21b492-43c9-0abd-0590-ffb83f366880 db21b492-43c9-0abd-0590-ffb83f366880}" 2025-07-29T00:57:05.464Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:05.358818777 +0000 UTC m=+105.048200081 (1 nodes), 16.2µs from start of poll, 346.61632ms from cluster start. Timeout at 2025-07-29 00:57:05.464076617 +0000 UTC m=+105.153457921, 105.25784ms after stability 2025-07-29T00:57:05.464Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: aborting pipeline replication: peer="{Voter server-1d7c5a64-df8b-e686-618e-fdcb9fb524c8 1d7c5a64-df8b-e686-618e-fdcb9fb524c8}" 2025-07-29T00:57:05.464Z [INFO] server-74e8fc2a-bd0a-5f2c-e44f-c7572912eb8d: aborting pipeline replication: peer="{Voter server-db21b492-43c9-0abd-0590-ffb83f366880 db21b492-43c9-0abd-0590-ffb83f366880}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.45s) === RUN TestRaft_LeadershipTransferToItself 2025-07-29T00:57:05.464Z [INFO] server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed Address:bc5af1a2-3dd3-99f1-7712-d998b042c4ed} {Suffrage:Voter ID:server-74044633-afd3-4970-5ccf-39701f55de0d Address:74044633-afd3-4970-5ccf-39701f55de0d} {Suffrage:Voter ID:server-8036953e-f15f-d11a-2912-9835c8003533 Address:8036953e-f15f-d11a-2912-9835c8003533}]" 2025-07-29T00:57:05.464Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed Address:bc5af1a2-3dd3-99f1-7712-d998b042c4ed} {Suffrage:Voter ID:server-74044633-afd3-4970-5ccf-39701f55de0d Address:74044633-afd3-4970-5ccf-39701f55de0d} {Suffrage:Voter ID:server-8036953e-f15f-d11a-2912-9835c8003533 Address:8036953e-f15f-d11a-2912-9835c8003533}]" 2025-07-29T00:57:05.464Z [INFO] server-8036953e-f15f-d11a-2912-9835c8003533: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed Address:bc5af1a2-3dd3-99f1-7712-d998b042c4ed} {Suffrage:Voter ID:server-74044633-afd3-4970-5ccf-39701f55de0d Address:74044633-afd3-4970-5ccf-39701f55de0d} {Suffrage:Voter ID:server-8036953e-f15f-d11a-2912-9835c8003533 Address:8036953e-f15f-d11a-2912-9835c8003533}]" 2025-07-29T00:57:05.464Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:05.464Z [INFO] server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed: entering follower state: follower="Node at bc5af1a2-3dd3-99f1-7712-d998b042c4ed [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.464Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: entering follower state: follower="Node at 74044633-afd3-4970-5ccf-39701f55de0d [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.464Z [INFO] server-8036953e-f15f-d11a-2912-9835c8003533: entering follower state: follower="Node at 8036953e-f15f-d11a-2912-9835c8003533 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.551Z [WARN] server-74044633-afd3-4970-5ccf-39701f55de0d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:05.552Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: entering candidate state: node="Node at 74044633-afd3-4970-5ccf-39701f55de0d [Candidate]" term=2 2025-07-29T00:57:05.552Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: election won: term=2 tally=2 2025-07-29T00:57:05.552Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: entering leader state: leader="Node at 74044633-afd3-4970-5ccf-39701f55de0d [Leader]" 2025-07-29T00:57:05.552Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: added peer, starting replication: peer=server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed 2025-07-29T00:57:05.552Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: added peer, starting replication: peer=server-8036953e-f15f-d11a-2912-9835c8003533 2025-07-29T00:57:05.552Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: pipelining replication: peer="{Voter server-8036953e-f15f-d11a-2912-9835c8003533 8036953e-f15f-d11a-2912-9835c8003533}" 2025-07-29T00:57:05.552Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: pipelining replication: peer="{Voter server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed bc5af1a2-3dd3-99f1-7712-d998b042c4ed}" 2025-07-29T00:57:05.659Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:05.552595177 +0000 UTC m=+105.241976441 (1 nodes), 87.83688ms from start of poll, 88.0342ms from cluster start. Timeout at 2025-07-29 00:57:05.659474217 +0000 UTC m=+105.348855521, 106.87908ms after stability 2025-07-29T00:57:05.659Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: cannot transfer leadership to itself 2025-07-29T00:57:05.659Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: aborting pipeline replication: peer="{Voter server-bc5af1a2-3dd3-99f1-7712-d998b042c4ed bc5af1a2-3dd3-99f1-7712-d998b042c4ed}" 2025-07-29T00:57:05.659Z [INFO] server-74044633-afd3-4970-5ccf-39701f55de0d: aborting pipeline replication: peer="{Voter server-8036953e-f15f-d11a-2912-9835c8003533 8036953e-f15f-d11a-2912-9835c8003533}" --- PASS: TestRaft_LeadershipTransferToItself (0.20s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2025-07-29T00:57:05.660Z [INFO] server-8e6b24f1-4739-dfa8-4821-09c51b1652f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e6b24f1-4739-dfa8-4821-09c51b1652f5 Address:8e6b24f1-4739-dfa8-4821-09c51b1652f5} {Suffrage:Voter ID:server-0b5b554f-56da-21c7-af1e-af6e6e25218f Address:0b5b554f-56da-21c7-af1e-af6e6e25218f} {Suffrage:Voter ID:server-c8014ba0-4efd-85fd-b8b4-77676805be61 Address:c8014ba0-4efd-85fd-b8b4-77676805be61}]" 2025-07-29T00:57:05.660Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e6b24f1-4739-dfa8-4821-09c51b1652f5 Address:8e6b24f1-4739-dfa8-4821-09c51b1652f5} {Suffrage:Voter ID:server-0b5b554f-56da-21c7-af1e-af6e6e25218f Address:0b5b554f-56da-21c7-af1e-af6e6e25218f} {Suffrage:Voter ID:server-c8014ba0-4efd-85fd-b8b4-77676805be61 Address:c8014ba0-4efd-85fd-b8b4-77676805be61}]" 2025-07-29T00:57:05.660Z [INFO] server-c8014ba0-4efd-85fd-b8b4-77676805be61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e6b24f1-4739-dfa8-4821-09c51b1652f5 Address:8e6b24f1-4739-dfa8-4821-09c51b1652f5} {Suffrage:Voter ID:server-0b5b554f-56da-21c7-af1e-af6e6e25218f Address:0b5b554f-56da-21c7-af1e-af6e6e25218f} {Suffrage:Voter ID:server-c8014ba0-4efd-85fd-b8b4-77676805be61 Address:c8014ba0-4efd-85fd-b8b4-77676805be61}]" 2025-07-29T00:57:05.660Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:05.660Z [INFO] server-8e6b24f1-4739-dfa8-4821-09c51b1652f5: entering follower state: follower="Node at 8e6b24f1-4739-dfa8-4821-09c51b1652f5 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.660Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: entering follower state: follower="Node at 0b5b554f-56da-21c7-af1e-af6e6e25218f [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.660Z [INFO] server-c8014ba0-4efd-85fd-b8b4-77676805be61: entering follower state: follower="Node at c8014ba0-4efd-85fd-b8b4-77676805be61 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.735Z [WARN] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:05.735Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: entering candidate state: node="Node at 0b5b554f-56da-21c7-af1e-af6e6e25218f [Candidate]" term=2 2025-07-29T00:57:05.735Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: election won: term=2 tally=2 2025-07-29T00:57:05.735Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: entering leader state: leader="Node at 0b5b554f-56da-21c7-af1e-af6e6e25218f [Leader]" 2025-07-29T00:57:05.735Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: added peer, starting replication: peer=server-8e6b24f1-4739-dfa8-4821-09c51b1652f5 2025-07-29T00:57:05.735Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: added peer, starting replication: peer=server-c8014ba0-4efd-85fd-b8b4-77676805be61 2025-07-29T00:57:05.735Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: pipelining replication: peer="{Voter server-c8014ba0-4efd-85fd-b8b4-77676805be61 c8014ba0-4efd-85fd-b8b4-77676805be61}" 2025-07-29T00:57:05.735Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: pipelining replication: peer="{Voter server-8e6b24f1-4739-dfa8-4821-09c51b1652f5 8e6b24f1-4739-dfa8-4821-09c51b1652f5}" 2025-07-29T00:57:05.847Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:05.735701177 +0000 UTC m=+105.425082481 (1 nodes), 75.1994ms from start of poll, 75.3976ms from cluster start. Timeout at 2025-07-29 00:57:05.847459257 +0000 UTC m=+105.536840561, 111.75808ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2025-07-29T00:57:05.847Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: aborting pipeline replication: peer="{Voter server-c8014ba0-4efd-85fd-b8b4-77676805be61 c8014ba0-4efd-85fd-b8b4-77676805be61}" 2025-07-29T00:57:05.847Z [INFO] server-0b5b554f-56da-21c7-af1e-af6e6e25218f: aborting pipeline replication: peer="{Voter server-8e6b24f1-4739-dfa8-4821-09c51b1652f5 8e6b24f1-4739-dfa8-4821-09c51b1652f5}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.19s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2025-07-29T00:57:05.848Z [INFO] server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 Address:78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9} {Suffrage:Voter ID:server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3 Address:b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3} {Suffrage:Voter ID:server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 Address:a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298}]" 2025-07-29T00:57:05.848Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 Address:78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9} {Suffrage:Voter ID:server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3 Address:b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3} {Suffrage:Voter ID:server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 Address:a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298}]" 2025-07-29T00:57:05.848Z [INFO] server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 Address:78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9} {Suffrage:Voter ID:server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3 Address:b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3} {Suffrage:Voter ID:server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 Address:a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298}]" 2025-07-29T00:57:05.848Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:05.848Z [INFO] server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9: entering follower state: follower="Node at 78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.848Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: entering follower state: follower="Node at b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.848Z [INFO] server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298: entering follower state: follower="Node at a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 [Follower]" leader-address= leader-id= 2025-07-29T00:57:05.901Z [WARN] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:05.901Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: entering candidate state: node="Node at b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3 [Candidate]" term=2 2025-07-29T00:57:05.901Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: election won: term=2 tally=2 2025-07-29T00:57:05.901Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: entering leader state: leader="Node at b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3 [Leader]" 2025-07-29T00:57:05.901Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: added peer, starting replication: peer=server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 2025-07-29T00:57:05.901Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: added peer, starting replication: peer=server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 2025-07-29T00:57:05.901Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: pipelining replication: peer="{Voter server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298}" 2025-07-29T00:57:05.902Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: pipelining replication: peer="{Voter server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9}" 2025-07-29T00:57:06.007Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:05.902088577 +0000 UTC m=+105.591469881 (1 nodes), 53.64772ms from start of poll, 53.83524ms from cluster start. Timeout at 2025-07-29 00:57:06.007455537 +0000 UTC m=+105.696836801, 105.36692ms after stability 2025-07-29T00:57:06.007Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:06.115Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:06.007545217 +0000 UTC m=+105.696926521 (2 nodes), 2.32µs from start of poll, 159.29188ms from cluster start. Timeout at 2025-07-29 00:57:06.115908537 +0000 UTC m=+105.805289841, 108.36332ms after stability 2025-07-29T00:57:06.679Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: aborting pipeline replication: peer="{Voter server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298}" 2025-07-29T00:57:06.683Z [INFO] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: aborting pipeline replication: peer="{Voter server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9}" 2025-07-29T00:57:07.179Z [ERROR] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: failed to appendEntries to: peer="{Voter server-a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298}" error="command timed out" 2025-07-29T00:57:07.183Z [ERROR] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: failed to heartbeat to: peer=a8d8ce6e-dad3-6e4e-c317-bf08bb3ab298 backoff time=10ms error="command timed out" 2025-07-29T00:57:07.183Z [ERROR] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: failed to heartbeat to: peer=78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 backoff time=10ms error="command timed out" 2025-07-29T00:57:07.183Z [ERROR] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: failed to appendEntries to: peer="{Voter server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9}" error="command timed out" 2025-07-29T00:57:07.687Z [ERROR] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: failed to appendEntries to: peer="{Voter server-78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9}" error="command timed out" 2025-07-29T00:57:07.716Z [ERROR] server-b4d02ea5-5ebc-78a3-71f1-c4b85c9cbac3: failed to heartbeat to: peer=78c7f464-4c47-36c6-41b0-aa2f1b3d2fd9 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (1.87s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2025-07-29T00:57:07.716Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd4e4527-087e-3105-953f-097249c93e6c Address:dd4e4527-087e-3105-953f-097249c93e6c} {Suffrage:Voter ID:server-5c03fd0d-af75-8b69-1208-58e368c1d866 Address:5c03fd0d-af75-8b69-1208-58e368c1d866}]" 2025-07-29T00:57:07.716Z [INFO] server-5c03fd0d-af75-8b69-1208-58e368c1d866: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd4e4527-087e-3105-953f-097249c93e6c Address:dd4e4527-087e-3105-953f-097249c93e6c} {Suffrage:Voter ID:server-5c03fd0d-af75-8b69-1208-58e368c1d866 Address:5c03fd0d-af75-8b69-1208-58e368c1d866}]" 2025-07-29T00:57:07.716Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:07.716Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: entering follower state: follower="Node at dd4e4527-087e-3105-953f-097249c93e6c [Follower]" leader-address= leader-id= 2025-07-29T00:57:07.716Z [INFO] server-5c03fd0d-af75-8b69-1208-58e368c1d866: entering follower state: follower="Node at 5c03fd0d-af75-8b69-1208-58e368c1d866 [Follower]" leader-address= leader-id= 2025-07-29T00:57:07.787Z [WARN] server-dd4e4527-087e-3105-953f-097249c93e6c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:07.787Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: entering candidate state: node="Node at dd4e4527-087e-3105-953f-097249c93e6c [Candidate]" term=2 2025-07-29T00:57:07.787Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: election won: term=2 tally=2 2025-07-29T00:57:07.787Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: entering leader state: leader="Node at dd4e4527-087e-3105-953f-097249c93e6c [Leader]" 2025-07-29T00:57:07.787Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: added peer, starting replication: peer=server-5c03fd0d-af75-8b69-1208-58e368c1d866 2025-07-29T00:57:07.787Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: pipelining replication: peer="{Voter server-5c03fd0d-af75-8b69-1208-58e368c1d866 5c03fd0d-af75-8b69-1208-58e368c1d866}" 2025-07-29T00:57:07.895Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:07.787710217 +0000 UTC m=+107.477091521 (1 nodes), 71.13012ms from start of poll, 71.27948ms from cluster start. Timeout at 2025-07-29 00:57:07.895669897 +0000 UTC m=+107.585051201, 107.95968ms after stability 2025-07-29T00:57:07.895Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:08.003Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:07.895773017 +0000 UTC m=+107.585154321 (1 nodes), 1.44µs from start of poll, 179.34228ms from cluster start. Timeout at 2025-07-29 00:57:08.003473577 +0000 UTC m=+107.692854881, 107.70056ms after stability 2025-07-29T00:57:08.003Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: updating configuration: command=DemoteVoter server-id=server-5c03fd0d-af75-8b69-1208-58e368c1d866 server-addr= servers="[{Suffrage:Voter ID:server-dd4e4527-087e-3105-953f-097249c93e6c Address:dd4e4527-087e-3105-953f-097249c93e6c} {Suffrage:Nonvoter ID:server-5c03fd0d-af75-8b69-1208-58e368c1d866 Address:5c03fd0d-af75-8b69-1208-58e368c1d866}]" 2025-07-29T00:57:08.003Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:08.111Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:08.003662697 +0000 UTC m=+107.693044001 (1 nodes), 560ns from start of poll, 287.23196ms from cluster start. Timeout at 2025-07-29 00:57:08.111815097 +0000 UTC m=+107.801196401, 108.1524ms after stability 2025-07-29T00:57:08.112Z [INFO] server-dd4e4527-087e-3105-953f-097249c93e6c: aborting pipeline replication: peer="{Voter server-5c03fd0d-af75-8b69-1208-58e368c1d866 5c03fd0d-af75-8b69-1208-58e368c1d866}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.40s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2025-07-29T00:57:08.112Z [INFO] server-ed0fb1b9-b7ed-8add-6264-907066a5981b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed0fb1b9-b7ed-8add-6264-907066a5981b Address:ed0fb1b9-b7ed-8add-6264-907066a5981b} {Suffrage:Voter ID:server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae Address:ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae}]" 2025-07-29T00:57:08.112Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed0fb1b9-b7ed-8add-6264-907066a5981b Address:ed0fb1b9-b7ed-8add-6264-907066a5981b} {Suffrage:Voter ID:server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae Address:ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae}]" 2025-07-29T00:57:08.112Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:08.112Z [INFO] server-ed0fb1b9-b7ed-8add-6264-907066a5981b: entering follower state: follower="Node at ed0fb1b9-b7ed-8add-6264-907066a5981b [Follower]" leader-address= leader-id= 2025-07-29T00:57:08.112Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: entering follower state: follower="Node at ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae [Follower]" leader-address= leader-id= 2025-07-29T00:57:08.179Z [WARN] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:08.179Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: entering candidate state: node="Node at ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae [Candidate]" term=2 2025-07-29T00:57:08.179Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: election won: term=2 tally=2 2025-07-29T00:57:08.179Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: entering leader state: leader="Node at ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae [Leader]" 2025-07-29T00:57:08.179Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: added peer, starting replication: peer=server-ed0fb1b9-b7ed-8add-6264-907066a5981b 2025-07-29T00:57:08.179Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: pipelining replication: peer="{Voter server-ed0fb1b9-b7ed-8add-6264-907066a5981b ed0fb1b9-b7ed-8add-6264-907066a5981b}" 2025-07-29T00:57:08.287Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:08.180143337 +0000 UTC m=+107.869524641 (1 nodes), 67.6362ms from start of poll, 67.77344ms from cluster start. Timeout at 2025-07-29 00:57:08.287457377 +0000 UTC m=+107.976838681, 107.31404ms after stability 2025-07-29T00:57:08.287Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:08.395Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:08.287580617 +0000 UTC m=+107.976961921 (1 nodes), 920ns from start of poll, 175.21072ms from cluster start. Timeout at 2025-07-29 00:57:08.395467417 +0000 UTC m=+108.084848721, 107.8868ms after stability 2025-07-29T00:57:08.415Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed0fb1b9-b7ed-8add-6264-907066a5981b Address:ed0fb1b9-b7ed-8add-6264-907066a5981b} {Suffrage:Voter ID:server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae Address:ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae}]" 2025-07-29T00:57:08.415Z [INFO] server-ce7ba8d4-f369-5fcd-5870-2d5fe77a2cae: aborting pipeline replication: peer="{Voter server-ed0fb1b9-b7ed-8add-6264-907066a5981b ed0fb1b9-b7ed-8add-6264-907066a5981b}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s) === RUN TestRaft_LogStoreIsMonotonic 2025-07-29T00:57:08.416Z [INFO] server-6f5f4b03-da71-b8ee-5cca-7acb2309fdaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f5f4b03-da71-b8ee-5cca-7acb2309fdaa Address:6f5f4b03-da71-b8ee-5cca-7acb2309fdaa}]" 2025-07-29T00:57:08.416Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:08.416Z [INFO] server-6f5f4b03-da71-b8ee-5cca-7acb2309fdaa: entering follower state: follower="Node at 6f5f4b03-da71-b8ee-5cca-7acb2309fdaa [Follower]" leader-address= leader-id= 2025-07-29T00:57:08.487Z [WARN] server-6f5f4b03-da71-b8ee-5cca-7acb2309fdaa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:08.487Z [INFO] server-6f5f4b03-da71-b8ee-5cca-7acb2309fdaa: entering candidate state: node="Node at 6f5f4b03-da71-b8ee-5cca-7acb2309fdaa [Candidate]" term=2 2025-07-29T00:57:08.503Z [INFO] server-6f5f4b03-da71-b8ee-5cca-7acb2309fdaa: election won: term=2 tally=1 2025-07-29T00:57:08.503Z [INFO] server-6f5f4b03-da71-b8ee-5cca-7acb2309fdaa: entering leader state: leader="Node at 6f5f4b03-da71-b8ee-5cca-7acb2309fdaa [Leader]" 2025-07-29T00:57:08.635Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:08.527690137 +0000 UTC m=+108.217071401 (1 nodes), 111.57172ms from start of poll, 111.66476ms from cluster start. Timeout at 2025-07-29 00:57:08.635628617 +0000 UTC m=+108.325009921, 107.93852ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.22s) === RUN TestRaft_CacheLogWithStoreError 2025-07-29T00:57:08.636Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd Address:c189479a-0ca3-70ad-b2a2-b83f3d813ebd} {Suffrage:Voter ID:server-33455e84-5471-63ea-b559-9c47d35892e6 Address:33455e84-5471-63ea-b559-9c47d35892e6}]" 2025-07-29T00:57:08.636Z [INFO] server-33455e84-5471-63ea-b559-9c47d35892e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd Address:c189479a-0ca3-70ad-b2a2-b83f3d813ebd} {Suffrage:Voter ID:server-33455e84-5471-63ea-b559-9c47d35892e6 Address:33455e84-5471-63ea-b559-9c47d35892e6}]" 2025-07-29T00:57:08.636Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:08.636Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: entering follower state: follower="Node at c189479a-0ca3-70ad-b2a2-b83f3d813ebd [Follower]" leader-address= leader-id= 2025-07-29T00:57:08.636Z [INFO] server-33455e84-5471-63ea-b559-9c47d35892e6: entering follower state: follower="Node at 33455e84-5471-63ea-b559-9c47d35892e6 [Follower]" leader-address= leader-id= 2025-07-29T00:57:08.695Z [WARN] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:08.695Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: entering candidate state: node="Node at c189479a-0ca3-70ad-b2a2-b83f3d813ebd [Candidate]" term=2 2025-07-29T00:57:08.695Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: election won: term=2 tally=2 2025-07-29T00:57:08.695Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: entering leader state: leader="Node at c189479a-0ca3-70ad-b2a2-b83f3d813ebd [Leader]" 2025-07-29T00:57:08.695Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: added peer, starting replication: peer=server-33455e84-5471-63ea-b559-9c47d35892e6 2025-07-29T00:57:08.695Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: pipelining replication: peer="{Voter server-33455e84-5471-63ea-b559-9c47d35892e6 33455e84-5471-63ea-b559-9c47d35892e6}" 2025-07-29T00:57:08.803Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:08.695928257 +0000 UTC m=+108.385309561 (1 nodes), 59.8278ms from start of poll, 59.97056ms from cluster start. Timeout at 2025-07-29 00:57:08.803447177 +0000 UTC m=+108.492828481, 107.51892ms after stability 2025-07-29T00:57:08.803Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:08.915Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:08.803527977 +0000 UTC m=+108.492909241 (1 nodes), 800ns from start of poll, 167.57024ms from cluster start. Timeout at 2025-07-29 00:57:08.915478337 +0000 UTC m=+108.604859601, 111.95036ms after stability 2025-07-29T00:57:08.915Z [ERROR] server-33455e84-5471-63ea-b559-9c47d35892e6: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2025-07-29T00:57:08.915Z [WARN] server-33455e84-5471-63ea-b559-9c47d35892e6: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2025-07-29T00:57:08.915Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: aborting pipeline replication: peer="{Voter server-33455e84-5471-63ea-b559-9c47d35892e6 33455e84-5471-63ea-b559-9c47d35892e6}" 2025-07-29T00:57:08.923Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: pipelining replication: peer="{Voter server-33455e84-5471-63ea-b559-9c47d35892e6 33455e84-5471-63ea-b559-9c47d35892e6}" 2025-07-29T00:57:08.927Z [INFO] server-33455e84-5471-63ea-b559-9c47d35892e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd Address:c189479a-0ca3-70ad-b2a2-b83f3d813ebd} {Suffrage:Voter ID:server-33455e84-5471-63ea-b559-9c47d35892e6 Address:33455e84-5471-63ea-b559-9c47d35892e6}]" 2025-07-29T00:57:08.927Z [INFO] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: aborting pipeline replication: peer="{Voter server-33455e84-5471-63ea-b559-9c47d35892e6 33455e84-5471-63ea-b559-9c47d35892e6}" 2025-07-29T00:57:09.427Z [ERROR] server-c189479a-0ca3-70ad-b2a2-b83f3d813ebd: failed to heartbeat to: peer=33455e84-5471-63ea-b559-9c47d35892e6 backoff time=10ms error="command timed out" --- PASS: TestRaft_CacheLogWithStoreError (0.79s) === RUN TestRaft_ReloadConfig 2025-07-29T00:57:09.427Z [INFO] server-0711939d-fdce-3be6-a71a-af5932167501: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0711939d-fdce-3be6-a71a-af5932167501 Address:0711939d-fdce-3be6-a71a-af5932167501}]" --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2025-07-29T00:57:09.428Z [INFO] server-4deb900b-80b3-f6fc-1763-e2823ff8c67f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4deb900b-80b3-f6fc-1763-e2823ff8c67f Address:4deb900b-80b3-f6fc-1763-e2823ff8c67f}]" --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2025-07-29T00:57:09.428Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2025-07-29T00:57:09.428Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-936bf041-dc30-98a6-47bb-d6435625aaf8 Address:936bf041-dc30-98a6-47bb-d6435625aaf8} {Suffrage:Voter ID:server-aabee077-104b-e1ed-1785-ea29d410e10b Address:aabee077-104b-e1ed-1785-ea29d410e10b} {Suffrage:Voter ID:server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e Address:9260eba5-f3ef-51c3-ffad-fa9672f3f15e}]" 2025-07-29T00:57:09.428Z [INFO] server-aabee077-104b-e1ed-1785-ea29d410e10b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-936bf041-dc30-98a6-47bb-d6435625aaf8 Address:936bf041-dc30-98a6-47bb-d6435625aaf8} {Suffrage:Voter ID:server-aabee077-104b-e1ed-1785-ea29d410e10b Address:aabee077-104b-e1ed-1785-ea29d410e10b} {Suffrage:Voter ID:server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e Address:9260eba5-f3ef-51c3-ffad-fa9672f3f15e}]" 2025-07-29T00:57:09.428Z [INFO] server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-936bf041-dc30-98a6-47bb-d6435625aaf8 Address:936bf041-dc30-98a6-47bb-d6435625aaf8} {Suffrage:Voter ID:server-aabee077-104b-e1ed-1785-ea29d410e10b Address:aabee077-104b-e1ed-1785-ea29d410e10b} {Suffrage:Voter ID:server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e Address:9260eba5-f3ef-51c3-ffad-fa9672f3f15e}]" 2025-07-29T00:57:09.428Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:09.428Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: entering follower state: follower="Node at 936bf041-dc30-98a6-47bb-d6435625aaf8 [Follower]" leader-address= leader-id= 2025-07-29T00:57:09.428Z [INFO] server-aabee077-104b-e1ed-1785-ea29d410e10b: entering follower state: follower="Node at aabee077-104b-e1ed-1785-ea29d410e10b [Follower]" leader-address= leader-id= 2025-07-29T00:57:09.428Z [INFO] server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e: entering follower state: follower="Node at 9260eba5-f3ef-51c3-ffad-fa9672f3f15e [Follower]" leader-address= leader-id= 2025-07-29T00:57:09.511Z [WARN] server-936bf041-dc30-98a6-47bb-d6435625aaf8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:09.511Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: entering candidate state: node="Node at 936bf041-dc30-98a6-47bb-d6435625aaf8 [Candidate]" term=2 2025-07-29T00:57:09.512Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: election won: term=2 tally=2 2025-07-29T00:57:09.512Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: entering leader state: leader="Node at 936bf041-dc30-98a6-47bb-d6435625aaf8 [Leader]" 2025-07-29T00:57:09.512Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: added peer, starting replication: peer=server-aabee077-104b-e1ed-1785-ea29d410e10b 2025-07-29T00:57:09.512Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: added peer, starting replication: peer=server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e 2025-07-29T00:57:09.512Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: pipelining replication: peer="{Voter server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e 9260eba5-f3ef-51c3-ffad-fa9672f3f15e}" 2025-07-29T00:57:09.512Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: pipelining replication: peer="{Voter server-aabee077-104b-e1ed-1785-ea29d410e10b aabee077-104b-e1ed-1785-ea29d410e10b}" 2025-07-29T00:57:09.619Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:09.512248737 +0000 UTC m=+109.201630041 (1 nodes), 83.60744ms from start of poll, 83.77416ms from cluster start. Timeout at 2025-07-29 00:57:09.619473377 +0000 UTC m=+109.308854681, 107.22464ms after stability 2025-07-29T00:57:09.627Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:09.735Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:09.627561097 +0000 UTC m=+109.316942401 (2 nodes), 4.44µs from start of poll, 199.08652ms from cluster start. Timeout at 2025-07-29 00:57:09.735798937 +0000 UTC m=+109.425180201, 108.2378ms after stability 2025-07-29T00:57:09.735Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: updating configuration: command=RemoveServer server-id=server-aabee077-104b-e1ed-1785-ea29d410e10b server-addr= servers="[{Suffrage:Voter ID:server-936bf041-dc30-98a6-47bb-d6435625aaf8 Address:936bf041-dc30-98a6-47bb-d6435625aaf8} {Suffrage:Voter ID:server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e Address:9260eba5-f3ef-51c3-ffad-fa9672f3f15e}]" 2025-07-29T00:57:09.735Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: removed peer, stopping replication: peer=server-aabee077-104b-e1ed-1785-ea29d410e10b last-index=3 2025-07-29T00:57:09.736Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: aborting pipeline replication: peer="{Voter server-aabee077-104b-e1ed-1785-ea29d410e10b aabee077-104b-e1ed-1785-ea29d410e10b}" 2025-07-29T00:57:09.824Z [WARN] server-aabee077-104b-e1ed-1785-ea29d410e10b: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-07-29T00:57:09.847Z [ERROR] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to heartbeat to: peer=9260eba5-f3ef-51c3-ffad-fa9672f3f15e backoff time=10ms error="failed to connect to peer: 9260eba5-f3ef-51c3-ffad-fa9672f3f15e" 2025-07-29T00:57:09.855Z [ERROR] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to pipeline appendEntries: peer="{Voter server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e 9260eba5-f3ef-51c3-ffad-fa9672f3f15e}" error="append pipeline closed" 2025-07-29T00:57:09.855Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: aborting pipeline replication: peer="{Voter server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e 9260eba5-f3ef-51c3-ffad-fa9672f3f15e}" 2025-07-29T00:57:09.875Z [ERROR] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to appendEntries to: peer="{Voter server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e 9260eba5-f3ef-51c3-ffad-fa9672f3f15e}" error="failed to connect to peer: 9260eba5-f3ef-51c3-ffad-fa9672f3f15e" 2025-07-29T00:57:09.875Z [ERROR] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to heartbeat to: peer=9260eba5-f3ef-51c3-ffad-fa9672f3f15e backoff time=10ms error="failed to connect to peer: 9260eba5-f3ef-51c3-ffad-fa9672f3f15e" 2025-07-29T00:57:09.891Z [WARN] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to contact: server-id=server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e time=51.89708ms 2025-07-29T00:57:09.891Z [WARN] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to contact quorum of nodes, stepping down 2025-07-29T00:57:09.891Z [INFO] server-936bf041-dc30-98a6-47bb-d6435625aaf8: entering follower state: follower="Node at 936bf041-dc30-98a6-47bb-d6435625aaf8 [Follower]" leader-address= leader-id= 2025-07-29T00:57:09.895Z [ERROR] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to appendEntries to: peer="{Voter server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e 9260eba5-f3ef-51c3-ffad-fa9672f3f15e}" error="failed to connect to peer: 9260eba5-f3ef-51c3-ffad-fa9672f3f15e" 2025-07-29T00:57:09.895Z [ERROR] server-936bf041-dc30-98a6-47bb-d6435625aaf8: failed to heartbeat to: peer=9260eba5-f3ef-51c3-ffad-fa9672f3f15e backoff time=10ms error="failed to connect to peer: 9260eba5-f3ef-51c3-ffad-fa9672f3f15e" 2025-07-29T00:57:09.927Z [WARN] server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e: heartbeat timeout reached, starting election: last-leader-addr=936bf041-dc30-98a6-47bb-d6435625aaf8 last-leader-id=server-936bf041-dc30-98a6-47bb-d6435625aaf8 2025-07-29T00:57:09.927Z [INFO] server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e: entering candidate state: node="Node at 9260eba5-f3ef-51c3-ffad-fa9672f3f15e [Candidate]" term=3 2025-07-29T00:57:09.927Z [ERROR] server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e: failed to make requestVote RPC: target="{Voter server-936bf041-dc30-98a6-47bb-d6435625aaf8 936bf041-dc30-98a6-47bb-d6435625aaf8}" error="failed to connect to peer: 936bf041-dc30-98a6-47bb-d6435625aaf8" term=3 2025-07-29T00:57:09.951Z [WARN] server-9260eba5-f3ef-51c3-ffad-fa9672f3f15e: rejecting vote request since node is not in configuration: from=aabee077-104b-e1ed-1785-ea29d410e10b --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.52s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2025-07-29T00:57:09.952Z [INFO] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 Address:31f920be-2fe8-13a2-0973-0c1410ca0bd9} {Suffrage:Voter ID:server-5d6207ce-a6c9-f725-d21e-76042527c586 Address:5d6207ce-a6c9-f725-d21e-76042527c586} {Suffrage:Voter ID:server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 Address:e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}]" 2025-07-29T00:57:09.952Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 Address:31f920be-2fe8-13a2-0973-0c1410ca0bd9} {Suffrage:Voter ID:server-5d6207ce-a6c9-f725-d21e-76042527c586 Address:5d6207ce-a6c9-f725-d21e-76042527c586} {Suffrage:Voter ID:server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 Address:e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}]" 2025-07-29T00:57:09.952Z [INFO] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 Address:31f920be-2fe8-13a2-0973-0c1410ca0bd9} {Suffrage:Voter ID:server-5d6207ce-a6c9-f725-d21e-76042527c586 Address:5d6207ce-a6c9-f725-d21e-76042527c586} {Suffrage:Voter ID:server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 Address:e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}]" 2025-07-29T00:57:09.952Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:09.952Z [INFO] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: entering follower state: follower="Node at 31f920be-2fe8-13a2-0973-0c1410ca0bd9 [Follower]" leader-address= leader-id= 2025-07-29T00:57:09.952Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: entering follower state: follower="Node at 5d6207ce-a6c9-f725-d21e-76042527c586 [Follower]" leader-address= leader-id= 2025-07-29T00:57:09.952Z [INFO] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: entering follower state: follower="Node at e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 [Follower]" leader-address= leader-id= 2025-07-29T00:57:10.035Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:10.035Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: entering candidate state: node="Node at 5d6207ce-a6c9-f725-d21e-76042527c586 [Candidate]" term=2 2025-07-29T00:57:10.035Z [WARN] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:10.035Z [INFO] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: entering candidate state: node="Node at e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 [Candidate]" term=2 2025-07-29T00:57:10.035Z [INFO] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: duplicate requestVote for same term: term=2 2025-07-29T00:57:10.035Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: election won: term=2 tally=2 2025-07-29T00:57:10.035Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: entering leader state: leader="Node at 5d6207ce-a6c9-f725-d21e-76042527c586 [Leader]" 2025-07-29T00:57:10.035Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: added peer, starting replication: peer=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 2025-07-29T00:57:10.035Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: added peer, starting replication: peer=server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 2025-07-29T00:57:10.035Z [INFO] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: entering follower state: follower="Node at e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 [Follower]" leader-address=5d6207ce-a6c9-f725-d21e-76042527c586 leader-id=server-5d6207ce-a6c9-f725-d21e-76042527c586 2025-07-29T00:57:10.035Z [INFO] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: duplicate requestVote for same term: term=2 2025-07-29T00:57:10.035Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: pipelining replication: peer="{Voter server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}" 2025-07-29T00:57:10.035Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: rejecting vote request since we have a leader: from=e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 leader=5d6207ce-a6c9-f725-d21e-76042527c586 leader-id=server-5d6207ce-a6c9-f725-d21e-76042527c586 2025-07-29T00:57:10.035Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: pipelining replication: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:10.143Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:10.035871857 +0000 UTC m=+109.725253161 (1 nodes), 83.34572ms from start of poll, 83.55736ms from cluster start. Timeout at 2025-07-29 00:57:10.143929097 +0000 UTC m=+109.833310401, 108.05724ms after stability 2025-07-29T00:57:10.151Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:10.260Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:10.151529057 +0000 UTC m=+109.840910361 (2 nodes), 4.32µs from start of poll, 199.21456ms from cluster start. Timeout at 2025-07-29 00:57:10.259982217 +0000 UTC m=+109.949363521, 108.45316ms after stability 2025-07-29T00:57:10.263Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to heartbeat to: peer=31f920be-2fe8-13a2-0973-0c1410ca0bd9 backoff time=10ms error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.267Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to pipeline appendEntries: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" error="append pipeline closed" 2025-07-29T00:57:10.267Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: aborting pipeline replication: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:10.279Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to appendEntries to: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.287Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to heartbeat to: peer=31f920be-2fe8-13a2-0973-0c1410ca0bd9 backoff time=10ms error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.299Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to appendEntries to: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.307Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to heartbeat to: peer=31f920be-2fe8-13a2-0973-0c1410ca0bd9 backoff time=10ms error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.311Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to contact: server-id=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 time=51.9484ms 2025-07-29T00:57:10.319Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to appendEntries to: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.327Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to heartbeat to: peer=31f920be-2fe8-13a2-0973-0c1410ca0bd9 backoff time=20ms error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.343Z [WARN] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: heartbeat timeout reached, starting election: last-leader-addr=5d6207ce-a6c9-f725-d21e-76042527c586 last-leader-id=server-5d6207ce-a6c9-f725-d21e-76042527c586 2025-07-29T00:57:10.343Z [INFO] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: entering candidate state: node="Node at 31f920be-2fe8-13a2-0973-0c1410ca0bd9 [Candidate]" term=3 2025-07-29T00:57:10.343Z [ERROR] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: failed to make requestVote RPC: target="{Voter server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}" error="failed to connect to peer: e845e2ac-05b7-9615-4a8e-ecc2c936c1b0" term=3 2025-07-29T00:57:10.344Z [ERROR] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: failed to make requestVote RPC: target="{Voter server-5d6207ce-a6c9-f725-d21e-76042527c586 5d6207ce-a6c9-f725-d21e-76042527c586}" error="failed to connect to peer: 5d6207ce-a6c9-f725-d21e-76042527c586" term=3 2025-07-29T00:57:10.347Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to appendEntries to: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.359Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to heartbeat to: peer=31f920be-2fe8-13a2-0973-0c1410ca0bd9 backoff time=25ms error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.363Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to contact: server-id=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 time=103.92788ms 2025-07-29T00:57:10.368Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: updating configuration: command=RemoveServer server-id=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 server-addr= servers="[{Suffrage:Voter ID:server-5d6207ce-a6c9-f725-d21e-76042527c586 Address:5d6207ce-a6c9-f725-d21e-76042527c586} {Suffrage:Voter ID:server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 Address:e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}]" 2025-07-29T00:57:10.368Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: removed peer, stopping replication: peer=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 last-index=3 2025-07-29T00:57:10.368Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: updating configuration: command=AddNonvoter server-id=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 server-addr=31f920be-2fe8-13a2-0973-0c1410ca0bd9 servers="[{Suffrage:Voter ID:server-5d6207ce-a6c9-f725-d21e-76042527c586 Address:5d6207ce-a6c9-f725-d21e-76042527c586} {Suffrage:Voter ID:server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 Address:e845e2ac-05b7-9615-4a8e-ecc2c936c1b0} {Suffrage:Nonvoter ID:server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 Address:31f920be-2fe8-13a2-0973-0c1410ca0bd9}]" 2025-07-29T00:57:10.368Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: added peer, starting replication: peer=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 2025-07-29T00:57:10.368Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: failed to appendEntries to: peer="{Nonvoter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" error="failed to connect to peer: 31f920be-2fe8-13a2-0973-0c1410ca0bd9" 2025-07-29T00:57:10.395Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: peer has newer term, stopping replication: peer="{Nonvoter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:10.395Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: entering follower state: follower="Node at 5d6207ce-a6c9-f725-d21e-76042527c586 [Follower]" leader-address= leader-id= 2025-07-29T00:57:10.395Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: aborting pipeline replication: peer="{Voter server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}" 2025-07-29T00:57:10.399Z [ERROR] server-5d6207ce-a6c9-f725-d21e-76042527c586: peer has newer term, stopping replication: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:10.415Z [WARN] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: Election timeout reached, restarting election 2025-07-29T00:57:10.415Z [INFO] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: entering candidate state: node="Node at 31f920be-2fe8-13a2-0973-0c1410ca0bd9 [Candidate]" term=23 2025-07-29T00:57:10.415Z [WARN] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: rejecting vote request since we have a leader: from=31f920be-2fe8-13a2-0973-0c1410ca0bd9 leader=5d6207ce-a6c9-f725-d21e-76042527c586 leader-id=server-5d6207ce-a6c9-f725-d21e-76042527c586 2025-07-29T00:57:10.415Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: rejecting vote request since node is not a voter: from=31f920be-2fe8-13a2-0973-0c1410ca0bd9 2025-07-29T00:57:10.471Z [WARN] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: heartbeat timeout reached, starting election: last-leader-addr=5d6207ce-a6c9-f725-d21e-76042527c586 last-leader-id=server-5d6207ce-a6c9-f725-d21e-76042527c586 2025-07-29T00:57:10.471Z [INFO] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: entering candidate state: node="Node at e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 [Candidate]" term=3 2025-07-29T00:57:10.471Z [INFO] server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0: entering follower state: follower="Node at e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 [Follower]" leader-address= leader-id= 2025-07-29T00:57:10.471Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:10.471Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: entering candidate state: node="Node at 5d6207ce-a6c9-f725-d21e-76042527c586 [Candidate]" term=24 2025-07-29T00:57:10.471Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: election won: term=24 tally=2 2025-07-29T00:57:10.471Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: entering leader state: leader="Node at 5d6207ce-a6c9-f725-d21e-76042527c586 [Leader]" 2025-07-29T00:57:10.471Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: added peer, starting replication: peer=server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 2025-07-29T00:57:10.471Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: added peer, starting replication: peer=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 2025-07-29T00:57:10.471Z [WARN] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: failed to get previous log: previous-index=4 last-index=2 error="log not found" 2025-07-29T00:57:10.471Z [INFO] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: entering follower state: follower="Node at 31f920be-2fe8-13a2-0973-0c1410ca0bd9 [Follower]" leader-address=5d6207ce-a6c9-f725-d21e-76042527c586 leader-id=server-5d6207ce-a6c9-f725-d21e-76042527c586 2025-07-29T00:57:10.471Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: appendEntries rejected, sending older logs: peer="{Nonvoter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" next=3 2025-07-29T00:57:10.472Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: pipelining replication: peer="{Nonvoter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:10.472Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: pipelining replication: peer="{Voter server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}" 2025-07-29T00:57:10.475Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:10.591Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:10.483750777 +0000 UTC m=+110.173132081 (1 nodes), 8.16264ms from start of poll, 531.43628ms from cluster start. Timeout at 2025-07-29 00:57:10.591451617 +0000 UTC m=+110.280832881, 107.7008ms after stability 2025-07-29T00:57:10.643Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: updating configuration: command=RemoveServer server-id=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 server-addr= servers="[{Suffrage:Voter ID:server-5d6207ce-a6c9-f725-d21e-76042527c586 Address:5d6207ce-a6c9-f725-d21e-76042527c586} {Suffrage:Voter ID:server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 Address:e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}]" 2025-07-29T00:57:10.643Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: removed peer, stopping replication: peer=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 last-index=106 2025-07-29T00:57:10.643Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: aborting pipeline replication: peer="{Nonvoter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:10.643Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: updating configuration: command=AddVoter server-id=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 server-addr=31f920be-2fe8-13a2-0973-0c1410ca0bd9 servers="[{Suffrage:Voter ID:server-5d6207ce-a6c9-f725-d21e-76042527c586 Address:5d6207ce-a6c9-f725-d21e-76042527c586} {Suffrage:Voter ID:server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 Address:e845e2ac-05b7-9615-4a8e-ecc2c936c1b0} {Suffrage:Voter ID:server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 Address:31f920be-2fe8-13a2-0973-0c1410ca0bd9}]" 2025-07-29T00:57:10.643Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: added peer, starting replication: peer=server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 2025-07-29T00:57:10.643Z [WARN] server-31f920be-2fe8-13a2-0973-0c1410ca0bd9: failed to get previous log: previous-index=107 last-index=106 error="log not found" 2025-07-29T00:57:10.643Z [WARN] server-5d6207ce-a6c9-f725-d21e-76042527c586: appendEntries rejected, sending older logs: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" next=107 2025-07-29T00:57:10.644Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: pipelining replication: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:11.719Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:11.826Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:11.719676417 +0000 UTC m=+111.409057681 (1 nodes), 1.16µs from start of poll, 1.76736188s from cluster start. Timeout at 2025-07-29 00:57:11.826209577 +0000 UTC m=+111.515590841, 106.53316ms after stability 2025-07-29T00:57:11.826Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: aborting pipeline replication: peer="{Voter server-31f920be-2fe8-13a2-0973-0c1410ca0bd9 31f920be-2fe8-13a2-0973-0c1410ca0bd9}" 2025-07-29T00:57:11.826Z [INFO] server-5d6207ce-a6c9-f725-d21e-76042527c586: aborting pipeline replication: peer="{Voter server-e845e2ac-05b7-9615-4a8e-ecc2c936c1b0 e845e2ac-05b7-9615-4a8e-ecc2c936c1b0}" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.87s) === RUN TestRaft_FollowerRemovalNoElection 2025-07-29T00:57:11.826Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5bbe77d-7984-0bc4-9834-62d46c204d83 Address:e5bbe77d-7984-0bc4-9834-62d46c204d83} {Suffrage:Voter ID:server-0e94a776-473f-7c37-e54a-f088e0d408b0 Address:0e94a776-473f-7c37-e54a-f088e0d408b0} {Suffrage:Voter ID:server-ef1dab60-c52c-2f03-533a-49631e11d91c Address:ef1dab60-c52c-2f03-533a-49631e11d91c}]" 2025-07-29T00:57:11.827Z [INFO] server-0e94a776-473f-7c37-e54a-f088e0d408b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5bbe77d-7984-0bc4-9834-62d46c204d83 Address:e5bbe77d-7984-0bc4-9834-62d46c204d83} {Suffrage:Voter ID:server-0e94a776-473f-7c37-e54a-f088e0d408b0 Address:0e94a776-473f-7c37-e54a-f088e0d408b0} {Suffrage:Voter ID:server-ef1dab60-c52c-2f03-533a-49631e11d91c Address:ef1dab60-c52c-2f03-533a-49631e11d91c}]" 2025-07-29T00:57:11.827Z [INFO] server-ef1dab60-c52c-2f03-533a-49631e11d91c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5bbe77d-7984-0bc4-9834-62d46c204d83 Address:e5bbe77d-7984-0bc4-9834-62d46c204d83} {Suffrage:Voter ID:server-0e94a776-473f-7c37-e54a-f088e0d408b0 Address:0e94a776-473f-7c37-e54a-f088e0d408b0} {Suffrage:Voter ID:server-ef1dab60-c52c-2f03-533a-49631e11d91c Address:ef1dab60-c52c-2f03-533a-49631e11d91c}]" 2025-07-29T00:57:11.827Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:11.827Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: entering follower state: follower="Node at e5bbe77d-7984-0bc4-9834-62d46c204d83 [Follower]" leader-address= leader-id= 2025-07-29T00:57:11.827Z [INFO] server-0e94a776-473f-7c37-e54a-f088e0d408b0: entering follower state: follower="Node at 0e94a776-473f-7c37-e54a-f088e0d408b0 [Follower]" leader-address= leader-id= 2025-07-29T00:57:11.827Z [INFO] server-ef1dab60-c52c-2f03-533a-49631e11d91c: entering follower state: follower="Node at ef1dab60-c52c-2f03-533a-49631e11d91c [Follower]" leader-address= leader-id= 2025-07-29T00:57:11.935Z [WARN] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:11.935Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: entering candidate state: node="Node at e5bbe77d-7984-0bc4-9834-62d46c204d83 [Candidate]" term=2 2025-07-29T00:57:11.935Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: election won: term=2 tally=2 2025-07-29T00:57:11.935Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: entering leader state: leader="Node at e5bbe77d-7984-0bc4-9834-62d46c204d83 [Leader]" 2025-07-29T00:57:11.935Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: added peer, starting replication: peer=server-0e94a776-473f-7c37-e54a-f088e0d408b0 2025-07-29T00:57:11.935Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: added peer, starting replication: peer=server-ef1dab60-c52c-2f03-533a-49631e11d91c 2025-07-29T00:57:11.935Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: pipelining replication: peer="{Voter server-ef1dab60-c52c-2f03-533a-49631e11d91c ef1dab60-c52c-2f03-533a-49631e11d91c}" 2025-07-29T00:57:11.935Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: pipelining replication: peer="{Voter server-0e94a776-473f-7c37-e54a-f088e0d408b0 0e94a776-473f-7c37-e54a-f088e0d408b0}" 2025-07-29T00:57:12.141Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:11.935997977 +0000 UTC m=+111.625379281 (1 nodes), 108.9198ms from start of poll, 109.11952ms from cluster start. Timeout at 2025-07-29 00:57:12.141492537 +0000 UTC m=+111.830873881, 205.4946ms after stability 2025-07-29T00:57:12.141Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:12.347Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:12.141600177 +0000 UTC m=+111.830981521 (1 nodes), 2.64µs from start of poll, 314.72176ms from cluster start. Timeout at 2025-07-29 00:57:12.347810257 +0000 UTC m=+112.037191601, 206.21008ms after stability 2025-07-29T00:57:12.354Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:12.563Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:12.354321697 +0000 UTC m=+112.043703001 (2 nodes), 2.32µs from start of poll, 527.44324ms from cluster start. Timeout at 2025-07-29 00:57:12.563465217 +0000 UTC m=+112.252846561, 209.14356ms after stability 2025-07-29T00:57:12.563Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:12.769Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:12.563831137 +0000 UTC m=+112.253212481 (1 nodes), 880ns from start of poll, 736.95272ms from cluster start. Timeout at 2025-07-29 00:57:12.769230017 +0000 UTC m=+112.458611361, 205.39888ms after stability raft_test.go:2961: [INFO] restarting Node at 0e94a776-473f-7c37-e54a-f088e0d408b0 [Follower] 2025-07-29T00:57:12.769Z [INFO] server-0e94a776-473f-7c37-e54a-f088e0d408b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5bbe77d-7984-0bc4-9834-62d46c204d83 Address:e5bbe77d-7984-0bc4-9834-62d46c204d83} {Suffrage:Voter ID:server-0e94a776-473f-7c37-e54a-f088e0d408b0 Address:0e94a776-473f-7c37-e54a-f088e0d408b0} {Suffrage:Voter ID:server-ef1dab60-c52c-2f03-533a-49631e11d91c Address:ef1dab60-c52c-2f03-533a-49631e11d91c}]" 2025-07-29T00:57:12.769Z [INFO] server-0e94a776-473f-7c37-e54a-f088e0d408b0: entering follower state: follower="Node at 0e94a776-473f-7c37-e54a-f088e0d408b0 [Follower]" leader-address= leader-id= 2025-07-29T00:57:13.020Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: aborting pipeline replication: peer="{Voter server-ef1dab60-c52c-2f03-533a-49631e11d91c ef1dab60-c52c-2f03-533a-49631e11d91c}" 2025-07-29T00:57:13.407Z [ERROR] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: failed to pipeline appendEntries: peer="{Voter server-0e94a776-473f-7c37-e54a-f088e0d408b0 0e94a776-473f-7c37-e54a-f088e0d408b0}" error="command enqueue timeout" 2025-07-29T00:57:13.407Z [INFO] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: aborting pipeline replication: peer="{Voter server-0e94a776-473f-7c37-e54a-f088e0d408b0 0e94a776-473f-7c37-e54a-f088e0d408b0}" 2025-07-29T00:57:13.535Z [ERROR] server-e5bbe77d-7984-0bc4-9834-62d46c204d83: failed to heartbeat to: peer=0e94a776-473f-7c37-e54a-f088e0d408b0 backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.71s) === RUN TestRaft_VoteWithNoIDNoAddr 2025-07-29T00:57:13.536Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13b38e0d-d9dc-0ada-a832-55c205952ef9 Address:13b38e0d-d9dc-0ada-a832-55c205952ef9} {Suffrage:Voter ID:server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 Address:93d0ad24-6a2e-1ab2-7614-b3ca41fc8854} {Suffrage:Voter ID:server-a53383fc-d158-73ec-4b39-314d8b8e090a Address:a53383fc-d158-73ec-4b39-314d8b8e090a}]" 2025-07-29T00:57:13.536Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: entering follower state: follower="Node at 13b38e0d-d9dc-0ada-a832-55c205952ef9 [Follower]" leader-address= leader-id= 2025-07-29T00:57:13.536Z [INFO] server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13b38e0d-d9dc-0ada-a832-55c205952ef9 Address:13b38e0d-d9dc-0ada-a832-55c205952ef9} {Suffrage:Voter ID:server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 Address:93d0ad24-6a2e-1ab2-7614-b3ca41fc8854} {Suffrage:Voter ID:server-a53383fc-d158-73ec-4b39-314d8b8e090a Address:a53383fc-d158-73ec-4b39-314d8b8e090a}]" 2025-07-29T00:57:13.536Z [INFO] server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854: entering follower state: follower="Node at 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 [Follower]" leader-address= leader-id= 2025-07-29T00:57:13.536Z [INFO] server-a53383fc-d158-73ec-4b39-314d8b8e090a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13b38e0d-d9dc-0ada-a832-55c205952ef9 Address:13b38e0d-d9dc-0ada-a832-55c205952ef9} {Suffrage:Voter ID:server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 Address:93d0ad24-6a2e-1ab2-7614-b3ca41fc8854} {Suffrage:Voter ID:server-a53383fc-d158-73ec-4b39-314d8b8e090a Address:a53383fc-d158-73ec-4b39-314d8b8e090a}]" 2025-07-29T00:57:13.536Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:13.536Z [INFO] server-a53383fc-d158-73ec-4b39-314d8b8e090a: entering follower state: follower="Node at a53383fc-d158-73ec-4b39-314d8b8e090a [Follower]" leader-address= leader-id= 2025-07-29T00:57:13.599Z [WARN] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-07-29T00:57:13.599Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: entering candidate state: node="Node at 13b38e0d-d9dc-0ada-a832-55c205952ef9 [Candidate]" term=2 2025-07-29T00:57:13.599Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: election won: term=2 tally=2 2025-07-29T00:57:13.599Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: entering leader state: leader="Node at 13b38e0d-d9dc-0ada-a832-55c205952ef9 [Leader]" 2025-07-29T00:57:13.599Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: added peer, starting replication: peer=server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 2025-07-29T00:57:13.599Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: added peer, starting replication: peer=server-a53383fc-d158-73ec-4b39-314d8b8e090a 2025-07-29T00:57:13.599Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: pipelining replication: peer="{Voter server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854}" 2025-07-29T00:57:13.599Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: pipelining replication: peer="{Voter server-a53383fc-d158-73ec-4b39-314d8b8e090a a53383fc-d158-73ec-4b39-314d8b8e090a}" 2025-07-29T00:57:13.704Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:13.599694057 +0000 UTC m=+113.289075441 (1 nodes), 62.74712ms from start of poll, 63.20044ms from cluster start. Timeout at 2025-07-29 00:57:13.704857577 +0000 UTC m=+113.394238961, 105.16352ms after stability 2025-07-29T00:57:13.704Z [INFO] cluster: starting stability test: raft-state=Leader 2025-07-29T00:57:13.810Z [INFO] cluster: stable state for Leader reached at 2025-07-29 00:57:13.704983497 +0000 UTC m=+113.394364881 (1 nodes), 1.56µs from start of poll, 168.48988ms from cluster start. Timeout at 2025-07-29 00:57:13.810750697 +0000 UTC m=+113.500132041, 105.76716ms after stability 2025-07-29T00:57:13.816Z [INFO] cluster: starting stability test: raft-state=Follower 2025-07-29T00:57:13.922Z [INFO] cluster: stable state for Follower reached at 2025-07-29 00:57:13.816689577 +0000 UTC m=+113.506070921 (2 nodes), 3.12µs from start of poll, 280.19592ms from cluster start. Timeout at 2025-07-29 00:57:13.922021017 +0000 UTC m=+113.611402361, 105.33144ms after stability 2025-07-29T00:57:13.924Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to heartbeat to: peer=93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 backoff time=10ms error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.924Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to pipeline appendEntries: peer="{Voter server-a53383fc-d158-73ec-4b39-314d8b8e090a a53383fc-d158-73ec-4b39-314d8b8e090a}" error="append pipeline closed" 2025-07-29T00:57:13.924Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: aborting pipeline replication: peer="{Voter server-a53383fc-d158-73ec-4b39-314d8b8e090a a53383fc-d158-73ec-4b39-314d8b8e090a}" 2025-07-29T00:57:13.926Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to pipeline appendEntries: peer="{Voter server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854}" error="append pipeline closed" 2025-07-29T00:57:13.926Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: aborting pipeline replication: peer="{Voter server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854}" 2025-07-29T00:57:13.929Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to heartbeat to: peer=a53383fc-d158-73ec-4b39-314d8b8e090a backoff time=10ms error="failed to connect to peer: a53383fc-d158-73ec-4b39-314d8b8e090a" 2025-07-29T00:57:13.932Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to appendEntries to: peer="{Voter server-a53383fc-d158-73ec-4b39-314d8b8e090a a53383fc-d158-73ec-4b39-314d8b8e090a}" error="failed to connect to peer: a53383fc-d158-73ec-4b39-314d8b8e090a" 2025-07-29T00:57:13.932Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to appendEntries to: peer="{Voter server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854}" error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.943Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to heartbeat to: peer=93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 backoff time=10ms error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.947Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to heartbeat to: peer=a53383fc-d158-73ec-4b39-314d8b8e090a backoff time=10ms error="failed to connect to peer: a53383fc-d158-73ec-4b39-314d8b8e090a" 2025-07-29T00:57:13.948Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to appendEntries to: peer="{Voter server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854}" error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.952Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to appendEntries to: peer="{Voter server-a53383fc-d158-73ec-4b39-314d8b8e090a a53383fc-d158-73ec-4b39-314d8b8e090a}" error="failed to connect to peer: a53383fc-d158-73ec-4b39-314d8b8e090a" 2025-07-29T00:57:13.960Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to heartbeat to: peer=93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 backoff time=10ms error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.966Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to heartbeat to: peer=a53383fc-d158-73ec-4b39-314d8b8e090a backoff time=10ms error="failed to connect to peer: a53383fc-d158-73ec-4b39-314d8b8e090a" 2025-07-29T00:57:13.966Z [WARN] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to contact: server-id=server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 time=51.05336ms 2025-07-29T00:57:13.966Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to appendEntries to: peer="{Voter server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854}" error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.971Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to appendEntries to: peer="{Voter server-a53383fc-d158-73ec-4b39-314d8b8e090a a53383fc-d158-73ec-4b39-314d8b8e090a}" error="failed to connect to peer: a53383fc-d158-73ec-4b39-314d8b8e090a" 2025-07-29T00:57:13.977Z [WARN] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to contact: server-id=server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 time=61.2026ms 2025-07-29T00:57:13.977Z [WARN] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to contact: server-id=server-a53383fc-d158-73ec-4b39-314d8b8e090a time=54.87628ms 2025-07-29T00:57:13.977Z [WARN] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to contact quorum of nodes, stepping down 2025-07-29T00:57:13.977Z [INFO] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: entering follower state: follower="Node at 13b38e0d-d9dc-0ada-a832-55c205952ef9 [Follower]" leader-address= leader-id= 2025-07-29T00:57:13.979Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to heartbeat to: peer=93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 backoff time=20ms error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.995Z [ERROR] server-13b38e0d-d9dc-0ada-a832-55c205952ef9: failed to appendEntries to: peer="{Voter server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854}" error="failed to connect to peer: 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854" 2025-07-29T00:57:13.997Z [WARN] server-a53383fc-d158-73ec-4b39-314d8b8e090a: heartbeat timeout reached, starting election: last-leader-addr=13b38e0d-d9dc-0ada-a832-55c205952ef9 last-leader-id=server-13b38e0d-d9dc-0ada-a832-55c205952ef9 2025-07-29T00:57:13.997Z [INFO] server-a53383fc-d158-73ec-4b39-314d8b8e090a: entering candidate state: node="Node at a53383fc-d158-73ec-4b39-314d8b8e090a [Candidate]" term=3 2025-07-29T00:57:13.997Z [WARN] server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854: rejecting vote request since we have a leader: from=a53383fc-d158-73ec-4b39-314d8b8e090a leader=13b38e0d-d9dc-0ada-a832-55c205952ef9 leader-id=server-13b38e0d-d9dc-0ada-a832-55c205952ef9 2025-07-29T00:57:13.997Z [ERROR] server-a53383fc-d158-73ec-4b39-314d8b8e090a: failed to make requestVote RPC: target="{Voter server-13b38e0d-d9dc-0ada-a832-55c205952ef9 13b38e0d-d9dc-0ada-a832-55c205952ef9}" error="failed to connect to peer: 13b38e0d-d9dc-0ada-a832-55c205952ef9" term=3 2025-07-29T00:57:14.042Z [WARN] server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854: heartbeat timeout reached, starting election: last-leader-addr=13b38e0d-d9dc-0ada-a832-55c205952ef9 last-leader-id=server-13b38e0d-d9dc-0ada-a832-55c205952ef9 2025-07-29T00:57:14.042Z [INFO] server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854: entering candidate state: node="Node at 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 [Candidate]" term=3 2025-07-29T00:57:14.042Z [INFO] server-a53383fc-d158-73ec-4b39-314d8b8e090a: duplicate requestVote for same term: term=3 2025-07-29T00:57:14.042Z [ERROR] server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854: failed to make requestVote RPC: target="{Voter server-13b38e0d-d9dc-0ada-a832-55c205952ef9 13b38e0d-d9dc-0ada-a832-55c205952ef9}" error="failed to connect to peer: 13b38e0d-d9dc-0ada-a832-55c205952ef9" term=3 2025-07-29T00:57:14.043Z [INFO] server-93d0ad24-6a2e-1ab2-7614-b3ca41fc8854: entering follower state: follower="Node at 93d0ad24-6a2e-1ab2-7614-b3ca41fc8854 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.51s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2025-07-29T00:57:14.044Z [INFO] 127.0.0.1:43547 :: starting node: addr=127.0.0.1:43547 2025-07-29T00:57:14.044Z [INFO] 127.0.0.1:43547 :: initial configuration: index=0 servers=[] 2025-07-29T00:57:14.044Z [INFO] 127.0.0.1:43547 :: entering follower state: follower="Node at 127.0.0.1:43547 [Follower]" leader-address= leader-id= 2025-07-29T00:57:14.110Z [WARN] 127.0.0.1:43547 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2025-07-29T00:57:14.195Z [INFO] 127.0.0.1:36687 :: starting node: addr=127.0.0.1:36687 2025-07-29T00:57:14.195Z [INFO] 127.0.0.1:36687 :: initial configuration: index=0 servers=[] 2025-07-29T00:57:14.195Z [INFO] 127.0.0.1:36687 :: entering follower state: follower="Node at 127.0.0.1:36687 [Follower]" leader-address= leader-id= 2025-07-29T00:57:14.256Z [WARN] 127.0.0.1:36687 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2025-07-29T00:57:14.346Z [INFO] 127.0.0.1:45597 :: starting node: addr=127.0.0.1:45597 2025-07-29T00:57:14.346Z [INFO] 127.0.0.1:45597 :: initial configuration: index=0 servers=[] 2025-07-29T00:57:14.346Z [INFO] 127.0.0.1:45597 :: entering follower state: follower="Node at 127.0.0.1:45597 [Follower]" leader-address= leader-id= 2025-07-29T00:57:14.400Z [WARN] 127.0.0.1:45597 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.45s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2025-07-29T00:57:14.497Z [INFO] 127.0.0.1:33317 :: starting node: addr=127.0.0.1:33317 2025-07-29T00:57:14.497Z [INFO] 127.0.0.1:33317 :: initial configuration: index=0 servers=[] 2025-07-29T00:57:14.497Z [INFO] 127.0.0.1:33317 :: entering follower state: follower="Node at 127.0.0.1:33317 [Follower]" leader-address= leader-id= 2025-07-29T00:57:14.497Z [WARN] 127.0.0.1:33317 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS ok github.com/hashicorp/raft 114.359s make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/2332697 and its subdirectories I: Current time: Mon Jul 28 12:57:20 -12 2025 I: pbuilder-time-stamp: 1753750640