Mon Jan 8 00:28:36 UTC 2024 I: starting to build golang-github-hashicorp-raft/bookworm/i386 on jenkins on '2024-01-08 00:28' Mon Jan 8 00:28:36 UTC 2024 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/i386_6/10598/console.log Mon Jan 8 00:28:36 UTC 2024 I: Downloading source for bookworm/golang-github-hashicorp-raft=1.1.2-1 --2024-01-08 00:28:36-- http://cdn-fastly.deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.1.2-1.dsc Connecting to 78.137.99.97:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2408 (2.4K) [text/prs.lines.tag] Saving to: ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ 0K .. 100% 258M=0s 2024-01-08 00:28:36 (258 MB/s) - ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ saved [2408/2408] Mon Jan 8 00:28:36 UTC 2024 I: golang-github-hashicorp-raft_1.1.2-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.4.1 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) Package-List: golang-github-hashicorp-raft-dev deb devel optional arch=all Checksums-Sha1: 6eb88916d5b8568cc5e49dfd778e53f425f02612 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3efa08a30605d234327a7c1293b36ecde63ffbbe 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Checksums-Sha256: 6c3383ee430aa6b421cc4cb4cbc485b0e612bda2e94e88d416d13d65d375dc89 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3877bf052d5719266221189613502262d1be1af152c736d3dc8ed49734ad6dd8 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Files: 65844d5b08afa2d62ea4dedaf43e6047 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 036528358012213daa0588e86be2a3d4 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEULx8+TnSDCcqawZWUra72VOWjRsFAl4kBCwACgkQUra72VOW jRsPQBAAxzZChwaTUFMUIQz9GgdwyR5o2SjXw48JJ//lfygslStaiz8ju52lp33x a1d56obRwTy2QmY2s6+oV3Mq4sQWs7U/1PGnywKeA71Us10A4AAv5d9wMRq1eqzp PcyVdhfM0zFcmJf8Wr93Mi8wbCr9PT+oTxc59+zWDruS7xCoe3aDtaekVfb8QZyj OZPsaTtneSxKjqk+V5Jyx8X6Y6P2cwv89GDd4nxTxapw8H38e2cBk7BsYmXUNXye 6N7dOChDN+x6gqgBU0O97oex+QOpd4LNoFBl2gEi7kukw+jrjrdyKQCANPh0CKx0 K4MPq2BeI3qKJV21l2z7S30BSjduvyktC7efckw5CX4zgT9/kJ5e7ZAf/og+M2vc CoCV+aoAH1OOWbkPu0dsuJiA5z1ZkKGa8ZB5MgvUentk7LVd/HaXW9QDdRTNZT8z Jdh6/wnpSn8ZZEhAixTtqNt64s0sNWmh4krF0WgGNZ1djzYd6gVI1LqlJTaSRKFo nxGsEe+jnC6kv2u8XpLG+xevTKcu4UIFHq2ebpanFvxzqxAv1dWTBDDI13NApMZb Po5huhcdLapUW4NGW/3pkdJkpcY5u26zH6YvTxDICU/tHXDo5/BqI9E1jEj+5m5J XrNqpKLFjW2I9gRcmT5vy3O4edLCPBFs6MGQBK7u1V+nEkh8pJc= =IObb -----END PGP SIGNATURE----- Mon Jan 8 00:28:36 UTC 2024 I: Checking whether the package is not for us Mon Jan 8 00:28:36 UTC 2024 I: Starting 1st build on remote node ionos6-i386.debian.net. Mon Jan 8 00:28:36 UTC 2024 I: Preparing to do remote build '1' on ionos6-i386.debian.net. Mon Jan 8 00:29:45 UTC 2024 I: Deleting $TMPDIR on ionos6-i386.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Sat Feb 8 18:51:37 -12 2025 I: pbuilder-time-stamp: 1739083897 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bookworm-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: using eatmydata during job I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: Signature made Sun Jan 19 07:24:28 2020 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.1.2-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/849/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='i386' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=16 ' DISTRIBUTION='bookworm' HOME='/root' HOST_ARCH='i386' IFS=' ' INVOCATION_ID='bb79f038973c4466b89794a82c89bea8' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' LD_LIBRARY_PATH='/usr/lib/libeatmydata' LD_PRELOAD='libeatmydata.so' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='849' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.F1G7eTIx/pbuilderrc_Ko24 --distribution bookworm --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.F1G7eTIx/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID='112' SUDO_UID='107' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://85.184.249.68:3128' I: uname -a Linux ionos6-i386 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux I: ls -l /bin total 6036 -rwxr-xr-x 1 root root 1408088 Apr 23 2023 bash -rwxr-xr-x 3 root root 38404 Sep 19 2022 bunzip2 -rwxr-xr-x 3 root root 38404 Sep 19 2022 bzcat lrwxrwxrwx 1 root root 6 Sep 19 2022 bzcmp -> bzdiff -rwxr-xr-x 1 root root 2225 Sep 19 2022 bzdiff lrwxrwxrwx 1 root root 6 Sep 19 2022 bzegrep -> bzgrep -rwxr-xr-x 1 root root 4893 Nov 27 2021 bzexe lrwxrwxrwx 1 root root 6 Sep 19 2022 bzfgrep -> bzgrep -rwxr-xr-x 1 root root 3775 Sep 19 2022 bzgrep -rwxr-xr-x 3 root root 38404 Sep 19 2022 bzip2 -rwxr-xr-x 1 root root 17892 Sep 19 2022 bzip2recover lrwxrwxrwx 1 root root 6 Sep 19 2022 bzless -> bzmore -rwxr-xr-x 1 root root 1297 Sep 19 2022 bzmore -rwxr-xr-x 1 root root 42920 Sep 20 2022 cat -rwxr-xr-x 1 root root 79816 Sep 20 2022 chgrp -rwxr-xr-x 1 root root 67496 Sep 20 2022 chmod -rwxr-xr-x 1 root root 79816 Sep 20 2022 chown -rwxr-xr-x 1 root root 162024 Sep 20 2022 cp -rwxr-xr-x 1 root root 136916 Jan 5 2023 dash -rwxr-xr-x 1 root root 137160 Sep 20 2022 date -rwxr-xr-x 1 root root 100364 Sep 20 2022 dd -rwxr-xr-x 1 root root 108940 Sep 20 2022 df -rwxr-xr-x 1 root root 162152 Sep 20 2022 dir -rwxr-xr-x 1 root root 87760 Mar 23 2023 dmesg lrwxrwxrwx 1 root root 8 Dec 19 2022 dnsdomainname -> hostname lrwxrwxrwx 1 root root 8 Dec 19 2022 domainname -> hostname -rwxr-xr-x 1 root root 38760 Sep 20 2022 echo -rwxr-xr-x 1 root root 41 Jan 24 2023 egrep -rwxr-xr-x 1 root root 34664 Sep 20 2022 false -rwxr-xr-x 1 root root 41 Jan 24 2023 fgrep -rwxr-xr-x 1 root root 84272 Mar 23 2023 findmnt -rwsr-xr-x 1 root root 30240 Mar 23 2023 fusermount -rwxr-xr-x 1 root root 218680 Jan 24 2023 grep -rwxr-xr-x 2 root root 2346 Apr 10 2022 gunzip -rwxr-xr-x 1 root root 6447 Apr 10 2022 gzexe -rwxr-xr-x 1 root root 100952 Apr 10 2022 gzip -rwxr-xr-x 1 root root 21916 Dec 19 2022 hostname -rwxr-xr-x 1 root root 75756 Sep 20 2022 ln -rwxr-xr-x 1 root root 55600 Mar 23 2023 login -rwxr-xr-x 1 root root 162152 Sep 20 2022 ls -rwxr-xr-x 1 root root 214568 Mar 23 2023 lsblk -rwxr-xr-x 1 root root 96328 Sep 20 2022 mkdir -rwxr-xr-x 1 root root 84008 Sep 20 2022 mknod -rwxr-xr-x 1 root root 38792 Sep 20 2022 mktemp -rwxr-xr-x 1 root root 63016 Mar 23 2023 more -rwsr-xr-x 1 root root 58912 Mar 23 2023 mount -rwxr-xr-x 1 root root 13856 Mar 23 2023 mountpoint -rwxr-xr-x 1 root root 157932 Sep 20 2022 mv lrwxrwxrwx 1 root root 8 Dec 19 2022 nisdomainname -> hostname lrwxrwxrwx 1 root root 14 Apr 3 2023 pidof -> /sbin/killall5 -rwxr-xr-x 1 root root 38792 Sep 20 2022 pwd lrwxrwxrwx 1 root root 4 Apr 23 2023 rbash -> bash -rwxr-xr-x 1 root root 51080 Sep 20 2022 readlink -rwxr-xr-x 1 root root 75720 Sep 20 2022 rm -rwxr-xr-x 1 root root 51080 Sep 20 2022 rmdir -rwxr-xr-x 1 root root 22308 Jul 28 2023 run-parts -rwxr-xr-x 1 root root 133224 Jan 5 2023 sed lrwxrwxrwx 1 root root 4 Jan 5 2023 sh -> dash -rwxr-xr-x 1 root root 38760 Sep 20 2022 sleep -rwxr-xr-x 1 root root 87976 Sep 20 2022 stty -rwsr-xr-x 1 root root 83492 Mar 23 2023 su -rwxr-xr-x 1 root root 38792 Sep 20 2022 sync -rwxr-xr-x 1 root root 598456 Apr 6 2023 tar -rwxr-xr-x 1 root root 13860 Jul 28 2023 tempfile -rwxr-xr-x 1 root root 120776 Sep 20 2022 touch -rwxr-xr-x 1 root root 34664 Sep 20 2022 true -rwxr-xr-x 1 root root 17892 Mar 23 2023 ulockmgr_server -rwsr-xr-x 1 root root 30236 Mar 23 2023 umount -rwxr-xr-x 1 root root 38760 Sep 20 2022 uname -rwxr-xr-x 2 root root 2346 Apr 10 2022 uncompress -rwxr-xr-x 1 root root 162152 Sep 20 2022 vdir -rwxr-xr-x 1 root root 71216 Mar 23 2023 wdctl lrwxrwxrwx 1 root root 8 Dec 19 2022 ypdomainname -> hostname -rwxr-xr-x 1 root root 1984 Apr 10 2022 zcat -rwxr-xr-x 1 root root 1678 Apr 10 2022 zcmp -rwxr-xr-x 1 root root 6460 Apr 10 2022 zdiff -rwxr-xr-x 1 root root 29 Apr 10 2022 zegrep -rwxr-xr-x 1 root root 29 Apr 10 2022 zfgrep -rwxr-xr-x 1 root root 2081 Apr 10 2022 zforce -rwxr-xr-x 1 root root 8103 Apr 10 2022 zgrep -rwxr-xr-x 1 root root 2206 Apr 10 2022 zless -rwxr-xr-x 1 root root 1842 Apr 10 2022 zmore -rwxr-xr-x 1 root root 4577 Apr 10 2022 znew I: user script /srv/workspace/pbuilder/849/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: i386 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 18156 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.19-go{a} golang-1.19-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-3-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-diff-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 98 newly installed, 0 to remove and 0 not upgraded. Need to get 117 MB of archives. After unpacking 760 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bookworm/main i386 sensible-utils all 0.0.17+nmu1 [19.0 kB] Get: 2 http://deb.debian.org/debian bookworm/main i386 libmagic-mgc i386 1:5.44-3 [305 kB] Get: 3 http://deb.debian.org/debian bookworm/main i386 libmagic1 i386 1:5.44-3 [114 kB] Get: 4 http://deb.debian.org/debian bookworm/main i386 file i386 1:5.44-3 [42.5 kB] Get: 5 http://deb.debian.org/debian bookworm/main i386 gettext-base i386 0.21-12 [162 kB] Get: 6 http://deb.debian.org/debian bookworm/main i386 libuchardet0 i386 0.0.7-1 [67.9 kB] Get: 7 http://deb.debian.org/debian bookworm/main i386 groff-base i386 1.22.4-10 [932 kB] Get: 8 http://deb.debian.org/debian bookworm/main i386 bsdextrautils i386 2.38.1-5+b1 [90.3 kB] Get: 9 http://deb.debian.org/debian bookworm/main i386 libpipeline1 i386 1.5.7-1 [40.0 kB] Get: 10 http://deb.debian.org/debian bookworm/main i386 man-db i386 2.11.2-2 [1397 kB] Get: 11 http://deb.debian.org/debian bookworm/main i386 m4 i386 1.4.19-3 [294 kB] Get: 12 http://deb.debian.org/debian bookworm/main i386 autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian bookworm/main i386 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian bookworm/main i386 automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian bookworm/main i386 autopoint all 0.21-12 [495 kB] Get: 16 http://deb.debian.org/debian bookworm/main i386 libdebhelper-perl all 13.11.4 [81.2 kB] Get: 17 http://deb.debian.org/debian bookworm/main i386 libtool all 2.4.7-5 [517 kB] Get: 18 http://deb.debian.org/debian bookworm/main i386 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian bookworm/main i386 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian bookworm/main i386 libsub-override-perl all 0.09-4 [9304 B] Get: 21 http://deb.debian.org/debian bookworm/main i386 libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB] Get: 22 http://deb.debian.org/debian bookworm/main i386 dh-strip-nondeterminism all 1.13.1-1 [8620 B] Get: 23 http://deb.debian.org/debian bookworm/main i386 libelf1 i386 0.188-2.1 [179 kB] Get: 24 http://deb.debian.org/debian bookworm/main i386 dwz i386 0.15-1 [118 kB] Get: 25 http://deb.debian.org/debian bookworm/main i386 libicu72 i386 72.1-3 [9541 kB] Get: 26 http://deb.debian.org/debian bookworm/main i386 libxml2 i386 2.9.14+dfsg-1.3~deb12u1 [720 kB] Get: 27 http://deb.debian.org/debian bookworm/main i386 gettext i386 0.21-12 [1311 kB] Get: 28 http://deb.debian.org/debian bookworm/main i386 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian bookworm/main i386 po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian bookworm/main i386 debhelper all 13.11.4 [942 kB] Get: 31 http://deb.debian.org/debian bookworm/main i386 dh-golang all 1.59 [27.8 kB] Get: 32 http://deb.debian.org/debian bookworm/main i386 golang-1.19-src all 1.19.8-2 [18.3 MB] Get: 33 http://deb.debian.org/debian bookworm/main i386 golang-1.19-go i386 1.19.8-2 [61.2 MB] Get: 34 http://deb.debian.org/debian bookworm/main i386 golang-src all 2:1.19~1 [5756 B] Get: 35 http://deb.debian.org/debian bookworm/main i386 golang-go i386 2:1.19~1 [45.0 kB] Get: 36 http://deb.debian.org/debian bookworm/main i386 golang-any i386 2:1.19~1 [5872 B] Get: 37 http://deb.debian.org/debian bookworm/main i386 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 38 http://deb.debian.org/debian bookworm/main i386 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian bookworm/main i386 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian bookworm/main i386 golang-golang-x-sys-dev all 0.3.0-1 [393 kB] Get: 41 http://deb.debian.org/debian bookworm/main i386 golang-github-mattn-go-isatty-dev all 0.0.17-1 [6016 B] Get: 42 http://deb.debian.org/debian bookworm/main i386 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian bookworm/main i386 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 44 http://deb.debian.org/debian bookworm/main i386 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian bookworm/main i386 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian bookworm/main i386 golang-github-stretchr-objx-dev all 0.5.0-1 [25.5 kB] Get: 47 http://deb.debian.org/debian bookworm/main i386 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian bookworm/main i386 golang-github-stretchr-testify-dev all 1.8.1-1 [67.9 kB] Get: 49 http://deb.debian.org/debian bookworm/main i386 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian bookworm/main i386 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian bookworm/main i386 golang-github-pkg-errors-dev all 0.9.1-2 [13.2 kB] Get: 52 http://deb.debian.org/debian bookworm/main i386 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian bookworm/main i386 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian bookworm/main i386 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian bookworm/main i386 golang-github-hashicorp-go-uuid-dev all 1.0.2-2 [8480 B] Get: 56 http://deb.debian.org/debian bookworm/main i386 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 57 http://deb.debian.org/debian bookworm/main i386 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 58 http://deb.debian.org/debian bookworm/main i386 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 59 http://deb.debian.org/debian bookworm/main i386 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 60 http://deb.debian.org/debian bookworm/main i386 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 61 http://deb.debian.org/debian bookworm/main i386 golang-github-golang-protobuf-1-3-dev all 1.3.5-4 [234 kB] Get: 62 http://deb.debian.org/debian bookworm/main i386 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 63 http://deb.debian.org/debian bookworm/main i386 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 64 http://deb.debian.org/debian bookworm/main i386 golang-github-json-iterator-go-dev all 1.1.12-1 [62.2 kB] Get: 65 http://deb.debian.org/debian bookworm/main i386 golang-github-prometheus-client-model-dev all 0.3.0-3 [10.5 kB] Get: 66 http://deb.debian.org/debian bookworm/main i386 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 67 http://deb.debian.org/debian bookworm/main i386 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 68 http://deb.debian.org/debian bookworm/main i386 golang-golang-x-text-dev all 0.7.0-1 [3863 kB] Get: 69 http://deb.debian.org/debian bookworm/main i386 golang-golang-x-net-dev all 1:0.7.0+dfsg-1 [731 kB] Get: 70 http://deb.debian.org/debian bookworm/main i386 golang-github-aws-aws-sdk-go-dev all 1.44.133-1 [10.5 MB] Get: 71 http://deb.debian.org/debian bookworm/main i386 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 72 http://deb.debian.org/debian bookworm/main i386 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 73 http://deb.debian.org/debian bookworm/main i386 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 74 http://deb.debian.org/debian bookworm/main i386 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 75 http://deb.debian.org/debian bookworm/main i386 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 76 http://deb.debian.org/debian bookworm/main i386 golang-golang-x-oauth2-dev all 0.3.0-1 [35.0 kB] Get: 77 http://deb.debian.org/debian bookworm/main i386 golang-google-protobuf-dev all 1.28.1-3 [542 kB] Get: 78 http://deb.debian.org/debian bookworm/main i386 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 79 http://deb.debian.org/debian bookworm/main i386 golang-protobuf-extensions-dev all 1.0.1-3 [29.5 kB] Get: 80 http://deb.debian.org/debian bookworm/main i386 golang-github-prometheus-common-dev all 0.39.0-2 [110 kB] Get: 81 http://deb.debian.org/debian bookworm/main i386 golang-golang-x-sync-dev all 0.1.0-1 [19.7 kB] Get: 82 http://deb.debian.org/debian bookworm/main i386 golang-github-prometheus-procfs-dev all 0.8.0-3 [148 kB] Get: 83 http://deb.debian.org/debian bookworm/main i386 golang-github-prometheus-client-golang-dev all 1.14.0-3 [174 kB] Get: 84 http://deb.debian.org/debian bookworm/main i386 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 85 http://deb.debian.org/debian bookworm/main i386 golang-github-creack-pty-dev all 1.1.18-1 [9796 B] Get: 86 http://deb.debian.org/debian bookworm/main i386 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 87 http://deb.debian.org/debian bookworm/main i386 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 88 http://deb.debian.org/debian bookworm/main i386 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 89 http://deb.debian.org/debian bookworm/main i386 libsasl2-modules-db i386 2.1.28+dfsg-10 [21.4 kB] Get: 90 http://deb.debian.org/debian bookworm/main i386 libsasl2-2 i386 2.1.28+dfsg-10 [62.7 kB] Get: 91 http://deb.debian.org/debian bookworm/main i386 libsasl2-dev i386 2.1.28+dfsg-10 [248 kB] Get: 92 http://deb.debian.org/debian bookworm/main i386 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 93 http://deb.debian.org/debian bookworm/main i386 golang-github-pkg-diff-dev all 0.0~git20210226.20ebb0f-1 [93.6 kB] Get: 94 http://deb.debian.org/debian bookworm/main i386 golang-github-rogpeppe-go-internal-dev all 1.9.0-1 [117 kB] Get: 95 http://deb.debian.org/debian bookworm/main i386 golang-github-kr-pretty-dev all 0.3.0-1 [14.9 kB] Get: 96 http://deb.debian.org/debian bookworm/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 97 http://deb.debian.org/debian bookworm/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 98 http://deb.debian.org/debian bookworm/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 117 MB in 1s (92.3 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 18156 files and directories currently installed.) Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ... Unpacking sensible-utils (0.0.17+nmu1) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../01-libmagic-mgc_1%3a5.44-3_i386.deb ... Unpacking libmagic-mgc (1:5.44-3) ... Selecting previously unselected package libmagic1:i386. Preparing to unpack .../02-libmagic1_1%3a5.44-3_i386.deb ... Unpacking libmagic1:i386 (1:5.44-3) ... Selecting previously unselected package file. Preparing to unpack .../03-file_1%3a5.44-3_i386.deb ... Unpacking file (1:5.44-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../04-gettext-base_0.21-12_i386.deb ... Unpacking gettext-base (0.21-12) ... Selecting previously unselected package libuchardet0:i386. Preparing to unpack .../05-libuchardet0_0.0.7-1_i386.deb ... Unpacking libuchardet0:i386 (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../06-groff-base_1.22.4-10_i386.deb ... Unpacking groff-base (1.22.4-10) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../07-bsdextrautils_2.38.1-5+b1_i386.deb ... Unpacking bsdextrautils (2.38.1-5+b1) ... Selecting previously unselected package libpipeline1:i386. Preparing to unpack .../08-libpipeline1_1.5.7-1_i386.deb ... Unpacking libpipeline1:i386 (1.5.7-1) ... Selecting previously unselected package man-db. Preparing to unpack .../09-man-db_2.11.2-2_i386.deb ... Unpacking man-db (2.11.2-2) ... Selecting previously unselected package m4. Preparing to unpack .../10-m4_1.4.19-3_i386.deb ... Unpacking m4 (1.4.19-3) ... Selecting previously unselected package autoconf. Preparing to unpack .../11-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../12-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../13-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../14-autopoint_0.21-12_all.deb ... Unpacking autopoint (0.21-12) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../15-libdebhelper-perl_13.11.4_all.deb ... Unpacking libdebhelper-perl (13.11.4) ... Selecting previously unselected package libtool. Preparing to unpack .../16-libtool_2.4.7-5_all.deb ... Unpacking libtool (2.4.7-5) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../17-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../18-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../19-libsub-override-perl_0.09-4_all.deb ... Unpacking libsub-override-perl (0.09-4) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../20-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../21-dh-strip-nondeterminism_1.13.1-1_all.deb ... Unpacking dh-strip-nondeterminism (1.13.1-1) ... Selecting previously unselected package libelf1:i386. Preparing to unpack .../22-libelf1_0.188-2.1_i386.deb ... Unpacking libelf1:i386 (0.188-2.1) ... Selecting previously unselected package dwz. Preparing to unpack .../23-dwz_0.15-1_i386.deb ... Unpacking dwz (0.15-1) ... Selecting previously unselected package libicu72:i386. Preparing to unpack .../24-libicu72_72.1-3_i386.deb ... Unpacking libicu72:i386 (72.1-3) ... Selecting previously unselected package libxml2:i386. Preparing to unpack .../25-libxml2_2.9.14+dfsg-1.3~deb12u1_i386.deb ... Unpacking libxml2:i386 (2.9.14+dfsg-1.3~deb12u1) ... Selecting previously unselected package gettext. Preparing to unpack .../26-gettext_0.21-12_i386.deb ... Unpacking gettext (0.21-12) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../27-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../28-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../29-debhelper_13.11.4_all.deb ... Unpacking debhelper (13.11.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../30-dh-golang_1.59_all.deb ... Unpacking dh-golang (1.59) ... Selecting previously unselected package golang-1.19-src. Preparing to unpack .../31-golang-1.19-src_1.19.8-2_all.deb ... Unpacking golang-1.19-src (1.19.8-2) ... Selecting previously unselected package golang-1.19-go. Preparing to unpack .../32-golang-1.19-go_1.19.8-2_i386.deb ... Unpacking golang-1.19-go (1.19.8-2) ... Selecting previously unselected package golang-src. Preparing to unpack .../33-golang-src_2%3a1.19~1_all.deb ... Unpacking golang-src (2:1.19~1) ... Selecting previously unselected package golang-go:i386. Preparing to unpack .../34-golang-go_2%3a1.19~1_i386.deb ... Unpacking golang-go:i386 (2:1.19~1) ... Selecting previously unselected package golang-any:i386. Preparing to unpack .../35-golang-any_2%3a1.19~1_i386.deb ... Unpacking golang-any:i386 (2:1.19~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../36-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../37-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../38-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../39-golang-golang-x-sys-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.3.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../40-golang-github-mattn-go-isatty-dev_0.0.17-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.17-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../41-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../42-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../43-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../44-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../45-golang-github-stretchr-objx-dev_0.5.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../46-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../47-golang-github-stretchr-testify-dev_1.8.1-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.8.1-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../48-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../49-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../50-golang-github-pkg-errors-dev_0.9.1-2_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-2) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../51-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../52-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../53-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../54-golang-github-hashicorp-go-uuid-dev_1.0.2-2_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../55-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../56-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../57-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../58-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../59-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-golang-protobuf-1-3-dev. Preparing to unpack .../60-golang-github-golang-protobuf-1-3-dev_1.3.5-4_all.deb ... Unpacking golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../61-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../62-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../63-golang-github-json-iterator-go-dev_1.1.12-1_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-1) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../64-golang-github-prometheus-client-model-dev_0.3.0-3_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.3.0-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../65-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../66-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../67-golang-golang-x-text-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.7.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../68-golang-golang-x-net-dev_1%3a0.7.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../69-golang-github-aws-aws-sdk-go-dev_1.44.133-1_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../70-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../71-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../72-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../73-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../74-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../75-golang-golang-x-oauth2-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.3.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../76-golang-google-protobuf-dev_1.28.1-3_all.deb ... Unpacking golang-google-protobuf-dev (1.28.1-3) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../77-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../78-golang-protobuf-extensions-dev_1.0.1-3_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-3) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../79-golang-github-prometheus-common-dev_0.39.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.39.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../80-golang-golang-x-sync-dev_0.1.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.1.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../81-golang-github-prometheus-procfs-dev_0.8.0-3_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.8.0-3) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../82-golang-github-prometheus-client-golang-dev_1.14.0-3_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.14.0-3) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../83-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../84-golang-github-creack-pty-dev_1.1.18-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.18-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../85-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../86-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../87-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:i386. Preparing to unpack .../88-libsasl2-modules-db_2.1.28+dfsg-10_i386.deb ... Unpacking libsasl2-modules-db:i386 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-2:i386. Preparing to unpack .../89-libsasl2-2_2.1.28+dfsg-10_i386.deb ... Unpacking libsasl2-2:i386 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../90-libsasl2-dev_2.1.28+dfsg-10_i386.deb ... Unpacking libsasl2-dev (2.1.28+dfsg-10) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../91-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-github-pkg-diff-dev. Preparing to unpack .../92-golang-github-pkg-diff-dev_0.0~git20210226.20ebb0f-1_all.deb ... Unpacking golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../93-golang-github-rogpeppe-go-internal-dev_1.9.0-1_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../94-golang-github-kr-pretty-dev_0.3.0-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../95-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../96-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../97-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:i386 (1.5.7-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Setting up libicu72:i386 (72.1-3) ... Setting up bsdextrautils (2.38.1-5+b1) ... Setting up golang-github-creack-pty-dev (1.1.18-1) ... Setting up golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Setting up golang-github-stretchr-objx-dev (0.5.0-1) ... Setting up libmagic-mgc (1:5.44-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-2) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.11.4) ... Setting up golang-golang-x-sys-dev (0.3.0-1) ... Setting up libmagic1:i386 (1:5.44-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-12) ... Setting up golang-1.19-src (1.19.8-2) ... Setting up m4 (1.4.19-3) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Setting up file (1:5.44-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:i386 (2.1.28+dfsg-10) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.28.1-3) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.8.1-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-golang-x-sync-dev (0.1.0-1) ... Setting up autopoint (0.21-12) ... Setting up libsasl2-2:i386 (2.1.28+dfsg-10) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up sensible-utils (0.0.17+nmu1) ... Setting up golang-golang-x-text-dev (0.7.0-1) ... Setting up libuchardet0:i386 (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.8.0-3) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up libsub-override-perl (0.09-4) ... Setting up golang-github-prometheus-client-model-dev (0.3.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Setting up golang-src (2:1.19~1) ... Setting up libelf1:i386 (0.188-2.1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:i386 (2.9.14+dfsg-1.3~deb12u1) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up golang-1.19-go (1.19.8-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-github-kr-pretty-dev (0.3.0-1) ... Setting up libfile-stripnondeterminism-perl (1.13.1-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.17-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up gettext (0.21-12) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-5) ... Setting up golang-go:i386 (2:1.19~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-protobuf-extensions-dev (1.0.1-3) ... Setting up golang-github-json-iterator-go-dev (1.1.12-1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:i386 (2:1.19~1) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg-10) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.13.1-1) ... Setting up dwz (0.15-1) ... Setting up groff-base (1.22.4-10) ... Setting up golang-golang-x-oauth2-dev (0.3.0-1) ... Setting up golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up man-db (2.11.2-2) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.11.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.59) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.39.0-2) ... Setting up golang-github-prometheus-client-golang-dev (1.14.0-3) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Processing triggers for libc-bin (2.36-9+deb12u3) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.31-1.2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture i386 fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-i686-linux-gnu && go install -trimpath -v -p 16 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/goarch internal/unsafeheader unicode/utf8 internal/goos container/list internal/goexperiment internal/itoa runtime/internal/atomic crypto/subtle runtime/internal/syscall internal/cpu internal/race unicode crypto/internal/subtle math/bits sync/atomic crypto/internal/boring/sig encoding internal/abi runtime/internal/math unicode/utf16 runtime/internal/sys internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle math internal/bytealg runtime internal/reflectlite sync internal/testlog internal/sysinfo internal/singleflight math/rand runtime/cgo errors sort internal/oserror io github.com/hashicorp/golang-lru/simplelru internal/safefilepath vendor/golang.org/x/net/dns/dnsmessage path strconv crypto/internal/nistec/fiat syscall hash crypto/internal/randutil bytes strings hash/crc32 hash/crc64 crypto crypto/rc4 reflect vendor/golang.org/x/text/transform bufio net/http/internal/ascii github.com/hashicorp/go-immutable-radix regexp/syntax html internal/syscall/execenv internal/syscall/unix time regexp crypto/internal/nistec context io/fs internal/poll internal/fmtsort encoding/binary os encoding/base64 crypto/cipher crypto/internal/edwards25519/field vendor/golang.org/x/crypto/internal/poly1305 crypto/md5 vendor/golang.org/x/crypto/curve25519/internal/field golang.org/x/sys/unix encoding/pem crypto/internal/edwards25519 crypto/internal/boring crypto/des vendor/golang.org/x/crypto/chacha20 crypto/hmac crypto/sha512 crypto/aes crypto/sha1 crypto/sha256 internal/godebug path/filepath fmt os/signal io/ioutil internal/intern vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf net/netip net log encoding/hex vendor/golang.org/x/crypto/curve25519 net/http/internal net/url encoding/json compress/flate mime vendor/golang.org/x/net/http2/hpack go/token mime/quotedprintable encoding/gob math/big vendor/golang.org/x/text/unicode/norm text/template/parse flag runtime/debug vendor/golang.org/x/text/unicode/bidi runtime/trace compress/gzip testing vendor/golang.org/x/text/secure/bidirule github.com/mattn/go-isatty text/template github.com/mattn/go-colorable github.com/fatih/color github.com/hashicorp/go-hclog vendor/golang.org/x/net/idna crypto/internal/boring/bbig crypto/elliptic crypto/rand crypto/dsa encoding/asn1 html/template crypto/ed25519 crypto/rsa crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-i686-linux-gnu && go test -vet=off -v -p 16 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2025-02-09T06:52:19.024Z [INFO] creating new snapshot: path=/tmp/raft2354081483/raft3054436695/snapshots/3-10-1739083939024.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2025-02-09T06:52:19.027Z [INFO] creating new snapshot: path=/tmp/raft3098021935/snapshots/3-10-1739083939027.tmp testing.go:153: 2025-02-09T06:52:19.029Z [WARN] found temporary snapshot: name=3-10-1739083939027.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2025-02-09T06:52:19.031Z [INFO] creating new snapshot: path=/tmp/raft1571085551/snapshots/3-10-1739083939031.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention testing.go:153: 2025-02-09T06:52:19.033Z [INFO] creating new snapshot: path=/tmp/raft2011223678/snapshots/3-10-1739083939033.tmp testing.go:153: 2025-02-09T06:52:19.036Z [INFO] creating new snapshot: path=/tmp/raft2011223678/snapshots/3-11-1739083939036.tmp testing.go:153: 2025-02-09T06:52:19.039Z [INFO] creating new snapshot: path=/tmp/raft2011223678/snapshots/3-12-1739083939039.tmp testing.go:153: 2025-02-09T06:52:19.042Z [INFO] reaping snapshot: path=/tmp/raft2011223678/snapshots/3-10-1739083939033 testing.go:153: 2025-02-09T06:52:19.042Z [INFO] creating new snapshot: path=/tmp/raft2011223678/snapshots/3-13-1739083939042.tmp testing.go:153: 2025-02-09T06:52:19.045Z [INFO] reaping snapshot: path=/tmp/raft2011223678/snapshots/3-11-1739083939036 testing.go:153: 2025-02-09T06:52:19.045Z [INFO] creating new snapshot: path=/tmp/raft2011223678/snapshots/3-14-1739083939045.tmp testing.go:153: 2025-02-09T06:52:19.048Z [INFO] reaping snapshot: path=/tmp/raft2011223678/snapshots/3-12-1739083939039 --- PASS: TestFileSS_Retention (0.02s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2025-02-09T06:52:19.049Z [INFO] creating new snapshot: path=/tmp/raft3649208096/snapshots/5-130350-1739083939049.tmp testing.go:153: 2025-02-09T06:52:19.052Z [INFO] creating new snapshot: path=/tmp/raft3649208096/snapshots/36-204917-1739083939052.tmp --- PASS: TestFileSS_Ordering (0.01s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.073Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.078Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.088Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.108Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.149Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.229Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.389Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:19.710Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:20.350Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:21.355Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2025-02-09T06:52:22.355Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-30f4cbd2-56a4-ef70-b693-1c18a9118dea: 2025-02-09T06:52:23.077Z [INFO] server-30f4cbd2-56a4-ef70-b693-1c18a9118dea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30f4cbd2-56a4-ef70-b693-1c18a9118dea Address:30f4cbd2-56a4-ef70-b693-1c18a9118dea}]" testing.go:149: server-30f4cbd2-56a4-ef70-b693-1c18a9118dea: 2025-02-09T06:52:23.077Z [INFO] server-30f4cbd2-56a4-ef70-b693-1c18a9118dea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30f4cbd2-56a4-ef70-b693-1c18a9118dea Address:30f4cbd2-56a4-ef70-b693-1c18a9118dea}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-576fcbf2-f959-d3e1-c712-8860ff5d9db4: 2025-02-09T06:52:23.077Z [INFO] server-576fcbf2-f959-d3e1-c712-8860ff5d9db4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-576fcbf2-f959-d3e1-c712-8860ff5d9db4 Address:576fcbf2-f959-d3e1-c712-8860ff5d9db4}]" testing.go:149: server-576fcbf2-f959-d3e1-c712-8860ff5d9db4: 2025-02-09T06:52:23.077Z [INFO] server-576fcbf2-f959-d3e1-c712-8860ff5d9db4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-576fcbf2-f959-d3e1-c712-8860ff5d9db4 Address:576fcbf2-f959-d3e1-c712-8860ff5d9db4}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.078Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: initial configuration: index=0 servers=[] testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.078Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.078Z [INFO] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: initial configuration: index=0 servers=[] testing.go:149: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.078Z [INFO] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.078Z [INFO] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.078Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: entering follower state: follower="Node at cbc143bc-f7b2-63e9-78b9-2d875509c874 [Follower]" leader= testLoggerAdapter verbose: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.078Z [INFO] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: entering follower state: follower="Node at eb255dfb-b4e4-1c0d-8958-0b5f42c242cf [Follower]" leader= testing.go:149: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.078Z [INFO] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: initial configuration: index=0 servers=[] testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.078Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: entering follower state: follower="Node at cbc143bc-f7b2-63e9-78b9-2d875509c874 [Follower]" leader= testLoggerAdapter verbose: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.078Z [INFO] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: entering follower state: follower="Node at 1fb07c3f-e017-89bb-8d6b-bc1027e97eec [Follower]" leader= testing.go:149: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.078Z [INFO] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: entering follower state: follower="Node at eb255dfb-b4e4-1c0d-8958-0b5f42c242cf [Follower]" leader= testing.go:149: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.078Z [INFO] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: entering follower state: follower="Node at 1fb07c3f-e017-89bb-8d6b-bc1027e97eec [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.079Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:23.079Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.129Z [WARN] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: no known peers, aborting election testing.go:149: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.129Z [WARN] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: no known peers, aborting election testLoggerAdapter verbose: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.140Z [WARN] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: no known peers, aborting election testing.go:149: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.140Z [WARN] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: no known peers, aborting election testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [WARN] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [WARN] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: entering candidate state: node="Node at cbc143bc-f7b2-63e9-78b9-2d875509c874 [Candidate]" term=2 testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: entering candidate state: node="Node at cbc143bc-f7b2-63e9-78b9-2d875509c874 [Candidate]" term=2 testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: election won: tally=2 testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: election won: tally=2 testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: entering leader state: leader="Node at cbc143bc-f7b2-63e9-78b9-2d875509c874 [Leader]" testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: entering leader state: leader="Node at cbc143bc-f7b2-63e9-78b9-2d875509c874 [Leader]" testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: added peer, starting replication: peer=server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: added peer, starting replication: peer=server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: added peer, starting replication: peer=server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.158Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: added peer, starting replication: peer=server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec testLoggerAdapter verbose: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.158Z [WARN] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: 2025-02-09T06:52:23.158Z [WARN] server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [WARN] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: appendEntries rejected, sending older logs: peer="{Voter server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec 1fb07c3f-e017-89bb-8d6b-bc1027e97eec}" next=1 testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [WARN] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: appendEntries rejected, sending older logs: peer="{Voter server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec 1fb07c3f-e017-89bb-8d6b-bc1027e97eec}" next=1 testLoggerAdapter verbose: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.158Z [WARN] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: 2025-02-09T06:52:23.158Z [WARN] server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: pipelining replication: peer="{Voter server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec 1fb07c3f-e017-89bb-8d6b-bc1027e97eec}" testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: pipelining replication: peer="{Voter server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec 1fb07c3f-e017-89bb-8d6b-bc1027e97eec}" testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [WARN] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: appendEntries rejected, sending older logs: peer="{Voter server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf eb255dfb-b4e4-1c0d-8958-0b5f42c242cf}" next=1 testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [WARN] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: appendEntries rejected, sending older logs: peer="{Voter server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf eb255dfb-b4e4-1c0d-8958-0b5f42c242cf}" next=1 testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: pipelining replication: peer="{Voter server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf eb255dfb-b4e4-1c0d-8958-0b5f42c242cf}" testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.159Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: pipelining replication: peer="{Voter server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf eb255dfb-b4e4-1c0d-8958-0b5f42c242cf}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.263Z [INFO] stable state for Follower reached at 2025-02-09 06:52:23.158854082 +0000 UTC m=+4.137662847 (2 nodes), 79.67914ms from start of poll, 80.319386ms from cluster start. Timeout at 2025-02-09 06:52:23.26395573 +0000 UTC m=+4.242764488, 105.101641ms after stability testing.go:149: cluster: 2025-02-09T06:52:23.263Z [INFO] stable state for Follower reached at 2025-02-09 06:52:23.158854082 +0000 UTC m=+4.137662847 (2 nodes), 79.67914ms from start of poll, 80.319386ms from cluster start. Timeout at 2025-02-09 06:52:23.26395573 +0000 UTC m=+4.242764488, 105.101641ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.264Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:23.264Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.369Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.264039303 +0000 UTC m=+4.242848066 (1 nodes), 834ns from start of poll, 185.504605ms from cluster start. Timeout at 2025-02-09 06:52:23.369226607 +0000 UTC m=+4.348035364, 105.187298ms after stability testing.go:149: cluster: 2025-02-09T06:52:23.369Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.264039303 +0000 UTC m=+4.242848066 (1 nodes), 834ns from start of poll, 185.504605ms from cluster start. Timeout at 2025-02-09 06:52:23.369226607 +0000 UTC m=+4.348035364, 105.187298ms after stability testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.380Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: aborting pipeline replication: peer="{Voter server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec 1fb07c3f-e017-89bb-8d6b-bc1027e97eec}" testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.380Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: aborting pipeline replication: peer="{Voter server-1fb07c3f-e017-89bb-8d6b-bc1027e97eec 1fb07c3f-e017-89bb-8d6b-bc1027e97eec}" testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.380Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: aborting pipeline replication: peer="{Voter server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf eb255dfb-b4e4-1c0d-8958-0b5f42c242cf}" testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.380Z [INFO] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: aborting pipeline replication: peer="{Voter server-eb255dfb-b4e4-1c0d-8958-0b5f42c242cf eb255dfb-b4e4-1c0d-8958-0b5f42c242cf}" testLoggerAdapter verbose: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.430Z [ERROR] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: failed to heartbeat to: peer=1fb07c3f-e017-89bb-8d6b-bc1027e97eec error="command timed out" testing.go:149: server-cbc143bc-f7b2-63e9-78b9-2d875509c874: 2025-02-09T06:52:23.430Z [ERROR] server-cbc143bc-f7b2-63e9-78b9-2d875509c874: failed to heartbeat to: peer=1fb07c3f-e017-89bb-8d6b-bc1027e97eec error="command timed out" --- PASS: TestRaft_LiveBootstrap (0.35s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-ec4ff957-b81b-9581-93be-6df83e75d18d: 2025-02-09T06:52:23.431Z [INFO] server-ec4ff957-b81b-9581-93be-6df83e75d18d: initial configuration: index=0 servers=[] testing.go:149: server-ec4ff957-b81b-9581-93be-6df83e75d18d: 2025-02-09T06:52:23.431Z [INFO] server-ec4ff957-b81b-9581-93be-6df83e75d18d: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-ec4ff957-b81b-9581-93be-6df83e75d18d: 2025-02-09T06:52:23.431Z [INFO] server-ec4ff957-b81b-9581-93be-6df83e75d18d: entering follower state: follower="Node at ec4ff957-b81b-9581-93be-6df83e75d18d [Shutdown]" leader= testing.go:149: server-ec4ff957-b81b-9581-93be-6df83e75d18d: 2025-02-09T06:52:23.431Z [INFO] server-ec4ff957-b81b-9581-93be-6df83e75d18d: entering follower state: follower="Node at ec4ff957-b81b-9581-93be-6df83e75d18d [Shutdown]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.432Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cfcbcd0-981b-e060-8e39-69dec5f5017f Address:4cfcbcd0-981b-e060-8e39-69dec5f5017f} {Suffrage:Voter ID:server-83070580-0ca6-9a61-3d95-daeb3820ce41 Address:83070580-0ca6-9a61-3d95-daeb3820ce41}]" testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.432Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cfcbcd0-981b-e060-8e39-69dec5f5017f Address:4cfcbcd0-981b-e060-8e39-69dec5f5017f} {Suffrage:Voter ID:server-83070580-0ca6-9a61-3d95-daeb3820ce41 Address:83070580-0ca6-9a61-3d95-daeb3820ce41}]" testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.432Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: entering follower state: follower="Node at 4cfcbcd0-981b-e060-8e39-69dec5f5017f [Follower]" leader= testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.432Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: entering follower state: follower="Node at 4cfcbcd0-981b-e060-8e39-69dec5f5017f [Follower]" leader= testLoggerAdapter verbose: server-83070580-0ca6-9a61-3d95-daeb3820ce41: 2025-02-09T06:52:23.432Z [INFO] server-83070580-0ca6-9a61-3d95-daeb3820ce41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cfcbcd0-981b-e060-8e39-69dec5f5017f Address:4cfcbcd0-981b-e060-8e39-69dec5f5017f} {Suffrage:Voter ID:server-83070580-0ca6-9a61-3d95-daeb3820ce41 Address:83070580-0ca6-9a61-3d95-daeb3820ce41}]" testing.go:149: server-83070580-0ca6-9a61-3d95-daeb3820ce41: 2025-02-09T06:52:23.432Z [INFO] server-83070580-0ca6-9a61-3d95-daeb3820ce41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cfcbcd0-981b-e060-8e39-69dec5f5017f Address:4cfcbcd0-981b-e060-8e39-69dec5f5017f} {Suffrage:Voter ID:server-83070580-0ca6-9a61-3d95-daeb3820ce41 Address:83070580-0ca6-9a61-3d95-daeb3820ce41}]" testLoggerAdapter verbose: server-83070580-0ca6-9a61-3d95-daeb3820ce41: 2025-02-09T06:52:23.432Z [INFO] server-83070580-0ca6-9a61-3d95-daeb3820ce41: entering follower state: follower="Node at 83070580-0ca6-9a61-3d95-daeb3820ce41 [Follower]" leader= testing.go:149: server-83070580-0ca6-9a61-3d95-daeb3820ce41: 2025-02-09T06:52:23.432Z [INFO] server-83070580-0ca6-9a61-3d95-daeb3820ce41: entering follower state: follower="Node at 83070580-0ca6-9a61-3d95-daeb3820ce41 [Follower]" leader= testLoggerAdapter verbose: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.432Z [INFO] server-09135aeb-9b38-9316-19a7-b432e5f60d23: initial configuration: index=0 servers=[] testing.go:149: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.432Z [INFO] server-09135aeb-9b38-9316-19a7-b432e5f60d23: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.432Z [INFO] server-09135aeb-9b38-9316-19a7-b432e5f60d23: entering follower state: follower="Node at 09135aeb-9b38-9316-19a7-b432e5f60d23 [Follower]" leader= testing.go:149: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.432Z [INFO] server-09135aeb-9b38-9316-19a7-b432e5f60d23: entering follower state: follower="Node at 09135aeb-9b38-9316-19a7-b432e5f60d23 [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.432Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:23.432Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.496Z [WARN] server-09135aeb-9b38-9316-19a7-b432e5f60d23: no known peers, aborting election testing.go:149: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.496Z [WARN] server-09135aeb-9b38-9316-19a7-b432e5f60d23: no known peers, aborting election testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [WARN] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [WARN] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: entering candidate state: node="Node at 4cfcbcd0-981b-e060-8e39-69dec5f5017f [Candidate]" term=2 testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: entering candidate state: node="Node at 4cfcbcd0-981b-e060-8e39-69dec5f5017f [Candidate]" term=2 testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: election won: tally=2 testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: election won: tally=2 testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: entering leader state: leader="Node at 4cfcbcd0-981b-e060-8e39-69dec5f5017f [Leader]" testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: entering leader state: leader="Node at 4cfcbcd0-981b-e060-8e39-69dec5f5017f [Leader]" testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: added peer, starting replication: peer=server-83070580-0ca6-9a61-3d95-daeb3820ce41 testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: added peer, starting replication: peer=server-83070580-0ca6-9a61-3d95-daeb3820ce41 testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: pipelining replication: peer="{Voter server-83070580-0ca6-9a61-3d95-daeb3820ce41 83070580-0ca6-9a61-3d95-daeb3820ce41}" testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.509Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: pipelining replication: peer="{Voter server-83070580-0ca6-9a61-3d95-daeb3820ce41 83070580-0ca6-9a61-3d95-daeb3820ce41}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.615Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.509806819 +0000 UTC m=+4.488615584 (1 nodes), 76.910087ms from start of poll, 77.565441ms from cluster start. Timeout at 2025-02-09 06:52:23.615558569 +0000 UTC m=+4.594367329, 105.751745ms after stability testing.go:149: cluster: 2025-02-09T06:52:23.615Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.509806819 +0000 UTC m=+4.488615584 (1 nodes), 76.910087ms from start of poll, 77.565441ms from cluster start. Timeout at 2025-02-09 06:52:23.615558569 +0000 UTC m=+4.594367329, 105.751745ms after stability testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: updating configuration: command=AddStaging server-id=server-09135aeb-9b38-9316-19a7-b432e5f60d23 server-addr=09135aeb-9b38-9316-19a7-b432e5f60d23 servers="[{Suffrage:Voter ID:server-4cfcbcd0-981b-e060-8e39-69dec5f5017f Address:4cfcbcd0-981b-e060-8e39-69dec5f5017f} {Suffrage:Voter ID:server-83070580-0ca6-9a61-3d95-daeb3820ce41 Address:83070580-0ca6-9a61-3d95-daeb3820ce41} {Suffrage:Voter ID:server-09135aeb-9b38-9316-19a7-b432e5f60d23 Address:09135aeb-9b38-9316-19a7-b432e5f60d23}]" testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: updating configuration: command=AddStaging server-id=server-09135aeb-9b38-9316-19a7-b432e5f60d23 server-addr=09135aeb-9b38-9316-19a7-b432e5f60d23 servers="[{Suffrage:Voter ID:server-4cfcbcd0-981b-e060-8e39-69dec5f5017f Address:4cfcbcd0-981b-e060-8e39-69dec5f5017f} {Suffrage:Voter ID:server-83070580-0ca6-9a61-3d95-daeb3820ce41 Address:83070580-0ca6-9a61-3d95-daeb3820ce41} {Suffrage:Voter ID:server-09135aeb-9b38-9316-19a7-b432e5f60d23 Address:09135aeb-9b38-9316-19a7-b432e5f60d23}]" testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: added peer, starting replication: peer=server-09135aeb-9b38-9316-19a7-b432e5f60d23 testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: added peer, starting replication: peer=server-09135aeb-9b38-9316-19a7-b432e5f60d23 testLoggerAdapter verbose: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.615Z [WARN] server-09135aeb-9b38-9316-19a7-b432e5f60d23: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-09135aeb-9b38-9316-19a7-b432e5f60d23: 2025-02-09T06:52:23.615Z [WARN] server-09135aeb-9b38-9316-19a7-b432e5f60d23: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [WARN] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: appendEntries rejected, sending older logs: peer="{Voter server-09135aeb-9b38-9316-19a7-b432e5f60d23 09135aeb-9b38-9316-19a7-b432e5f60d23}" next=1 testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [WARN] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: appendEntries rejected, sending older logs: peer="{Voter server-09135aeb-9b38-9316-19a7-b432e5f60d23 09135aeb-9b38-9316-19a7-b432e5f60d23}" next=1 testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: pipelining replication: peer="{Voter server-09135aeb-9b38-9316-19a7-b432e5f60d23 09135aeb-9b38-9316-19a7-b432e5f60d23}" testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.615Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: pipelining replication: peer="{Voter server-09135aeb-9b38-9316-19a7-b432e5f60d23 09135aeb-9b38-9316-19a7-b432e5f60d23}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.621Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:23.621Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.727Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.621956981 +0000 UTC m=+4.600765752 (1 nodes), 506ns from start of poll, 189.715609ms from cluster start. Timeout at 2025-02-09 06:52:23.72709286 +0000 UTC m=+4.705901620, 105.135868ms after stability testing.go:149: cluster: 2025-02-09T06:52:23.727Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.621956981 +0000 UTC m=+4.600765752 (1 nodes), 506ns from start of poll, 189.715609ms from cluster start. Timeout at 2025-02-09 06:52:23.72709286 +0000 UTC m=+4.705901620, 105.135868ms after stability testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.727Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: aborting pipeline replication: peer="{Voter server-83070580-0ca6-9a61-3d95-daeb3820ce41 83070580-0ca6-9a61-3d95-daeb3820ce41}" testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.727Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: aborting pipeline replication: peer="{Voter server-83070580-0ca6-9a61-3d95-daeb3820ce41 83070580-0ca6-9a61-3d95-daeb3820ce41}" testLoggerAdapter verbose: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.727Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: aborting pipeline replication: peer="{Voter server-09135aeb-9b38-9316-19a7-b432e5f60d23 09135aeb-9b38-9316-19a7-b432e5f60d23}" testing.go:149: server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: 2025-02-09T06:52:23.727Z [INFO] server-4cfcbcd0-981b-e060-8e39-69dec5f5017f: aborting pipeline replication: peer="{Voter server-09135aeb-9b38-9316-19a7-b432e5f60d23 09135aeb-9b38-9316-19a7-b432e5f60d23}" --- PASS: TestRaft_HasExistingState (0.30s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.727Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be2792e7-5695-74ef-abcc-17f6cf32c26e Address:be2792e7-5695-74ef-abcc-17f6cf32c26e}]" testing.go:149: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.727Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be2792e7-5695-74ef-abcc-17f6cf32c26e Address:be2792e7-5695-74ef-abcc-17f6cf32c26e}]" testLoggerAdapter verbose: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.728Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: entering follower state: follower="Node at be2792e7-5695-74ef-abcc-17f6cf32c26e [Follower]" leader= testing.go:149: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.728Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: entering follower state: follower="Node at be2792e7-5695-74ef-abcc-17f6cf32c26e [Follower]" leader= testLoggerAdapter verbose: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [WARN] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [WARN] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: entering candidate state: node="Node at be2792e7-5695-74ef-abcc-17f6cf32c26e [Candidate]" term=2 testing.go:149: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: entering candidate state: node="Node at be2792e7-5695-74ef-abcc-17f6cf32c26e [Candidate]" term=2 testLoggerAdapter verbose: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: election won: tally=1 testing.go:149: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: election won: tally=1 testLoggerAdapter verbose: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: entering leader state: leader="Node at be2792e7-5695-74ef-abcc-17f6cf32c26e [Leader]" testing.go:149: server-be2792e7-5695-74ef-abcc-17f6cf32c26e: 2025-02-09T06:52:23.818Z [INFO] server-be2792e7-5695-74ef-abcc-17f6cf32c26e: entering leader state: leader="Node at be2792e7-5695-74ef-abcc-17f6cf32c26e [Leader]" --- PASS: TestRaft_SingleNode (0.09s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-7b2e7087-dcf9-edad-2fab-5170f13283d3: 2025-02-09T06:52:23.819Z [INFO] server-7b2e7087-dcf9-edad-2fab-5170f13283d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2e7087-dcf9-edad-2fab-5170f13283d3 Address:7b2e7087-dcf9-edad-2fab-5170f13283d3} {Suffrage:Voter ID:server-c72960bd-5d54-05fa-7da6-ed70508d3e8a Address:c72960bd-5d54-05fa-7da6-ed70508d3e8a} {Suffrage:Voter ID:server-ea541217-9cc8-5c19-a099-dae1392cf9b3 Address:ea541217-9cc8-5c19-a099-dae1392cf9b3}]" testing.go:149: server-7b2e7087-dcf9-edad-2fab-5170f13283d3: 2025-02-09T06:52:23.819Z [INFO] server-7b2e7087-dcf9-edad-2fab-5170f13283d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2e7087-dcf9-edad-2fab-5170f13283d3 Address:7b2e7087-dcf9-edad-2fab-5170f13283d3} {Suffrage:Voter ID:server-c72960bd-5d54-05fa-7da6-ed70508d3e8a Address:c72960bd-5d54-05fa-7da6-ed70508d3e8a} {Suffrage:Voter ID:server-ea541217-9cc8-5c19-a099-dae1392cf9b3 Address:ea541217-9cc8-5c19-a099-dae1392cf9b3}]" testLoggerAdapter verbose: server-7b2e7087-dcf9-edad-2fab-5170f13283d3: 2025-02-09T06:52:23.819Z [INFO] server-7b2e7087-dcf9-edad-2fab-5170f13283d3: entering follower state: follower="Node at 7b2e7087-dcf9-edad-2fab-5170f13283d3 [Follower]" leader= testing.go:149: server-7b2e7087-dcf9-edad-2fab-5170f13283d3: 2025-02-09T06:52:23.819Z [INFO] server-7b2e7087-dcf9-edad-2fab-5170f13283d3: entering follower state: follower="Node at 7b2e7087-dcf9-edad-2fab-5170f13283d3 [Follower]" leader= testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.819Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2e7087-dcf9-edad-2fab-5170f13283d3 Address:7b2e7087-dcf9-edad-2fab-5170f13283d3} {Suffrage:Voter ID:server-c72960bd-5d54-05fa-7da6-ed70508d3e8a Address:c72960bd-5d54-05fa-7da6-ed70508d3e8a} {Suffrage:Voter ID:server-ea541217-9cc8-5c19-a099-dae1392cf9b3 Address:ea541217-9cc8-5c19-a099-dae1392cf9b3}]" testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.819Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2e7087-dcf9-edad-2fab-5170f13283d3 Address:7b2e7087-dcf9-edad-2fab-5170f13283d3} {Suffrage:Voter ID:server-c72960bd-5d54-05fa-7da6-ed70508d3e8a Address:c72960bd-5d54-05fa-7da6-ed70508d3e8a} {Suffrage:Voter ID:server-ea541217-9cc8-5c19-a099-dae1392cf9b3 Address:ea541217-9cc8-5c19-a099-dae1392cf9b3}]" testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.819Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: entering follower state: follower="Node at c72960bd-5d54-05fa-7da6-ed70508d3e8a [Follower]" leader= testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.819Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: entering follower state: follower="Node at c72960bd-5d54-05fa-7da6-ed70508d3e8a [Follower]" leader= testLoggerAdapter verbose: server-ea541217-9cc8-5c19-a099-dae1392cf9b3: 2025-02-09T06:52:23.819Z [INFO] server-ea541217-9cc8-5c19-a099-dae1392cf9b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2e7087-dcf9-edad-2fab-5170f13283d3 Address:7b2e7087-dcf9-edad-2fab-5170f13283d3} {Suffrage:Voter ID:server-c72960bd-5d54-05fa-7da6-ed70508d3e8a Address:c72960bd-5d54-05fa-7da6-ed70508d3e8a} {Suffrage:Voter ID:server-ea541217-9cc8-5c19-a099-dae1392cf9b3 Address:ea541217-9cc8-5c19-a099-dae1392cf9b3}]" testing.go:149: server-ea541217-9cc8-5c19-a099-dae1392cf9b3: 2025-02-09T06:52:23.819Z [INFO] server-ea541217-9cc8-5c19-a099-dae1392cf9b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2e7087-dcf9-edad-2fab-5170f13283d3 Address:7b2e7087-dcf9-edad-2fab-5170f13283d3} {Suffrage:Voter ID:server-c72960bd-5d54-05fa-7da6-ed70508d3e8a Address:c72960bd-5d54-05fa-7da6-ed70508d3e8a} {Suffrage:Voter ID:server-ea541217-9cc8-5c19-a099-dae1392cf9b3 Address:ea541217-9cc8-5c19-a099-dae1392cf9b3}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.820Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:23.820Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-ea541217-9cc8-5c19-a099-dae1392cf9b3: 2025-02-09T06:52:23.820Z [INFO] server-ea541217-9cc8-5c19-a099-dae1392cf9b3: entering follower state: follower="Node at ea541217-9cc8-5c19-a099-dae1392cf9b3 [Follower]" leader= testing.go:149: server-ea541217-9cc8-5c19-a099-dae1392cf9b3: 2025-02-09T06:52:23.820Z [INFO] server-ea541217-9cc8-5c19-a099-dae1392cf9b3: entering follower state: follower="Node at ea541217-9cc8-5c19-a099-dae1392cf9b3 [Follower]" leader= testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [WARN] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [WARN] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: entering candidate state: node="Node at c72960bd-5d54-05fa-7da6-ed70508d3e8a [Candidate]" term=2 testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: entering candidate state: node="Node at c72960bd-5d54-05fa-7da6-ed70508d3e8a [Candidate]" term=2 testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: election won: tally=2 testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: election won: tally=2 testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: entering leader state: leader="Node at c72960bd-5d54-05fa-7da6-ed70508d3e8a [Leader]" testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: entering leader state: leader="Node at c72960bd-5d54-05fa-7da6-ed70508d3e8a [Leader]" testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: added peer, starting replication: peer=server-7b2e7087-dcf9-edad-2fab-5170f13283d3 testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: added peer, starting replication: peer=server-7b2e7087-dcf9-edad-2fab-5170f13283d3 testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: added peer, starting replication: peer=server-ea541217-9cc8-5c19-a099-dae1392cf9b3 testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: added peer, starting replication: peer=server-ea541217-9cc8-5c19-a099-dae1392cf9b3 testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: pipelining replication: peer="{Voter server-ea541217-9cc8-5c19-a099-dae1392cf9b3 ea541217-9cc8-5c19-a099-dae1392cf9b3}" testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: pipelining replication: peer="{Voter server-ea541217-9cc8-5c19-a099-dae1392cf9b3 ea541217-9cc8-5c19-a099-dae1392cf9b3}" testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: pipelining replication: peer="{Voter server-7b2e7087-dcf9-edad-2fab-5170f13283d3 7b2e7087-dcf9-edad-2fab-5170f13283d3}" testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:23.871Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: pipelining replication: peer="{Voter server-7b2e7087-dcf9-edad-2fab-5170f13283d3 7b2e7087-dcf9-edad-2fab-5170f13283d3}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.977Z [INFO] stable state for Follower reached at 2025-02-09 06:52:23.871708584 +0000 UTC m=+4.850517347 (2 nodes), 51.622837ms from start of poll, 52.076696ms from cluster start. Timeout at 2025-02-09 06:52:23.977218918 +0000 UTC m=+4.956027687, 105.51034ms after stability testing.go:149: cluster: 2025-02-09T06:52:23.977Z [INFO] stable state for Follower reached at 2025-02-09 06:52:23.871708584 +0000 UTC m=+4.850517347 (2 nodes), 51.622837ms from start of poll, 52.076696ms from cluster start. Timeout at 2025-02-09 06:52:23.977218918 +0000 UTC m=+4.956027687, 105.51034ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:23.977Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:23.977Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.082Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.977291405 +0000 UTC m=+4.956100170 (1 nodes), 468ns from start of poll, 157.659519ms from cluster start. Timeout at 2025-02-09 06:52:24.08242389 +0000 UTC m=+5.061232675, 105.132505ms after stability testing.go:149: cluster: 2025-02-09T06:52:24.082Z [INFO] stable state for Leader reached at 2025-02-09 06:52:23.977291405 +0000 UTC m=+4.956100170 (1 nodes), 468ns from start of poll, 157.659519ms from cluster start. Timeout at 2025-02-09 06:52:24.08242389 +0000 UTC m=+5.061232675, 105.132505ms after stability testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:24.093Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: aborting pipeline replication: peer="{Voter server-ea541217-9cc8-5c19-a099-dae1392cf9b3 ea541217-9cc8-5c19-a099-dae1392cf9b3}" testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:24.093Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: aborting pipeline replication: peer="{Voter server-ea541217-9cc8-5c19-a099-dae1392cf9b3 ea541217-9cc8-5c19-a099-dae1392cf9b3}" testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:24.093Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: aborting pipeline replication: peer="{Voter server-7b2e7087-dcf9-edad-2fab-5170f13283d3 7b2e7087-dcf9-edad-2fab-5170f13283d3}" testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:24.093Z [INFO] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: aborting pipeline replication: peer="{Voter server-7b2e7087-dcf9-edad-2fab-5170f13283d3 7b2e7087-dcf9-edad-2fab-5170f13283d3}" testLoggerAdapter verbose: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:24.143Z [ERROR] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: failed to heartbeat to: peer=ea541217-9cc8-5c19-a099-dae1392cf9b3 error="command timed out" testing.go:149: server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: 2025-02-09T06:52:24.143Z [ERROR] server-c72960bd-5d54-05fa-7da6-ed70508d3e8a: failed to heartbeat to: peer=ea541217-9cc8-5c19-a099-dae1392cf9b3 error="command timed out" --- PASS: TestRaft_TripleNode (0.33s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.144Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b3a9873-81f7-9223-b14f-d303c904b789 Address:9b3a9873-81f7-9223-b14f-d303c904b789} {Suffrage:Voter ID:server-054499dc-6587-90bb-231e-873c097815a2 Address:054499dc-6587-90bb-231e-873c097815a2} {Suffrage:Voter ID:server-b25a8bce-8672-f03b-e815-2af799dd3a0e Address:b25a8bce-8672-f03b-e815-2af799dd3a0e}]" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.144Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b3a9873-81f7-9223-b14f-d303c904b789 Address:9b3a9873-81f7-9223-b14f-d303c904b789} {Suffrage:Voter ID:server-054499dc-6587-90bb-231e-873c097815a2 Address:054499dc-6587-90bb-231e-873c097815a2} {Suffrage:Voter ID:server-b25a8bce-8672-f03b-e815-2af799dd3a0e Address:b25a8bce-8672-f03b-e815-2af799dd3a0e}]" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.145Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering follower state: follower="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Follower]" leader= testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.145Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b3a9873-81f7-9223-b14f-d303c904b789 Address:9b3a9873-81f7-9223-b14f-d303c904b789} {Suffrage:Voter ID:server-054499dc-6587-90bb-231e-873c097815a2 Address:054499dc-6587-90bb-231e-873c097815a2} {Suffrage:Voter ID:server-b25a8bce-8672-f03b-e815-2af799dd3a0e Address:b25a8bce-8672-f03b-e815-2af799dd3a0e}]" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.145Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering follower state: follower="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Follower]" leader= testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.145Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b3a9873-81f7-9223-b14f-d303c904b789 Address:9b3a9873-81f7-9223-b14f-d303c904b789} {Suffrage:Voter ID:server-054499dc-6587-90bb-231e-873c097815a2 Address:054499dc-6587-90bb-231e-873c097815a2} {Suffrage:Voter ID:server-b25a8bce-8672-f03b-e815-2af799dd3a0e Address:b25a8bce-8672-f03b-e815-2af799dd3a0e}]" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.145Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering follower state: follower="Node at 054499dc-6587-90bb-231e-873c097815a2 [Follower]" leader= testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.145Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering follower state: follower="Node at 054499dc-6587-90bb-231e-873c097815a2 [Follower]" leader= testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.145Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b3a9873-81f7-9223-b14f-d303c904b789 Address:9b3a9873-81f7-9223-b14f-d303c904b789} {Suffrage:Voter ID:server-054499dc-6587-90bb-231e-873c097815a2 Address:054499dc-6587-90bb-231e-873c097815a2} {Suffrage:Voter ID:server-b25a8bce-8672-f03b-e815-2af799dd3a0e Address:b25a8bce-8672-f03b-e815-2af799dd3a0e}]" testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.145Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b3a9873-81f7-9223-b14f-d303c904b789 Address:9b3a9873-81f7-9223-b14f-d303c904b789} {Suffrage:Voter ID:server-054499dc-6587-90bb-231e-873c097815a2 Address:054499dc-6587-90bb-231e-873c097815a2} {Suffrage:Voter ID:server-b25a8bce-8672-f03b-e815-2af799dd3a0e Address:b25a8bce-8672-f03b-e815-2af799dd3a0e}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.145Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:24.145Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.145Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering follower state: follower="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Follower]" leader= testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.145Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering follower state: follower="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Follower]" leader= testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [WARN] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [WARN] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering candidate state: node="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Candidate]" term=2 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering candidate state: node="Node at 054499dc-6587-90bb-231e-873c097815a2 [Candidate]" term=2 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering candidate state: node="Node at 054499dc-6587-90bb-231e-873c097815a2 [Candidate]" term=2 testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering candidate state: node="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Candidate]" term=2 testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: election won: tally=2 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: election won: tally=2 testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering leader state: leader="Node at 054499dc-6587-90bb-231e-873c097815a2 [Leader]" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering leader state: leader="Node at 054499dc-6587-90bb-231e-873c097815a2 [Leader]" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: added peer, starting replication: peer=server-9b3a9873-81f7-9223-b14f-d303c904b789 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: added peer, starting replication: peer=server-9b3a9873-81f7-9223-b14f-d303c904b789 testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: added peer, starting replication: peer=server-b25a8bce-8672-f03b-e815-2af799dd3a0e testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: added peer, starting replication: peer=server-b25a8bce-8672-f03b-e815-2af799dd3a0e testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: rejecting vote request since we have a leader: from=b25a8bce-8672-f03b-e815-2af799dd3a0e leader=054499dc-6587-90bb-231e-873c097815a2 testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.210Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: duplicate requestVote for same term: term=2 testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: duplicate requestVote for same term: term=2 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: rejecting vote request since we have a leader: from=b25a8bce-8672-f03b-e815-2af799dd3a0e leader=054499dc-6587-90bb-231e-873c097815a2 testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering follower state: follower="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Follower]" leader=054499dc-6587-90bb-231e-873c097815a2 testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.210Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering follower state: follower="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Follower]" leader=054499dc-6587-90bb-231e-873c097815a2 testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: pipelining replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.210Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: duplicate requestVote for same term: term=2 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: pipelining replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: pipelining replication: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.210Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: pipelining replication: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.316Z [INFO] stable state for Follower reached at 2025-02-09 06:52:24.210848601 +0000 UTC m=+5.189657392 (2 nodes), 65.556986ms from start of poll, 66.013584ms from cluster start. Timeout at 2025-02-09 06:52:24.316627114 +0000 UTC m=+5.295435881, 105.778489ms after stability testing.go:149: cluster: 2025-02-09T06:52:24.316Z [INFO] stable state for Follower reached at 2025-02-09 06:52:24.210848601 +0000 UTC m=+5.189657392 (2 nodes), 65.556986ms from start of poll, 66.013584ms from cluster start. Timeout at 2025-02-09 06:52:24.316627114 +0000 UTC m=+5.295435881, 105.778489ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.316Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:24.316Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.422Z [INFO] stable state for Leader reached at 2025-02-09 06:52:24.316751283 +0000 UTC m=+5.295560047 (1 nodes), 763ns from start of poll, 171.916239ms from cluster start. Timeout at 2025-02-09 06:52:24.422751964 +0000 UTC m=+5.401560722, 106.000675ms after stability testing.go:149: cluster: 2025-02-09T06:52:24.422Z [INFO] stable state for Leader reached at 2025-02-09 06:52:24.316751283 +0000 UTC m=+5.295560047 (1 nodes), 763ns from start of poll, 171.916239ms from cluster start. Timeout at 2025-02-09 06:52:24.422751964 +0000 UTC m=+5.401560722, 106.000675ms after stability raft_test.go:336: [INFO] Disconnecting Node at 054499dc-6587-90bb-231e-873c097815a2 [Leader] testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.434Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.434Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.434Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to pipeline appendEntries: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="append pipeline closed" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.434Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to pipeline appendEntries: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="append pipeline closed" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.434Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: aborting pipeline replication: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.434Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: aborting pipeline replication: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.437Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to pipeline appendEntries: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="append pipeline closed" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.437Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to pipeline appendEntries: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="append pipeline closed" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.437Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: aborting pipeline replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.437Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: aborting pipeline replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.438Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.438Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.439Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:24.439Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.443Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.443Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.443Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.443Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.451Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.451Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.457Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.457Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.460Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.460Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.464Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.464Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.468Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.468Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.476Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.476Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.478Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact: server-id=server-9b3a9873-81f7-9223-b14f-d303c904b789 time=50.63595ms testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.478Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact: server-id=server-9b3a9873-81f7-9223-b14f-d303c904b789 time=50.63595ms testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.480Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.480Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.483Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.483Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact: server-id=server-9b3a9873-81f7-9223-b14f-d303c904b789 time=61.400165ms testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact: server-id=server-9b3a9873-81f7-9223-b14f-d303c904b789 time=61.400165ms testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact: server-id=server-b25a8bce-8672-f03b-e815-2af799dd3a0e time=59.171303ms testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact: server-id=server-b25a8bce-8672-f03b-e815-2af799dd3a0e time=59.171303ms testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact quorum of nodes, stepping down testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering follower state: follower="Node at 054499dc-6587-90bb-231e-873c097815a2 [Follower]" leader= testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.489Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering follower state: follower="Node at 054499dc-6587-90bb-231e-873c097815a2 [Follower]" leader= testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.496Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.496Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=9b3a9873-81f7-9223-b14f-d303c904b789 error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.502Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.502Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to heartbeat to: peer=b25a8bce-8672-f03b-e815-2af799dd3a0e error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.510Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.510Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.510Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.510Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to appendEntries to: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.514Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: heartbeat timeout reached, starting election: last-leader=054499dc-6587-90bb-231e-873c097815a2 testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.514Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: heartbeat timeout reached, starting election: last-leader=054499dc-6587-90bb-231e-873c097815a2 testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.514Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering candidate state: node="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Candidate]" term=3 testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.514Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering candidate state: node="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Candidate]" term=3 testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.514Z [WARN] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: rejecting vote request since we have a leader: from=9b3a9873-81f7-9223-b14f-d303c904b789 leader=054499dc-6587-90bb-231e-873c097815a2 testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.514Z [WARN] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: rejecting vote request since we have a leader: from=9b3a9873-81f7-9223-b14f-d303c904b789 leader=054499dc-6587-90bb-231e-873c097815a2 testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.514Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to make requestVote RPC: target="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.514Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to make requestVote RPC: target="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.516Z [WARN] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: heartbeat timeout reached, starting election: last-leader=054499dc-6587-90bb-231e-873c097815a2 testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.516Z [WARN] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: heartbeat timeout reached, starting election: last-leader=054499dc-6587-90bb-231e-873c097815a2 testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.516Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering candidate state: node="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Candidate]" term=3 testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.516Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering candidate state: node="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Candidate]" term=3 testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.516Z [ERROR] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: failed to make requestVote RPC: target="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.516Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: duplicate requestVote for same term: term=3 testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.516Z [ERROR] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: failed to make requestVote RPC: target="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.516Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering candidate state: node="Node at 054499dc-6587-90bb-231e-873c097815a2 [Candidate]" term=3 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering candidate state: node="Node at 054499dc-6587-90bb-231e-873c097815a2 [Candidate]" term=3 testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.576Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: Election timeout reached, restarting election testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: Election timeout reached, restarting election testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering candidate state: node="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Candidate]" term=4 testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering candidate state: node="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Candidate]" term=4 testLoggerAdapter verbose: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.580Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering follower state: follower="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Follower]" leader= testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to make requestVote RPC: target="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-b25a8bce-8672-f03b-e815-2af799dd3a0e: 2025-02-09T06:52:24.580Z [INFO] server-b25a8bce-8672-f03b-e815-2af799dd3a0e: entering follower state: follower="Node at b25a8bce-8672-f03b-e815-2af799dd3a0e [Follower]" leader= testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to make requestVote RPC: target="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: election won: tally=2 testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: election won: tally=2 testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering leader state: leader="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Leader]" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: entering leader state: leader="Node at 9b3a9873-81f7-9223-b14f-d303c904b789 [Leader]" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: added peer, starting replication: peer=server-054499dc-6587-90bb-231e-873c097815a2 testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: added peer, starting replication: peer=server-054499dc-6587-90bb-231e-873c097815a2 testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: added peer, starting replication: peer=server-b25a8bce-8672-f03b-e815-2af799dd3a0e testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: added peer, starting replication: peer=server-b25a8bce-8672-f03b-e815-2af799dd3a0e testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: pipelining replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.580Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: pipelining replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.581Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.581Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.590Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.590Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.599Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.599Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.609Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.609Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.619Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.619Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.627Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.627Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.631Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to contact: server-id=server-054499dc-6587-90bb-231e-873c097815a2 time=50.681605ms testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.631Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to contact: server-id=server-054499dc-6587-90bb-231e-873c097815a2 time=50.681605ms testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.646Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.646Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to appendEntries to: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.653Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.653Z [ERROR] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to heartbeat to: peer=054499dc-6587-90bb-231e-873c097815a2 error="failed to connect to peer: 054499dc-6587-90bb-231e-873c097815a2" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: Election timeout reached, restarting election testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [WARN] server-054499dc-6587-90bb-231e-873c097815a2: Election timeout reached, restarting election testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering candidate state: node="Node at 054499dc-6587-90bb-231e-873c097815a2 [Candidate]" term=4 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering candidate state: node="Node at 054499dc-6587-90bb-231e-873c097815a2 [Candidate]" term=4 testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" error="failed to connect to peer: b25a8bce-8672-f03b-e815-2af799dd3a0e" testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.659Z [ERROR] server-054499dc-6587-90bb-231e-873c097815a2: failed to make requestVote RPC: target="{Voter server-9b3a9873-81f7-9223-b14f-d303c904b789 9b3a9873-81f7-9223-b14f-d303c904b789}" error="failed to connect to peer: 9b3a9873-81f7-9223-b14f-d303c904b789" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.679Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to contact: server-id=server-054499dc-6587-90bb-231e-873c097815a2 time=98.40232ms testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.679Z [WARN] server-9b3a9873-81f7-9223-b14f-d303c904b789: failed to contact: server-id=server-054499dc-6587-90bb-231e-873c097815a2 time=98.40232ms testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.686Z [INFO] stable state for Leader reached at 2025-02-09 06:52:24.580973346 +0000 UTC m=+5.559782129 (1 nodes), 141.73754ms from start of poll, 436.138321ms from cluster start. Timeout at 2025-02-09 06:52:24.686537699 +0000 UTC m=+5.665346467, 105.564338ms after stability testing.go:149: cluster: 2025-02-09T06:52:24.686Z [INFO] stable state for Leader reached at 2025-02-09 06:52:24.580973346 +0000 UTC m=+5.559782129 (1 nodes), 141.73754ms from start of poll, 436.138321ms from cluster start. Timeout at 2025-02-09 06:52:24.686537699 +0000 UTC m=+5.665346467, 105.564338ms after stability raft_test.go:371: [INFO] Reconnecting Node at 054499dc-6587-90bb-231e-873c097815a2 [Candidate] testLoggerAdapter verbose: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.695Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering follower state: follower="Node at 054499dc-6587-90bb-231e-873c097815a2 [Follower]" leader=9b3a9873-81f7-9223-b14f-d303c904b789 testing.go:149: server-054499dc-6587-90bb-231e-873c097815a2: 2025-02-09T06:52:24.695Z [INFO] server-054499dc-6587-90bb-231e-873c097815a2: entering follower state: follower="Node at 054499dc-6587-90bb-231e-873c097815a2 [Follower]" leader=9b3a9873-81f7-9223-b14f-d303c904b789 testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.695Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: pipelining replication: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.695Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: pipelining replication: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.700Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: aborting pipeline replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.700Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: aborting pipeline replication: peer="{Voter server-b25a8bce-8672-f03b-e815-2af799dd3a0e b25a8bce-8672-f03b-e815-2af799dd3a0e}" testLoggerAdapter verbose: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.700Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: aborting pipeline replication: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" testing.go:149: server-9b3a9873-81f7-9223-b14f-d303c904b789: 2025-02-09T06:52:24.700Z [INFO] server-9b3a9873-81f7-9223-b14f-d303c904b789: aborting pipeline replication: peer="{Voter server-054499dc-6587-90bb-231e-873c097815a2 054499dc-6587-90bb-231e-873c097815a2}" --- PASS: TestRaft_LeaderFail (0.56s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.701Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd Address:5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd} {Suffrage:Voter ID:server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 Address:0c532726-50ee-1ccc-2cd1-a3fbeb54fc02} {Suffrage:Voter ID:server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 Address:4451b7fd-c69b-caca-673d-3bf91c2d7ed0}]" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.701Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd Address:5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd} {Suffrage:Voter ID:server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 Address:0c532726-50ee-1ccc-2cd1-a3fbeb54fc02} {Suffrage:Voter ID:server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 Address:4451b7fd-c69b-caca-673d-3bf91c2d7ed0}]" testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.701Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: entering follower state: follower="Node at 5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd [Follower]" leader= testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.701Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: entering follower state: follower="Node at 5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd [Follower]" leader= testLoggerAdapter verbose: server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: 2025-02-09T06:52:24.701Z [INFO] server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd Address:5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd} {Suffrage:Voter ID:server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 Address:0c532726-50ee-1ccc-2cd1-a3fbeb54fc02} {Suffrage:Voter ID:server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 Address:4451b7fd-c69b-caca-673d-3bf91c2d7ed0}]" testing.go:149: server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: 2025-02-09T06:52:24.701Z [INFO] server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd Address:5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd} {Suffrage:Voter ID:server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 Address:0c532726-50ee-1ccc-2cd1-a3fbeb54fc02} {Suffrage:Voter ID:server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 Address:4451b7fd-c69b-caca-673d-3bf91c2d7ed0}]" testLoggerAdapter verbose: server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: 2025-02-09T06:52:24.701Z [INFO] server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: entering follower state: follower="Node at 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 [Follower]" leader= testing.go:149: server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: 2025-02-09T06:52:24.701Z [INFO] server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02: entering follower state: follower="Node at 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 [Follower]" leader= testLoggerAdapter verbose: server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: 2025-02-09T06:52:24.702Z [INFO] server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd Address:5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd} {Suffrage:Voter ID:server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 Address:0c532726-50ee-1ccc-2cd1-a3fbeb54fc02} {Suffrage:Voter ID:server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 Address:4451b7fd-c69b-caca-673d-3bf91c2d7ed0}]" testing.go:149: server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: 2025-02-09T06:52:24.702Z [INFO] server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd Address:5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd} {Suffrage:Voter ID:server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 Address:0c532726-50ee-1ccc-2cd1-a3fbeb54fc02} {Suffrage:Voter ID:server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 Address:4451b7fd-c69b-caca-673d-3bf91c2d7ed0}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.702Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:24.702Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: 2025-02-09T06:52:24.702Z [INFO] server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: entering follower state: follower="Node at 4451b7fd-c69b-caca-673d-3bf91c2d7ed0 [Follower]" leader= testing.go:149: server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: 2025-02-09T06:52:24.702Z [INFO] server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0: entering follower state: follower="Node at 4451b7fd-c69b-caca-673d-3bf91c2d7ed0 [Follower]" leader= testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [WARN] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [WARN] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: entering candidate state: node="Node at 5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd [Candidate]" term=2 testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: entering candidate state: node="Node at 5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd [Candidate]" term=2 testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: election won: tally=2 testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: election won: tally=2 testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: entering leader state: leader="Node at 5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd [Leader]" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: entering leader state: leader="Node at 5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd [Leader]" testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: added peer, starting replication: peer=server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: added peer, starting replication: peer=server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: added peer, starting replication: peer=server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: added peer, starting replication: peer=server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: pipelining replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.767Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: pipelining replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.768Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: pipelining replication: peer="{Voter server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 4451b7fd-c69b-caca-673d-3bf91c2d7ed0}" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.768Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: pipelining replication: peer="{Voter server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 4451b7fd-c69b-caca-673d-3bf91c2d7ed0}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.873Z [INFO] stable state for Leader reached at 2025-02-09 06:52:24.767954859 +0000 UTC m=+5.746763632 (1 nodes), 65.85725ms from start of poll, 66.302732ms from cluster start. Timeout at 2025-02-09 06:52:24.873505288 +0000 UTC m=+5.852314063, 105.550431ms after stability testing.go:149: cluster: 2025-02-09T06:52:24.873Z [INFO] stable state for Leader reached at 2025-02-09 06:52:24.767954859 +0000 UTC m=+5.746763632 (1 nodes), 65.85725ms from start of poll, 66.302732ms from cluster start. Timeout at 2025-02-09 06:52:24.873505288 +0000 UTC m=+5.852314063, 105.550431ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.873Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:24.873Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:24.979Z [INFO] stable state for Follower reached at 2025-02-09 06:52:24.873597771 +0000 UTC m=+5.852406541 (2 nodes), 684ns from start of poll, 171.945641ms from cluster start. Timeout at 2025-02-09 06:52:24.979425396 +0000 UTC m=+5.958234165, 105.827624ms after stability testing.go:149: cluster: 2025-02-09T06:52:24.979Z [INFO] stable state for Follower reached at 2025-02-09 06:52:24.873597771 +0000 UTC m=+5.852406541 (2 nodes), 684ns from start of poll, 171.945641ms from cluster start. Timeout at 2025-02-09 06:52:24.979425396 +0000 UTC m=+5.958234165, 105.827624ms after stability testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.979Z [ERROR] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: failed to pipeline appendEntries: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" error="append pipeline closed" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.979Z [ERROR] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: failed to pipeline appendEntries: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" error="append pipeline closed" testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.979Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: aborting pipeline replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.979Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: aborting pipeline replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.979Z [ERROR] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: failed to appendEntries to: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" error="failed to connect to peer: 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.979Z [ERROR] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: failed to appendEntries to: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" error="failed to connect to peer: 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.989Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: pipelining replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:24.989Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: pipelining replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.002Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:25.002Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.108Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.002758626 +0000 UTC m=+5.981567394 (1 nodes), 478ns from start of poll, 301.106494ms from cluster start. Timeout at 2025-02-09 06:52:25.108448446 +0000 UTC m=+6.087257214, 105.68982ms after stability testing.go:149: cluster: 2025-02-09T06:52:25.108Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.002758626 +0000 UTC m=+5.981567394 (1 nodes), 478ns from start of poll, 301.106494ms from cluster start. Timeout at 2025-02-09 06:52:25.108448446 +0000 UTC m=+6.087257214, 105.68982ms after stability testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:25.108Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: aborting pipeline replication: peer="{Voter server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 4451b7fd-c69b-caca-673d-3bf91c2d7ed0}" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:25.108Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: aborting pipeline replication: peer="{Voter server-4451b7fd-c69b-caca-673d-3bf91c2d7ed0 4451b7fd-c69b-caca-673d-3bf91c2d7ed0}" testLoggerAdapter verbose: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:25.108Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: aborting pipeline replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" testing.go:149: server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: 2025-02-09T06:52:25.108Z [INFO] server-5284d03a-60d3-c4bb-f2ec-ef2156bbc8dd: aborting pipeline replication: peer="{Voter server-0c532726-50ee-1ccc-2cd1-a3fbeb54fc02 0c532726-50ee-1ccc-2cd1-a3fbeb54fc02}" --- PASS: TestRaft_BehindFollower (0.41s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-24fc268c-7d21-51c8-02e1-3cd3597bc926: 2025-02-09T06:52:25.109Z [INFO] server-24fc268c-7d21-51c8-02e1-3cd3597bc926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24fc268c-7d21-51c8-02e1-3cd3597bc926 Address:24fc268c-7d21-51c8-02e1-3cd3597bc926} {Suffrage:Voter ID:server-b547d4e9-ca08-04ce-7a61-da70135403c1 Address:b547d4e9-ca08-04ce-7a61-da70135403c1} {Suffrage:Voter ID:server-bd572076-5f37-fe83-1979-73c4797ce710 Address:bd572076-5f37-fe83-1979-73c4797ce710}]" testing.go:149: server-24fc268c-7d21-51c8-02e1-3cd3597bc926: 2025-02-09T06:52:25.109Z [INFO] server-24fc268c-7d21-51c8-02e1-3cd3597bc926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24fc268c-7d21-51c8-02e1-3cd3597bc926 Address:24fc268c-7d21-51c8-02e1-3cd3597bc926} {Suffrage:Voter ID:server-b547d4e9-ca08-04ce-7a61-da70135403c1 Address:b547d4e9-ca08-04ce-7a61-da70135403c1} {Suffrage:Voter ID:server-bd572076-5f37-fe83-1979-73c4797ce710 Address:bd572076-5f37-fe83-1979-73c4797ce710}]" testLoggerAdapter verbose: server-b547d4e9-ca08-04ce-7a61-da70135403c1: 2025-02-09T06:52:25.109Z [INFO] server-b547d4e9-ca08-04ce-7a61-da70135403c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24fc268c-7d21-51c8-02e1-3cd3597bc926 Address:24fc268c-7d21-51c8-02e1-3cd3597bc926} {Suffrage:Voter ID:server-b547d4e9-ca08-04ce-7a61-da70135403c1 Address:b547d4e9-ca08-04ce-7a61-da70135403c1} {Suffrage:Voter ID:server-bd572076-5f37-fe83-1979-73c4797ce710 Address:bd572076-5f37-fe83-1979-73c4797ce710}]" testing.go:149: server-b547d4e9-ca08-04ce-7a61-da70135403c1: 2025-02-09T06:52:25.109Z [INFO] server-b547d4e9-ca08-04ce-7a61-da70135403c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24fc268c-7d21-51c8-02e1-3cd3597bc926 Address:24fc268c-7d21-51c8-02e1-3cd3597bc926} {Suffrage:Voter ID:server-b547d4e9-ca08-04ce-7a61-da70135403c1 Address:b547d4e9-ca08-04ce-7a61-da70135403c1} {Suffrage:Voter ID:server-bd572076-5f37-fe83-1979-73c4797ce710 Address:bd572076-5f37-fe83-1979-73c4797ce710}]" testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.109Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24fc268c-7d21-51c8-02e1-3cd3597bc926 Address:24fc268c-7d21-51c8-02e1-3cd3597bc926} {Suffrage:Voter ID:server-b547d4e9-ca08-04ce-7a61-da70135403c1 Address:b547d4e9-ca08-04ce-7a61-da70135403c1} {Suffrage:Voter ID:server-bd572076-5f37-fe83-1979-73c4797ce710 Address:bd572076-5f37-fe83-1979-73c4797ce710}]" testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.109Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24fc268c-7d21-51c8-02e1-3cd3597bc926 Address:24fc268c-7d21-51c8-02e1-3cd3597bc926} {Suffrage:Voter ID:server-b547d4e9-ca08-04ce-7a61-da70135403c1 Address:b547d4e9-ca08-04ce-7a61-da70135403c1} {Suffrage:Voter ID:server-bd572076-5f37-fe83-1979-73c4797ce710 Address:bd572076-5f37-fe83-1979-73c4797ce710}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.109Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:25.109Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-24fc268c-7d21-51c8-02e1-3cd3597bc926: 2025-02-09T06:52:25.109Z [INFO] server-24fc268c-7d21-51c8-02e1-3cd3597bc926: entering follower state: follower="Node at 24fc268c-7d21-51c8-02e1-3cd3597bc926 [Follower]" leader= testing.go:149: server-24fc268c-7d21-51c8-02e1-3cd3597bc926: 2025-02-09T06:52:25.109Z [INFO] server-24fc268c-7d21-51c8-02e1-3cd3597bc926: entering follower state: follower="Node at 24fc268c-7d21-51c8-02e1-3cd3597bc926 [Follower]" leader= testLoggerAdapter verbose: server-b547d4e9-ca08-04ce-7a61-da70135403c1: 2025-02-09T06:52:25.109Z [INFO] server-b547d4e9-ca08-04ce-7a61-da70135403c1: entering follower state: follower="Node at b547d4e9-ca08-04ce-7a61-da70135403c1 [Follower]" leader= testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.109Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: entering follower state: follower="Node at bd572076-5f37-fe83-1979-73c4797ce710 [Follower]" leader= testing.go:149: server-b547d4e9-ca08-04ce-7a61-da70135403c1: 2025-02-09T06:52:25.109Z [INFO] server-b547d4e9-ca08-04ce-7a61-da70135403c1: entering follower state: follower="Node at b547d4e9-ca08-04ce-7a61-da70135403c1 [Follower]" leader= testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.109Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: entering follower state: follower="Node at bd572076-5f37-fe83-1979-73c4797ce710 [Follower]" leader= testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [WARN] server-bd572076-5f37-fe83-1979-73c4797ce710: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [WARN] server-bd572076-5f37-fe83-1979-73c4797ce710: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: entering candidate state: node="Node at bd572076-5f37-fe83-1979-73c4797ce710 [Candidate]" term=2 testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: entering candidate state: node="Node at bd572076-5f37-fe83-1979-73c4797ce710 [Candidate]" term=2 testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: election won: tally=2 testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: election won: tally=2 testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: entering leader state: leader="Node at bd572076-5f37-fe83-1979-73c4797ce710 [Leader]" testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: entering leader state: leader="Node at bd572076-5f37-fe83-1979-73c4797ce710 [Leader]" testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: added peer, starting replication: peer=server-24fc268c-7d21-51c8-02e1-3cd3597bc926 testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: added peer, starting replication: peer=server-24fc268c-7d21-51c8-02e1-3cd3597bc926 testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: added peer, starting replication: peer=server-b547d4e9-ca08-04ce-7a61-da70135403c1 testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: added peer, starting replication: peer=server-b547d4e9-ca08-04ce-7a61-da70135403c1 testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: pipelining replication: peer="{Voter server-24fc268c-7d21-51c8-02e1-3cd3597bc926 24fc268c-7d21-51c8-02e1-3cd3597bc926}" testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: pipelining replication: peer="{Voter server-24fc268c-7d21-51c8-02e1-3cd3597bc926 24fc268c-7d21-51c8-02e1-3cd3597bc926}" testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: pipelining replication: peer="{Voter server-b547d4e9-ca08-04ce-7a61-da70135403c1 b547d4e9-ca08-04ce-7a61-da70135403c1}" testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.178Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: pipelining replication: peer="{Voter server-b547d4e9-ca08-04ce-7a61-da70135403c1 b547d4e9-ca08-04ce-7a61-da70135403c1}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.283Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.178332458 +0000 UTC m=+6.157141260 (1 nodes), 68.418307ms from start of poll, 68.867813ms from cluster start. Timeout at 2025-02-09 06:52:25.283367483 +0000 UTC m=+6.262176250, 105.03499ms after stability testing.go:149: cluster: 2025-02-09T06:52:25.283Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.178332458 +0000 UTC m=+6.157141260 (1 nodes), 68.418307ms from start of poll, 68.867813ms from cluster start. Timeout at 2025-02-09 06:52:25.283367483 +0000 UTC m=+6.262176250, 105.03499ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.283Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:25.283Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.389Z [INFO] stable state for Follower reached at 2025-02-09 06:52:25.28343057 +0000 UTC m=+6.262239335 (2 nodes), 703ns from start of poll, 173.965888ms from cluster start. Timeout at 2025-02-09 06:52:25.38918756 +0000 UTC m=+6.367996328, 105.756993ms after stability testing.go:149: cluster: 2025-02-09T06:52:25.389Z [INFO] stable state for Follower reached at 2025-02-09 06:52:25.28343057 +0000 UTC m=+6.262239335 (2 nodes), 703ns from start of poll, 173.965888ms from cluster start. Timeout at 2025-02-09 06:52:25.38918756 +0000 UTC m=+6.367996328, 105.756993ms after stability testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.389Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: aborting pipeline replication: peer="{Voter server-b547d4e9-ca08-04ce-7a61-da70135403c1 b547d4e9-ca08-04ce-7a61-da70135403c1}" testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.389Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: aborting pipeline replication: peer="{Voter server-b547d4e9-ca08-04ce-7a61-da70135403c1 b547d4e9-ca08-04ce-7a61-da70135403c1}" testLoggerAdapter verbose: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.389Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: aborting pipeline replication: peer="{Voter server-24fc268c-7d21-51c8-02e1-3cd3597bc926 24fc268c-7d21-51c8-02e1-3cd3597bc926}" testing.go:149: server-bd572076-5f37-fe83-1979-73c4797ce710: 2025-02-09T06:52:25.389Z [INFO] server-bd572076-5f37-fe83-1979-73c4797ce710: aborting pipeline replication: peer="{Voter server-24fc268c-7d21-51c8-02e1-3cd3597bc926 24fc268c-7d21-51c8-02e1-3cd3597bc926}" --- PASS: TestRaft_ApplyNonLeader (0.28s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-e761ae66-e3fa-4d4d-b517-728cac79cd66: 2025-02-09T06:52:25.390Z [INFO] server-e761ae66-e3fa-4d4d-b517-728cac79cd66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e761ae66-e3fa-4d4d-b517-728cac79cd66 Address:e761ae66-e3fa-4d4d-b517-728cac79cd66} {Suffrage:Voter ID:server-8552be4b-133a-3012-18f3-b660365357dc Address:8552be4b-133a-3012-18f3-b660365357dc} {Suffrage:Voter ID:server-50e7fee0-037c-4c7b-a19c-5c3b887655ef Address:50e7fee0-037c-4c7b-a19c-5c3b887655ef}]" testing.go:149: server-e761ae66-e3fa-4d4d-b517-728cac79cd66: 2025-02-09T06:52:25.390Z [INFO] server-e761ae66-e3fa-4d4d-b517-728cac79cd66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e761ae66-e3fa-4d4d-b517-728cac79cd66 Address:e761ae66-e3fa-4d4d-b517-728cac79cd66} {Suffrage:Voter ID:server-8552be4b-133a-3012-18f3-b660365357dc Address:8552be4b-133a-3012-18f3-b660365357dc} {Suffrage:Voter ID:server-50e7fee0-037c-4c7b-a19c-5c3b887655ef Address:50e7fee0-037c-4c7b-a19c-5c3b887655ef}]" testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.390Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e761ae66-e3fa-4d4d-b517-728cac79cd66 Address:e761ae66-e3fa-4d4d-b517-728cac79cd66} {Suffrage:Voter ID:server-8552be4b-133a-3012-18f3-b660365357dc Address:8552be4b-133a-3012-18f3-b660365357dc} {Suffrage:Voter ID:server-50e7fee0-037c-4c7b-a19c-5c3b887655ef Address:50e7fee0-037c-4c7b-a19c-5c3b887655ef}]" testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.390Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e761ae66-e3fa-4d4d-b517-728cac79cd66 Address:e761ae66-e3fa-4d4d-b517-728cac79cd66} {Suffrage:Voter ID:server-8552be4b-133a-3012-18f3-b660365357dc Address:8552be4b-133a-3012-18f3-b660365357dc} {Suffrage:Voter ID:server-50e7fee0-037c-4c7b-a19c-5c3b887655ef Address:50e7fee0-037c-4c7b-a19c-5c3b887655ef}]" testLoggerAdapter verbose: server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: 2025-02-09T06:52:25.390Z [INFO] server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e761ae66-e3fa-4d4d-b517-728cac79cd66 Address:e761ae66-e3fa-4d4d-b517-728cac79cd66} {Suffrage:Voter ID:server-8552be4b-133a-3012-18f3-b660365357dc Address:8552be4b-133a-3012-18f3-b660365357dc} {Suffrage:Voter ID:server-50e7fee0-037c-4c7b-a19c-5c3b887655ef Address:50e7fee0-037c-4c7b-a19c-5c3b887655ef}]" testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.390Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: entering follower state: follower="Node at 8552be4b-133a-3012-18f3-b660365357dc [Follower]" leader= testLoggerAdapter verbose: server-e761ae66-e3fa-4d4d-b517-728cac79cd66: 2025-02-09T06:52:25.390Z [INFO] server-e761ae66-e3fa-4d4d-b517-728cac79cd66: entering follower state: follower="Node at e761ae66-e3fa-4d4d-b517-728cac79cd66 [Follower]" leader= testing.go:149: server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: 2025-02-09T06:52:25.390Z [INFO] server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e761ae66-e3fa-4d4d-b517-728cac79cd66 Address:e761ae66-e3fa-4d4d-b517-728cac79cd66} {Suffrage:Voter ID:server-8552be4b-133a-3012-18f3-b660365357dc Address:8552be4b-133a-3012-18f3-b660365357dc} {Suffrage:Voter ID:server-50e7fee0-037c-4c7b-a19c-5c3b887655ef Address:50e7fee0-037c-4c7b-a19c-5c3b887655ef}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.390Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:25.390Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: 2025-02-09T06:52:25.390Z [INFO] server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: entering follower state: follower="Node at 50e7fee0-037c-4c7b-a19c-5c3b887655ef [Follower]" leader= testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.390Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: entering follower state: follower="Node at 8552be4b-133a-3012-18f3-b660365357dc [Follower]" leader= testing.go:149: server-e761ae66-e3fa-4d4d-b517-728cac79cd66: 2025-02-09T06:52:25.390Z [INFO] server-e761ae66-e3fa-4d4d-b517-728cac79cd66: entering follower state: follower="Node at e761ae66-e3fa-4d4d-b517-728cac79cd66 [Follower]" leader= testing.go:149: server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: 2025-02-09T06:52:25.390Z [INFO] server-50e7fee0-037c-4c7b-a19c-5c3b887655ef: entering follower state: follower="Node at 50e7fee0-037c-4c7b-a19c-5c3b887655ef [Follower]" leader= testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [WARN] server-8552be4b-133a-3012-18f3-b660365357dc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [WARN] server-8552be4b-133a-3012-18f3-b660365357dc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: entering candidate state: node="Node at 8552be4b-133a-3012-18f3-b660365357dc [Candidate]" term=2 testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: entering candidate state: node="Node at 8552be4b-133a-3012-18f3-b660365357dc [Candidate]" term=2 testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: election won: tally=2 testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: election won: tally=2 testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: entering leader state: leader="Node at 8552be4b-133a-3012-18f3-b660365357dc [Leader]" testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: entering leader state: leader="Node at 8552be4b-133a-3012-18f3-b660365357dc [Leader]" testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: added peer, starting replication: peer=server-e761ae66-e3fa-4d4d-b517-728cac79cd66 testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: added peer, starting replication: peer=server-e761ae66-e3fa-4d4d-b517-728cac79cd66 testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: added peer, starting replication: peer=server-50e7fee0-037c-4c7b-a19c-5c3b887655ef testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: added peer, starting replication: peer=server-50e7fee0-037c-4c7b-a19c-5c3b887655ef testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: pipelining replication: peer="{Voter server-e761ae66-e3fa-4d4d-b517-728cac79cd66 e761ae66-e3fa-4d4d-b517-728cac79cd66}" testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: pipelining replication: peer="{Voter server-e761ae66-e3fa-4d4d-b517-728cac79cd66 e761ae66-e3fa-4d4d-b517-728cac79cd66}" testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: pipelining replication: peer="{Voter server-50e7fee0-037c-4c7b-a19c-5c3b887655ef 50e7fee0-037c-4c7b-a19c-5c3b887655ef}" testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.507Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: pipelining replication: peer="{Voter server-50e7fee0-037c-4c7b-a19c-5c3b887655ef 50e7fee0-037c-4c7b-a19c-5c3b887655ef}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.712Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.507618887 +0000 UTC m=+6.486427657 (1 nodes), 117.016117ms from start of poll, 117.457266ms from cluster start. Timeout at 2025-02-09 06:52:25.712767928 +0000 UTC m=+6.691576696, 205.149039ms after stability testing.go:149: cluster: 2025-02-09T06:52:25.712Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.507618887 +0000 UTC m=+6.486427657 (1 nodes), 117.016117ms from start of poll, 117.457266ms from cluster start. Timeout at 2025-02-09 06:52:25.712767928 +0000 UTC m=+6.691576696, 205.149039ms after stability testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.724Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: aborting pipeline replication: peer="{Voter server-50e7fee0-037c-4c7b-a19c-5c3b887655ef 50e7fee0-037c-4c7b-a19c-5c3b887655ef}" testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.724Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: aborting pipeline replication: peer="{Voter server-50e7fee0-037c-4c7b-a19c-5c3b887655ef 50e7fee0-037c-4c7b-a19c-5c3b887655ef}" testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.724Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: aborting pipeline replication: peer="{Voter server-e761ae66-e3fa-4d4d-b517-728cac79cd66 e761ae66-e3fa-4d4d-b517-728cac79cd66}" testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.724Z [INFO] server-8552be4b-133a-3012-18f3-b660365357dc: aborting pipeline replication: peer="{Voter server-e761ae66-e3fa-4d4d-b517-728cac79cd66 e761ae66-e3fa-4d4d-b517-728cac79cd66}" testLoggerAdapter verbose: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.774Z [ERROR] server-8552be4b-133a-3012-18f3-b660365357dc: failed to heartbeat to: peer=e761ae66-e3fa-4d4d-b517-728cac79cd66 error="command timed out" testing.go:149: server-8552be4b-133a-3012-18f3-b660365357dc: 2025-02-09T06:52:25.774Z [ERROR] server-8552be4b-133a-3012-18f3-b660365357dc: failed to heartbeat to: peer=e761ae66-e3fa-4d4d-b517-728cac79cd66 error="command timed out" --- PASS: TestRaft_ApplyConcurrent (0.39s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.775Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d Address:dc607b2a-5e8a-119e-ee9a-7a350a77e40d}]" testing.go:149: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.775Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d Address:dc607b2a-5e8a-119e-ee9a-7a350a77e40d}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:25.775Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:25.775Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.775Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: entering follower state: follower="Node at dc607b2a-5e8a-119e-ee9a-7a350a77e40d [Follower]" leader= testing.go:149: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.775Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: entering follower state: follower="Node at dc607b2a-5e8a-119e-ee9a-7a350a77e40d [Follower]" leader= testLoggerAdapter verbose: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [WARN] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [WARN] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: entering candidate state: node="Node at dc607b2a-5e8a-119e-ee9a-7a350a77e40d [Candidate]" term=2 testing.go:149: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: entering candidate state: node="Node at dc607b2a-5e8a-119e-ee9a-7a350a77e40d [Candidate]" term=2 testLoggerAdapter verbose: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: election won: tally=1 testing.go:149: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: election won: tally=1 testLoggerAdapter verbose: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: entering leader state: leader="Node at dc607b2a-5e8a-119e-ee9a-7a350a77e40d [Leader]" testing.go:149: server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: 2025-02-09T06:52:25.921Z [INFO] server-dc607b2a-5e8a-119e-ee9a-7a350a77e40d: entering leader state: leader="Node at dc607b2a-5e8a-119e-ee9a-7a350a77e40d [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.123Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.921807154 +0000 UTC m=+6.900615937 (1 nodes), 146.464935ms from start of poll, 146.65945ms from cluster start. Timeout at 2025-02-09 06:52:26.123505157 +0000 UTC m=+7.102313941, 201.698004ms after stability testing.go:149: cluster: 2025-02-09T06:52:26.123Z [INFO] stable state for Leader reached at 2025-02-09 06:52:25.921807154 +0000 UTC m=+6.900615937 (1 nodes), 146.464935ms from start of poll, 146.65945ms from cluster start. Timeout at 2025-02-09 06:52:26.123505157 +0000 UTC m=+7.102313941, 201.698004ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.35s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.124Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a49a5533-1dc6-6d7a-9697-cec15b098f11 Address:a49a5533-1dc6-6d7a-9697-cec15b098f11} {Suffrage:Voter ID:server-93eb6166-f567-334d-42d6-870eb7277315 Address:93eb6166-f567-334d-42d6-870eb7277315}]" testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.124Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a49a5533-1dc6-6d7a-9697-cec15b098f11 Address:a49a5533-1dc6-6d7a-9697-cec15b098f11} {Suffrage:Voter ID:server-93eb6166-f567-334d-42d6-870eb7277315 Address:93eb6166-f567-334d-42d6-870eb7277315}]" testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.124Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering follower state: follower="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Follower]" leader= testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.124Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering follower state: follower="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Follower]" leader= testLoggerAdapter verbose: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.124Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a49a5533-1dc6-6d7a-9697-cec15b098f11 Address:a49a5533-1dc6-6d7a-9697-cec15b098f11} {Suffrage:Voter ID:server-93eb6166-f567-334d-42d6-870eb7277315 Address:93eb6166-f567-334d-42d6-870eb7277315}]" testing.go:149: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.124Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a49a5533-1dc6-6d7a-9697-cec15b098f11 Address:a49a5533-1dc6-6d7a-9697-cec15b098f11} {Suffrage:Voter ID:server-93eb6166-f567-334d-42d6-870eb7277315 Address:93eb6166-f567-334d-42d6-870eb7277315}]" testLoggerAdapter verbose: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.124Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: entering follower state: follower="Node at 93eb6166-f567-334d-42d6-870eb7277315 [Follower]" leader= testing.go:149: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.124Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: entering follower state: follower="Node at 93eb6166-f567-334d-42d6-870eb7277315 [Follower]" leader= testLoggerAdapter verbose: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.124Z [INFO] server-ca87de4a-7740-adf4-b3d7-64b46f167789: initial configuration: index=0 servers=[] testing.go:149: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.124Z [INFO] server-ca87de4a-7740-adf4-b3d7-64b46f167789: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.124Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:26.124Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.124Z [INFO] server-ca87de4a-7740-adf4-b3d7-64b46f167789: entering follower state: follower="Node at ca87de4a-7740-adf4-b3d7-64b46f167789 [Follower]" leader= testing.go:149: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.124Z [INFO] server-ca87de4a-7740-adf4-b3d7-64b46f167789: entering follower state: follower="Node at ca87de4a-7740-adf4-b3d7-64b46f167789 [Follower]" leader= testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.208Z [WARN] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.208Z [WARN] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.208Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering candidate state: node="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Candidate]" term=2 testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.208Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering candidate state: node="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Candidate]" term=2 testLoggerAdapter verbose: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.208Z [WARN] server-93eb6166-f567-334d-42d6-870eb7277315: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.208Z [WARN] server-93eb6166-f567-334d-42d6-870eb7277315: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.208Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: entering candidate state: node="Node at 93eb6166-f567-334d-42d6-870eb7277315 [Candidate]" term=2 testing.go:149: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.208Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: entering candidate state: node="Node at 93eb6166-f567-334d-42d6-870eb7277315 [Candidate]" term=2 testLoggerAdapter verbose: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.208Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: duplicate requestVote for same term: term=2 testing.go:149: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.208Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.208Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: duplicate requestVote for same term: term=2 testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.208Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.211Z [WARN] server-ca87de4a-7740-adf4-b3d7-64b46f167789: no known peers, aborting election testing.go:149: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.211Z [WARN] server-ca87de4a-7740-adf4-b3d7-64b46f167789: no known peers, aborting election testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [WARN] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: Election timeout reached, restarting election testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [WARN] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: Election timeout reached, restarting election testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering candidate state: node="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Candidate]" term=3 testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering candidate state: node="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Candidate]" term=3 testLoggerAdapter verbose: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.266Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: entering follower state: follower="Node at 93eb6166-f567-334d-42d6-870eb7277315 [Follower]" leader= testing.go:149: server-93eb6166-f567-334d-42d6-870eb7277315: 2025-02-09T06:52:26.266Z [INFO] server-93eb6166-f567-334d-42d6-870eb7277315: entering follower state: follower="Node at 93eb6166-f567-334d-42d6-870eb7277315 [Follower]" leader= testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: election won: tally=2 testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: election won: tally=2 testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering leader state: leader="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Leader]" testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: entering leader state: leader="Node at a49a5533-1dc6-6d7a-9697-cec15b098f11 [Leader]" testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: added peer, starting replication: peer=server-93eb6166-f567-334d-42d6-870eb7277315 testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: added peer, starting replication: peer=server-93eb6166-f567-334d-42d6-870eb7277315 testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: pipelining replication: peer="{Voter server-93eb6166-f567-334d-42d6-870eb7277315 93eb6166-f567-334d-42d6-870eb7277315}" testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.266Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: pipelining replication: peer="{Voter server-93eb6166-f567-334d-42d6-870eb7277315 93eb6166-f567-334d-42d6-870eb7277315}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.371Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.266277205 +0000 UTC m=+7.245085971 (1 nodes), 141.563202ms from start of poll, 142.06085ms from cluster start. Timeout at 2025-02-09 06:52:26.371863987 +0000 UTC m=+7.350672746, 105.586775ms after stability testing.go:149: cluster: 2025-02-09T06:52:26.371Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.266277205 +0000 UTC m=+7.245085971 (1 nodes), 141.563202ms from start of poll, 142.06085ms from cluster start. Timeout at 2025-02-09 06:52:26.371863987 +0000 UTC m=+7.350672746, 105.586775ms after stability testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.371Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: updating configuration: command=AddStaging server-id=server-ca87de4a-7740-adf4-b3d7-64b46f167789 server-addr=ca87de4a-7740-adf4-b3d7-64b46f167789 servers="[{Suffrage:Voter ID:server-a49a5533-1dc6-6d7a-9697-cec15b098f11 Address:a49a5533-1dc6-6d7a-9697-cec15b098f11} {Suffrage:Voter ID:server-93eb6166-f567-334d-42d6-870eb7277315 Address:93eb6166-f567-334d-42d6-870eb7277315} {Suffrage:Voter ID:server-ca87de4a-7740-adf4-b3d7-64b46f167789 Address:ca87de4a-7740-adf4-b3d7-64b46f167789}]" testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.371Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: updating configuration: command=AddStaging server-id=server-ca87de4a-7740-adf4-b3d7-64b46f167789 server-addr=ca87de4a-7740-adf4-b3d7-64b46f167789 servers="[{Suffrage:Voter ID:server-a49a5533-1dc6-6d7a-9697-cec15b098f11 Address:a49a5533-1dc6-6d7a-9697-cec15b098f11} {Suffrage:Voter ID:server-93eb6166-f567-334d-42d6-870eb7277315 Address:93eb6166-f567-334d-42d6-870eb7277315} {Suffrage:Voter ID:server-ca87de4a-7740-adf4-b3d7-64b46f167789 Address:ca87de4a-7740-adf4-b3d7-64b46f167789}]" testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.371Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: added peer, starting replication: peer=server-ca87de4a-7740-adf4-b3d7-64b46f167789 testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.371Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: added peer, starting replication: peer=server-ca87de4a-7740-adf4-b3d7-64b46f167789 testLoggerAdapter verbose: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.372Z [WARN] server-ca87de4a-7740-adf4-b3d7-64b46f167789: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-ca87de4a-7740-adf4-b3d7-64b46f167789: 2025-02-09T06:52:26.372Z [WARN] server-ca87de4a-7740-adf4-b3d7-64b46f167789: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.372Z [WARN] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: appendEntries rejected, sending older logs: peer="{Voter server-ca87de4a-7740-adf4-b3d7-64b46f167789 ca87de4a-7740-adf4-b3d7-64b46f167789}" next=1 testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.372Z [WARN] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: appendEntries rejected, sending older logs: peer="{Voter server-ca87de4a-7740-adf4-b3d7-64b46f167789 ca87de4a-7740-adf4-b3d7-64b46f167789}" next=1 testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.372Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:26.372Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.372Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: pipelining replication: peer="{Voter server-ca87de4a-7740-adf4-b3d7-64b46f167789 ca87de4a-7740-adf4-b3d7-64b46f167789}" testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.372Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: pipelining replication: peer="{Voter server-ca87de4a-7740-adf4-b3d7-64b46f167789 ca87de4a-7740-adf4-b3d7-64b46f167789}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.477Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.37221757 +0000 UTC m=+7.351026332 (1 nodes), 524ns from start of poll, 248.001211ms from cluster start. Timeout at 2025-02-09 06:52:26.47792676 +0000 UTC m=+7.456735529, 105.709197ms after stability testing.go:149: cluster: 2025-02-09T06:52:26.477Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.37221757 +0000 UTC m=+7.351026332 (1 nodes), 524ns from start of poll, 248.001211ms from cluster start. Timeout at 2025-02-09 06:52:26.47792676 +0000 UTC m=+7.456735529, 105.709197ms after stability testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.478Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: aborting pipeline replication: peer="{Voter server-ca87de4a-7740-adf4-b3d7-64b46f167789 ca87de4a-7740-adf4-b3d7-64b46f167789}" testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.478Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: aborting pipeline replication: peer="{Voter server-ca87de4a-7740-adf4-b3d7-64b46f167789 ca87de4a-7740-adf4-b3d7-64b46f167789}" testLoggerAdapter verbose: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.478Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: aborting pipeline replication: peer="{Voter server-93eb6166-f567-334d-42d6-870eb7277315 93eb6166-f567-334d-42d6-870eb7277315}" testing.go:149: server-a49a5533-1dc6-6d7a-9697-cec15b098f11: 2025-02-09T06:52:26.478Z [INFO] server-a49a5533-1dc6-6d7a-9697-cec15b098f11: aborting pipeline replication: peer="{Voter server-93eb6166-f567-334d-42d6-870eb7277315 93eb6166-f567-334d-42d6-870eb7277315}" --- PASS: TestRaft_JoinNode (0.35s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.478Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06a37545-98a9-5e84-74a8-fc56d794d5b0 Address:06a37545-98a9-5e84-74a8-fc56d794d5b0}]" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.478Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06a37545-98a9-5e84-74a8-fc56d794d5b0 Address:06a37545-98a9-5e84-74a8-fc56d794d5b0}]" testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.478Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: entering follower state: follower="Node at 06a37545-98a9-5e84-74a8-fc56d794d5b0 [Follower]" leader= testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.478Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: entering follower state: follower="Node at 06a37545-98a9-5e84-74a8-fc56d794d5b0 [Follower]" leader= testLoggerAdapter verbose: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.478Z [INFO] server-fa50b62f-5914-cf72-40da-c508a2882fa2: initial configuration: index=0 servers=[] testing.go:149: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.478Z [INFO] server-fa50b62f-5914-cf72-40da-c508a2882fa2: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.479Z [INFO] server-fa50b62f-5914-cf72-40da-c508a2882fa2: entering follower state: follower="Node at fa50b62f-5914-cf72-40da-c508a2882fa2 [Follower]" leader= testing.go:149: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.479Z [INFO] server-fa50b62f-5914-cf72-40da-c508a2882fa2: entering follower state: follower="Node at fa50b62f-5914-cf72-40da-c508a2882fa2 [Follower]" leader= testLoggerAdapter verbose: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.479Z [INFO] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: initial configuration: index=0 servers=[] testing.go:149: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.479Z [INFO] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.479Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:26.479Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.479Z [INFO] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: entering follower state: follower="Node at f9d6f83f-2682-e217-a8d0-30bac4ef600d [Follower]" leader= testing.go:149: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.479Z [INFO] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: entering follower state: follower="Node at f9d6f83f-2682-e217-a8d0-30bac4ef600d [Follower]" leader= testLoggerAdapter verbose: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.554Z [WARN] server-fa50b62f-5914-cf72-40da-c508a2882fa2: no known peers, aborting election testing.go:149: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.554Z [WARN] server-fa50b62f-5914-cf72-40da-c508a2882fa2: no known peers, aborting election testLoggerAdapter verbose: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.563Z [WARN] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: no known peers, aborting election testing.go:149: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.563Z [WARN] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: no known peers, aborting election testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [WARN] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [WARN] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: entering candidate state: node="Node at 06a37545-98a9-5e84-74a8-fc56d794d5b0 [Candidate]" term=2 testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: entering candidate state: node="Node at 06a37545-98a9-5e84-74a8-fc56d794d5b0 [Candidate]" term=2 testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: election won: tally=1 testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: election won: tally=1 testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: entering leader state: leader="Node at 06a37545-98a9-5e84-74a8-fc56d794d5b0 [Leader]" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.572Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: entering leader state: leader="Node at 06a37545-98a9-5e84-74a8-fc56d794d5b0 [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.677Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.572268901 +0000 UTC m=+7.551077664 (1 nodes), 93.043349ms from start of poll, 93.607956ms from cluster start. Timeout at 2025-02-09 06:52:26.677418669 +0000 UTC m=+7.656227438, 105.149774ms after stability testing.go:149: cluster: 2025-02-09T06:52:26.677Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.572268901 +0000 UTC m=+7.551077664 (1 nodes), 93.043349ms from start of poll, 93.607956ms from cluster start. Timeout at 2025-02-09 06:52:26.677418669 +0000 UTC m=+7.656227438, 105.149774ms after stability testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: updating configuration: command=AddStaging server-id=server-fa50b62f-5914-cf72-40da-c508a2882fa2 server-addr=fa50b62f-5914-cf72-40da-c508a2882fa2 servers="[{Suffrage:Voter ID:server-06a37545-98a9-5e84-74a8-fc56d794d5b0 Address:06a37545-98a9-5e84-74a8-fc56d794d5b0} {Suffrage:Voter ID:server-fa50b62f-5914-cf72-40da-c508a2882fa2 Address:fa50b62f-5914-cf72-40da-c508a2882fa2}]" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: updating configuration: command=AddStaging server-id=server-fa50b62f-5914-cf72-40da-c508a2882fa2 server-addr=fa50b62f-5914-cf72-40da-c508a2882fa2 servers="[{Suffrage:Voter ID:server-06a37545-98a9-5e84-74a8-fc56d794d5b0 Address:06a37545-98a9-5e84-74a8-fc56d794d5b0} {Suffrage:Voter ID:server-fa50b62f-5914-cf72-40da-c508a2882fa2 Address:fa50b62f-5914-cf72-40da-c508a2882fa2}]" testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: added peer, starting replication: peer=server-fa50b62f-5914-cf72-40da-c508a2882fa2 testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: added peer, starting replication: peer=server-fa50b62f-5914-cf72-40da-c508a2882fa2 testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.677Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.677Z [WARN] server-fa50b62f-5914-cf72-40da-c508a2882fa2: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: cluster: 2025-02-09T06:52:26.677Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-fa50b62f-5914-cf72-40da-c508a2882fa2: 2025-02-09T06:52:26.677Z [WARN] server-fa50b62f-5914-cf72-40da-c508a2882fa2: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [WARN] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: appendEntries rejected, sending older logs: peer="{Voter server-fa50b62f-5914-cf72-40da-c508a2882fa2 fa50b62f-5914-cf72-40da-c508a2882fa2}" next=1 testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [WARN] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: appendEntries rejected, sending older logs: peer="{Voter server-fa50b62f-5914-cf72-40da-c508a2882fa2 fa50b62f-5914-cf72-40da-c508a2882fa2}" next=1 testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: pipelining replication: peer="{Voter server-fa50b62f-5914-cf72-40da-c508a2882fa2 fa50b62f-5914-cf72-40da-c508a2882fa2}" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.677Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: pipelining replication: peer="{Voter server-fa50b62f-5914-cf72-40da-c508a2882fa2 fa50b62f-5914-cf72-40da-c508a2882fa2}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.783Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.677627384 +0000 UTC m=+7.656436148 (1 nodes), 405ns from start of poll, 198.96644ms from cluster start. Timeout at 2025-02-09 06:52:26.783313656 +0000 UTC m=+7.762122425, 105.686277ms after stability testing.go:149: cluster: 2025-02-09T06:52:26.783Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.677627384 +0000 UTC m=+7.656436148 (1 nodes), 405ns from start of poll, 198.96644ms from cluster start. Timeout at 2025-02-09 06:52:26.783313656 +0000 UTC m=+7.762122425, 105.686277ms after stability testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: updating configuration: command=AddStaging server-id=server-f9d6f83f-2682-e217-a8d0-30bac4ef600d server-addr=f9d6f83f-2682-e217-a8d0-30bac4ef600d servers="[{Suffrage:Voter ID:server-06a37545-98a9-5e84-74a8-fc56d794d5b0 Address:06a37545-98a9-5e84-74a8-fc56d794d5b0} {Suffrage:Voter ID:server-fa50b62f-5914-cf72-40da-c508a2882fa2 Address:fa50b62f-5914-cf72-40da-c508a2882fa2} {Suffrage:Voter ID:server-f9d6f83f-2682-e217-a8d0-30bac4ef600d Address:f9d6f83f-2682-e217-a8d0-30bac4ef600d}]" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: updating configuration: command=AddStaging server-id=server-f9d6f83f-2682-e217-a8d0-30bac4ef600d server-addr=f9d6f83f-2682-e217-a8d0-30bac4ef600d servers="[{Suffrage:Voter ID:server-06a37545-98a9-5e84-74a8-fc56d794d5b0 Address:06a37545-98a9-5e84-74a8-fc56d794d5b0} {Suffrage:Voter ID:server-fa50b62f-5914-cf72-40da-c508a2882fa2 Address:fa50b62f-5914-cf72-40da-c508a2882fa2} {Suffrage:Voter ID:server-f9d6f83f-2682-e217-a8d0-30bac4ef600d Address:f9d6f83f-2682-e217-a8d0-30bac4ef600d}]" testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: added peer, starting replication: peer=server-f9d6f83f-2682-e217-a8d0-30bac4ef600d testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: added peer, starting replication: peer=server-f9d6f83f-2682-e217-a8d0-30bac4ef600d testLoggerAdapter verbose: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.783Z [WARN] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: 2025-02-09T06:52:26.783Z [WARN] server-f9d6f83f-2682-e217-a8d0-30bac4ef600d: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [WARN] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: appendEntries rejected, sending older logs: peer="{Voter server-f9d6f83f-2682-e217-a8d0-30bac4ef600d f9d6f83f-2682-e217-a8d0-30bac4ef600d}" next=1 testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [WARN] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: appendEntries rejected, sending older logs: peer="{Voter server-f9d6f83f-2682-e217-a8d0-30bac4ef600d f9d6f83f-2682-e217-a8d0-30bac4ef600d}" next=1 testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.783Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:26.783Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: pipelining replication: peer="{Voter server-f9d6f83f-2682-e217-a8d0-30bac4ef600d f9d6f83f-2682-e217-a8d0-30bac4ef600d}" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.783Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: pipelining replication: peer="{Voter server-f9d6f83f-2682-e217-a8d0-30bac4ef600d f9d6f83f-2682-e217-a8d0-30bac4ef600d}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.888Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.783637964 +0000 UTC m=+7.762446727 (1 nodes), 463ns from start of poll, 304.977019ms from cluster start. Timeout at 2025-02-09 06:52:26.88872786 +0000 UTC m=+7.867536619, 105.089892ms after stability testing.go:149: cluster: 2025-02-09T06:52:26.888Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.783637964 +0000 UTC m=+7.762446727 (1 nodes), 463ns from start of poll, 304.977019ms from cluster start. Timeout at 2025-02-09 06:52:26.88872786 +0000 UTC m=+7.867536619, 105.089892ms after stability testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.888Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: aborting pipeline replication: peer="{Voter server-f9d6f83f-2682-e217-a8d0-30bac4ef600d f9d6f83f-2682-e217-a8d0-30bac4ef600d}" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.888Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: aborting pipeline replication: peer="{Voter server-f9d6f83f-2682-e217-a8d0-30bac4ef600d f9d6f83f-2682-e217-a8d0-30bac4ef600d}" testLoggerAdapter verbose: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.888Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: aborting pipeline replication: peer="{Voter server-fa50b62f-5914-cf72-40da-c508a2882fa2 fa50b62f-5914-cf72-40da-c508a2882fa2}" testing.go:149: server-06a37545-98a9-5e84-74a8-fc56d794d5b0: 2025-02-09T06:52:26.888Z [INFO] server-06a37545-98a9-5e84-74a8-fc56d794d5b0: aborting pipeline replication: peer="{Voter server-fa50b62f-5914-cf72-40da-c508a2882fa2 fa50b62f-5914-cf72-40da-c508a2882fa2}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-0f854936-6a58-c968-3e32-225fc0cb9cfd: 2025-02-09T06:52:26.889Z [INFO] server-0f854936-6a58-c968-3e32-225fc0cb9cfd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f854936-6a58-c968-3e32-225fc0cb9cfd Address:0f854936-6a58-c968-3e32-225fc0cb9cfd} {Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testing.go:149: server-0f854936-6a58-c968-3e32-225fc0cb9cfd: 2025-02-09T06:52:26.889Z [INFO] server-0f854936-6a58-c968-3e32-225fc0cb9cfd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f854936-6a58-c968-3e32-225fc0cb9cfd Address:0f854936-6a58-c968-3e32-225fc0cb9cfd} {Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testLoggerAdapter verbose: server-0f854936-6a58-c968-3e32-225fc0cb9cfd: 2025-02-09T06:52:26.889Z [INFO] server-0f854936-6a58-c968-3e32-225fc0cb9cfd: entering follower state: follower="Node at 0f854936-6a58-c968-3e32-225fc0cb9cfd [Follower]" leader= testing.go:149: server-0f854936-6a58-c968-3e32-225fc0cb9cfd: 2025-02-09T06:52:26.889Z [INFO] server-0f854936-6a58-c968-3e32-225fc0cb9cfd: entering follower state: follower="Node at 0f854936-6a58-c968-3e32-225fc0cb9cfd [Follower]" leader= testLoggerAdapter verbose: server-f1dda9c9-5b2b-4c8f-c9da-043485878064: 2025-02-09T06:52:26.889Z [INFO] server-f1dda9c9-5b2b-4c8f-c9da-043485878064: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f854936-6a58-c968-3e32-225fc0cb9cfd Address:0f854936-6a58-c968-3e32-225fc0cb9cfd} {Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testing.go:149: server-f1dda9c9-5b2b-4c8f-c9da-043485878064: 2025-02-09T06:52:26.889Z [INFO] server-f1dda9c9-5b2b-4c8f-c9da-043485878064: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f854936-6a58-c968-3e32-225fc0cb9cfd Address:0f854936-6a58-c968-3e32-225fc0cb9cfd} {Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testLoggerAdapter verbose: server-f1dda9c9-5b2b-4c8f-c9da-043485878064: 2025-02-09T06:52:26.889Z [INFO] server-f1dda9c9-5b2b-4c8f-c9da-043485878064: entering follower state: follower="Node at f1dda9c9-5b2b-4c8f-c9da-043485878064 [Follower]" leader= testing.go:149: server-f1dda9c9-5b2b-4c8f-c9da-043485878064: 2025-02-09T06:52:26.889Z [INFO] server-f1dda9c9-5b2b-4c8f-c9da-043485878064: entering follower state: follower="Node at f1dda9c9-5b2b-4c8f-c9da-043485878064 [Follower]" leader= testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.890Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f854936-6a58-c968-3e32-225fc0cb9cfd Address:0f854936-6a58-c968-3e32-225fc0cb9cfd} {Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.890Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f854936-6a58-c968-3e32-225fc0cb9cfd Address:0f854936-6a58-c968-3e32-225fc0cb9cfd} {Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:26.890Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:26.890Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.890Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: entering follower state: follower="Node at c265bb91-4990-e65e-6aca-05e246ace2a1 [Follower]" leader= testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.890Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: entering follower state: follower="Node at c265bb91-4990-e65e-6aca-05e246ace2a1 [Follower]" leader= testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [WARN] server-c265bb91-4990-e65e-6aca-05e246ace2a1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [WARN] server-c265bb91-4990-e65e-6aca-05e246ace2a1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: entering candidate state: node="Node at c265bb91-4990-e65e-6aca-05e246ace2a1 [Candidate]" term=2 testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: entering candidate state: node="Node at c265bb91-4990-e65e-6aca-05e246ace2a1 [Candidate]" term=2 testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: election won: tally=2 testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: election won: tally=2 testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: entering leader state: leader="Node at c265bb91-4990-e65e-6aca-05e246ace2a1 [Leader]" testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: entering leader state: leader="Node at c265bb91-4990-e65e-6aca-05e246ace2a1 [Leader]" testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: added peer, starting replication: peer=server-0f854936-6a58-c968-3e32-225fc0cb9cfd testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: added peer, starting replication: peer=server-0f854936-6a58-c968-3e32-225fc0cb9cfd testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: added peer, starting replication: peer=server-f1dda9c9-5b2b-4c8f-c9da-043485878064 testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: added peer, starting replication: peer=server-f1dda9c9-5b2b-4c8f-c9da-043485878064 testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: pipelining replication: peer="{Voter server-f1dda9c9-5b2b-4c8f-c9da-043485878064 f1dda9c9-5b2b-4c8f-c9da-043485878064}" testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: pipelining replication: peer="{Voter server-f1dda9c9-5b2b-4c8f-c9da-043485878064 f1dda9c9-5b2b-4c8f-c9da-043485878064}" testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: pipelining replication: peer="{Voter server-0f854936-6a58-c968-3e32-225fc0cb9cfd 0f854936-6a58-c968-3e32-225fc0cb9cfd}" testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:26.969Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: pipelining replication: peer="{Voter server-0f854936-6a58-c968-3e32-225fc0cb9cfd 0f854936-6a58-c968-3e32-225fc0cb9cfd}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.075Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.969841427 +0000 UTC m=+7.948650192 (1 nodes), 79.76955ms from start of poll, 80.172886ms from cluster start. Timeout at 2025-02-09 06:52:27.075767719 +0000 UTC m=+8.054576486, 105.926294ms after stability testing.go:149: cluster: 2025-02-09T06:52:27.075Z [INFO] stable state for Leader reached at 2025-02-09 06:52:26.969841427 +0000 UTC m=+7.948650192 (1 nodes), 79.76955ms from start of poll, 80.172886ms from cluster start. Timeout at 2025-02-09 06:52:27.075767719 +0000 UTC m=+8.054576486, 105.926294ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.081Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:27.081Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.186Z [INFO] stable state for Follower reached at 2025-02-09 06:52:27.081187911 +0000 UTC m=+8.059996674 (2 nodes), 750ns from start of poll, 191.519368ms from cluster start. Timeout at 2025-02-09 06:52:27.186791861 +0000 UTC m=+8.165600618, 105.603944ms after stability testing.go:149: cluster: 2025-02-09T06:52:27.186Z [INFO] stable state for Follower reached at 2025-02-09 06:52:27.081187911 +0000 UTC m=+8.059996674 (2 nodes), 750ns from start of poll, 191.519368ms from cluster start. Timeout at 2025-02-09 06:52:27.186791861 +0000 UTC m=+8.165600618, 105.603944ms after stability testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.186Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: updating configuration: command=RemoveServer server-id=server-0f854936-6a58-c968-3e32-225fc0cb9cfd server-addr= servers="[{Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.186Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: updating configuration: command=RemoveServer server-id=server-0f854936-6a58-c968-3e32-225fc0cb9cfd server-addr= servers="[{Suffrage:Voter ID:server-f1dda9c9-5b2b-4c8f-c9da-043485878064 Address:f1dda9c9-5b2b-4c8f-c9da-043485878064} {Suffrage:Voter ID:server-c265bb91-4990-e65e-6aca-05e246ace2a1 Address:c265bb91-4990-e65e-6aca-05e246ace2a1}]" testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.186Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: removed peer, stopping replication: peer=server-0f854936-6a58-c968-3e32-225fc0cb9cfd last-index=3 testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.186Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: removed peer, stopping replication: peer=server-0f854936-6a58-c968-3e32-225fc0cb9cfd last-index=3 testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.186Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: aborting pipeline replication: peer="{Voter server-0f854936-6a58-c968-3e32-225fc0cb9cfd 0f854936-6a58-c968-3e32-225fc0cb9cfd}" testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.186Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: aborting pipeline replication: peer="{Voter server-0f854936-6a58-c968-3e32-225fc0cb9cfd 0f854936-6a58-c968-3e32-225fc0cb9cfd}" testLoggerAdapter verbose: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.292Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: aborting pipeline replication: peer="{Voter server-f1dda9c9-5b2b-4c8f-c9da-043485878064 f1dda9c9-5b2b-4c8f-c9da-043485878064}" testing.go:149: server-c265bb91-4990-e65e-6aca-05e246ace2a1: 2025-02-09T06:52:27.292Z [INFO] server-c265bb91-4990-e65e-6aca-05e246ace2a1: aborting pipeline replication: peer="{Voter server-f1dda9c9-5b2b-4c8f-c9da-043485878064 f1dda9c9-5b2b-4c8f-c9da-043485878064}" --- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.293Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-4adad467-cf70-40ba-a03f-54e74fe7c051 Address:4adad467-cf70-40ba-a03f-54e74fe7c051} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testing.go:149: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.293Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-4adad467-cf70-40ba-a03f-54e74fe7c051 Address:4adad467-cf70-40ba-a03f-54e74fe7c051} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testLoggerAdapter verbose: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.293Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: entering follower state: follower="Node at ba268597-bfc2-1596-3cdc-c543ef2f8346 [Follower]" leader= testing.go:149: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.293Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: entering follower state: follower="Node at ba268597-bfc2-1596-3cdc-c543ef2f8346 [Follower]" leader= testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.293Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-4adad467-cf70-40ba-a03f-54e74fe7c051 Address:4adad467-cf70-40ba-a03f-54e74fe7c051} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.293Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-4adad467-cf70-40ba-a03f-54e74fe7c051 Address:4adad467-cf70-40ba-a03f-54e74fe7c051} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.293Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: entering follower state: follower="Node at 4adad467-cf70-40ba-a03f-54e74fe7c051 [Follower]" leader= testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.293Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: entering follower state: follower="Node at 4adad467-cf70-40ba-a03f-54e74fe7c051 [Follower]" leader= testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.293Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-4adad467-cf70-40ba-a03f-54e74fe7c051 Address:4adad467-cf70-40ba-a03f-54e74fe7c051} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.293Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-4adad467-cf70-40ba-a03f-54e74fe7c051 Address:4adad467-cf70-40ba-a03f-54e74fe7c051} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.293Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.293Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering follower state: follower="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Follower]" leader= testing.go:149: cluster: 2025-02-09T06:52:27.293Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.293Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering follower state: follower="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Follower]" leader= testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [WARN] server-4adad467-cf70-40ba-a03f-54e74fe7c051: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [WARN] server-4adad467-cf70-40ba-a03f-54e74fe7c051: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: entering candidate state: node="Node at 4adad467-cf70-40ba-a03f-54e74fe7c051 [Candidate]" term=2 testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: entering candidate state: node="Node at 4adad467-cf70-40ba-a03f-54e74fe7c051 [Candidate]" term=2 testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: election won: tally=2 testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: election won: tally=2 testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: entering leader state: leader="Node at 4adad467-cf70-40ba-a03f-54e74fe7c051 [Leader]" testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: entering leader state: leader="Node at 4adad467-cf70-40ba-a03f-54e74fe7c051 [Leader]" testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: added peer, starting replication: peer=server-ba268597-bfc2-1596-3cdc-c543ef2f8346 testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: added peer, starting replication: peer=server-ba268597-bfc2-1596-3cdc-c543ef2f8346 testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: added peer, starting replication: peer=server-8555462e-559d-a74b-2a63-5f27b8de9a78 testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.350Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: added peer, starting replication: peer=server-8555462e-559d-a74b-2a63-5f27b8de9a78 testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.351Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: pipelining replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.351Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: pipelining replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.351Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: pipelining replication: peer="{Voter server-8555462e-559d-a74b-2a63-5f27b8de9a78 8555462e-559d-a74b-2a63-5f27b8de9a78}" testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.351Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: pipelining replication: peer="{Voter server-8555462e-559d-a74b-2a63-5f27b8de9a78 8555462e-559d-a74b-2a63-5f27b8de9a78}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.456Z [INFO] stable state for Leader reached at 2025-02-09 06:52:27.350907673 +0000 UTC m=+8.329716437 (1 nodes), 57.599477ms from start of poll, 57.952249ms from cluster start. Timeout at 2025-02-09 06:52:27.456580153 +0000 UTC m=+8.435388921, 105.672484ms after stability testing.go:149: cluster: 2025-02-09T06:52:27.456Z [INFO] stable state for Leader reached at 2025-02-09 06:52:27.350907673 +0000 UTC m=+8.329716437 (1 nodes), 57.599477ms from start of poll, 57.952249ms from cluster start. Timeout at 2025-02-09 06:52:27.456580153 +0000 UTC m=+8.435388921, 105.672484ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.461Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:27.461Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.567Z [INFO] stable state for Follower reached at 2025-02-09 06:52:27.461986758 +0000 UTC m=+8.440795523 (2 nodes), 650ns from start of poll, 169.031335ms from cluster start. Timeout at 2025-02-09 06:52:27.567482722 +0000 UTC m=+8.546291492, 105.495969ms after stability testing.go:149: cluster: 2025-02-09T06:52:27.567Z [INFO] stable state for Follower reached at 2025-02-09 06:52:27.461986758 +0000 UTC m=+8.440795523 (2 nodes), 650ns from start of poll, 169.031335ms from cluster start. Timeout at 2025-02-09 06:52:27.567482722 +0000 UTC m=+8.546291492, 105.495969ms after stability testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: updating configuration: command=RemoveServer server-id=server-4adad467-cf70-40ba-a03f-54e74fe7c051 server-addr= servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: updating configuration: command=RemoveServer server-id=server-4adad467-cf70-40ba-a03f-54e74fe7c051 server-addr= servers="[{Suffrage:Voter ID:server-ba268597-bfc2-1596-3cdc-c543ef2f8346 Address:ba268597-bfc2-1596-3cdc-c543ef2f8346} {Suffrage:Voter ID:server-8555462e-559d-a74b-2a63-5f27b8de9a78 Address:8555462e-559d-a74b-2a63-5f27b8de9a78}]" testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: removed ourself, shutting down testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: removed ourself, shutting down testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: aborting pipeline replication: peer="{Voter server-8555462e-559d-a74b-2a63-5f27b8de9a78 8555462e-559d-a74b-2a63-5f27b8de9a78}" testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: aborting pipeline replication: peer="{Voter server-8555462e-559d-a74b-2a63-5f27b8de9a78 8555462e-559d-a74b-2a63-5f27b8de9a78}" testLoggerAdapter verbose: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: aborting pipeline replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testing.go:149: server-4adad467-cf70-40ba-a03f-54e74fe7c051: 2025-02-09T06:52:27.567Z [INFO] server-4adad467-cf70-40ba-a03f-54e74fe7c051: aborting pipeline replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.641Z [WARN] server-8555462e-559d-a74b-2a63-5f27b8de9a78: heartbeat timeout reached, starting election: last-leader=4adad467-cf70-40ba-a03f-54e74fe7c051 testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.641Z [WARN] server-8555462e-559d-a74b-2a63-5f27b8de9a78: heartbeat timeout reached, starting election: last-leader=4adad467-cf70-40ba-a03f-54e74fe7c051 testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.641Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering candidate state: node="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Candidate]" term=3 testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.641Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering candidate state: node="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Candidate]" term=3 testLoggerAdapter verbose: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.641Z [WARN] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: rejecting vote request since we have a leader: from=8555462e-559d-a74b-2a63-5f27b8de9a78 leader=4adad467-cf70-40ba-a03f-54e74fe7c051 testing.go:149: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.641Z [WARN] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: rejecting vote request since we have a leader: from=8555462e-559d-a74b-2a63-5f27b8de9a78 leader=4adad467-cf70-40ba-a03f-54e74fe7c051 testLoggerAdapter verbose: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.674Z [WARN] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: heartbeat timeout reached, starting election: last-leader=4adad467-cf70-40ba-a03f-54e74fe7c051 testing.go:149: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.674Z [WARN] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: heartbeat timeout reached, starting election: last-leader=4adad467-cf70-40ba-a03f-54e74fe7c051 testLoggerAdapter verbose: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.674Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: entering candidate state: node="Node at ba268597-bfc2-1596-3cdc-c543ef2f8346 [Candidate]" term=3 testing.go:149: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.674Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: entering candidate state: node="Node at ba268597-bfc2-1596-3cdc-c543ef2f8346 [Candidate]" term=3 testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.674Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: duplicate requestVote for same term: term=3 testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.674Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [WARN] server-8555462e-559d-a74b-2a63-5f27b8de9a78: Election timeout reached, restarting election testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [WARN] server-8555462e-559d-a74b-2a63-5f27b8de9a78: Election timeout reached, restarting election testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering candidate state: node="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Candidate]" term=4 testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering candidate state: node="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Candidate]" term=4 testLoggerAdapter verbose: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.731Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: entering follower state: follower="Node at ba268597-bfc2-1596-3cdc-c543ef2f8346 [Follower]" leader= testing.go:149: server-ba268597-bfc2-1596-3cdc-c543ef2f8346: 2025-02-09T06:52:27.731Z [INFO] server-ba268597-bfc2-1596-3cdc-c543ef2f8346: entering follower state: follower="Node at ba268597-bfc2-1596-3cdc-c543ef2f8346 [Follower]" leader= testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: election won: tally=2 testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: election won: tally=2 testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering leader state: leader="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Leader]" testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: entering leader state: leader="Node at 8555462e-559d-a74b-2a63-5f27b8de9a78 [Leader]" testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: added peer, starting replication: peer=server-ba268597-bfc2-1596-3cdc-c543ef2f8346 testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: added peer, starting replication: peer=server-ba268597-bfc2-1596-3cdc-c543ef2f8346 testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: pipelining replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.731Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: pipelining replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.778Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:27.778Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.884Z [INFO] stable state for Leader reached at 2025-02-09 06:52:27.778862307 +0000 UTC m=+8.757671070 (1 nodes), 21.925µs from start of poll, 485.906882ms from cluster start. Timeout at 2025-02-09 06:52:27.884204181 +0000 UTC m=+8.863012950, 105.34188ms after stability testing.go:149: cluster: 2025-02-09T06:52:27.884Z [INFO] stable state for Leader reached at 2025-02-09 06:52:27.778862307 +0000 UTC m=+8.757671070 (1 nodes), 21.925µs from start of poll, 485.906882ms from cluster start. Timeout at 2025-02-09 06:52:27.884204181 +0000 UTC m=+8.863012950, 105.34188ms after stability testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.884Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: aborting pipeline replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.884Z [INFO] server-8555462e-559d-a74b-2a63-5f27b8de9a78: aborting pipeline replication: peer="{Voter server-ba268597-bfc2-1596-3cdc-c543ef2f8346 ba268597-bfc2-1596-3cdc-c543ef2f8346}" testLoggerAdapter verbose: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.934Z [ERROR] server-8555462e-559d-a74b-2a63-5f27b8de9a78: failed to heartbeat to: peer=ba268597-bfc2-1596-3cdc-c543ef2f8346 error="command timed out" testing.go:149: server-8555462e-559d-a74b-2a63-5f27b8de9a78: 2025-02-09T06:52:27.934Z [ERROR] server-8555462e-559d-a74b-2a63-5f27b8de9a78: failed to heartbeat to: peer=ba268597-bfc2-1596-3cdc-c543ef2f8346 error="command timed out" --- PASS: TestRaft_RemoveLeader (0.64s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:27.935Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-0838c6e9-239d-13e4-fd19-9e5d97774dfa Address:0838c6e9-239d-13e4-fd19-9e5d97774dfa} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testing.go:149: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:27.935Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-0838c6e9-239d-13e4-fd19-9e5d97774dfa Address:0838c6e9-239d-13e4-fd19-9e5d97774dfa} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testLoggerAdapter verbose: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:27.935Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: entering follower state: follower="Node at c677d058-06e6-6ad9-91fa-9babc09d6d1c [Follower]" leader= testing.go:149: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:27.935Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: entering follower state: follower="Node at c677d058-06e6-6ad9-91fa-9babc09d6d1c [Follower]" leader= testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.935Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-0838c6e9-239d-13e4-fd19-9e5d97774dfa Address:0838c6e9-239d-13e4-fd19-9e5d97774dfa} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.935Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-0838c6e9-239d-13e4-fd19-9e5d97774dfa Address:0838c6e9-239d-13e4-fd19-9e5d97774dfa} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.935Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering follower state: follower="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Follower]" leader= testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.935Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering follower state: follower="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Follower]" leader= testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:27.935Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-0838c6e9-239d-13e4-fd19-9e5d97774dfa Address:0838c6e9-239d-13e4-fd19-9e5d97774dfa} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:27.935Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-0838c6e9-239d-13e4-fd19-9e5d97774dfa Address:0838c6e9-239d-13e4-fd19-9e5d97774dfa} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:27.935Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:27.935Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:27.935Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering follower state: follower="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Follower]" leader= testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:27.935Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering follower state: follower="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Follower]" leader= testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.989Z [WARN] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.989Z [WARN] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.989Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering candidate state: node="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Candidate]" term=2 testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.989Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering candidate state: node="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Candidate]" term=2 testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: election won: tally=2 testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: election won: tally=2 testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering leader state: leader="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Leader]" testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering leader state: leader="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Leader]" testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: added peer, starting replication: peer=server-c677d058-06e6-6ad9-91fa-9babc09d6d1c testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: added peer, starting replication: peer=server-c677d058-06e6-6ad9-91fa-9babc09d6d1c testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: added peer, starting replication: peer=server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: added peer, starting replication: peer=server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: pipelining replication: peer="{Voter server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 ae580321-9259-4b53-4ab2-b11f6c17aaa8}" testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: pipelining replication: peer="{Voter server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 ae580321-9259-4b53-4ab2-b11f6c17aaa8}" testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: pipelining replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:27.990Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: pipelining replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.095Z [INFO] stable state for Follower reached at 2025-02-09 06:52:27.990103778 +0000 UTC m=+8.968912543 (2 nodes), 54.473464ms from start of poll, 54.853943ms from cluster start. Timeout at 2025-02-09 06:52:28.095424295 +0000 UTC m=+9.074233052, 105.320509ms after stability testing.go:149: cluster: 2025-02-09T06:52:28.095Z [INFO] stable state for Follower reached at 2025-02-09 06:52:27.990103778 +0000 UTC m=+8.968912543 (2 nodes), 54.473464ms from start of poll, 54.853943ms from cluster start. Timeout at 2025-02-09 06:52:28.095424295 +0000 UTC m=+9.074233052, 105.320509ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.095Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:28.095Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.200Z [INFO] stable state for Leader reached at 2025-02-09 06:52:28.095485378 +0000 UTC m=+9.074294144 (1 nodes), 465ns from start of poll, 160.235544ms from cluster start. Timeout at 2025-02-09 06:52:28.200883484 +0000 UTC m=+9.179692252, 105.398108ms after stability testing.go:149: cluster: 2025-02-09T06:52:28.200Z [INFO] stable state for Leader reached at 2025-02-09 06:52:28.095485378 +0000 UTC m=+9.074294144 (1 nodes), 465ns from start of poll, 160.235544ms from cluster start. Timeout at 2025-02-09 06:52:28.200883484 +0000 UTC m=+9.179692252, 105.398108ms after stability testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.201Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: updating configuration: command=RemoveServer server-id=server-0838c6e9-239d-13e4-fd19-9e5d97774dfa server-addr= servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.201Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: updating configuration: command=RemoveServer server-id=server-0838c6e9-239d-13e4-fd19-9e5d97774dfa server-addr= servers="[{Suffrage:Voter ID:server-c677d058-06e6-6ad9-91fa-9babc09d6d1c Address:c677d058-06e6-6ad9-91fa-9babc09d6d1c} {Suffrage:Voter ID:server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 Address:ae580321-9259-4b53-4ab2-b11f6c17aaa8}]" testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: removed ourself, transitioning to follower testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: removed ourself, transitioning to follower testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering follower state: follower="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Follower]" leader= testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: entering follower state: follower="Node at 0838c6e9-239d-13e4-fd19-9e5d97774dfa [Follower]" leader= testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: aborting pipeline replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: aborting pipeline replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: aborting pipeline replication: peer="{Voter server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 ae580321-9259-4b53-4ab2-b11f6c17aaa8}" testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.202Z [INFO] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: aborting pipeline replication: peer="{Voter server-ae580321-9259-4b53-4ab2-b11f6c17aaa8 ae580321-9259-4b53-4ab2-b11f6c17aaa8}" testLoggerAdapter verbose: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.293Z [WARN] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: not part of stable configuration, aborting election testing.go:149: server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: 2025-02-09T06:52:28.293Z [WARN] server-0838c6e9-239d-13e4-fd19-9e5d97774dfa: not part of stable configuration, aborting election testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.298Z [WARN] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: heartbeat timeout reached, starting election: last-leader=0838c6e9-239d-13e4-fd19-9e5d97774dfa testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.298Z [WARN] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: heartbeat timeout reached, starting election: last-leader=0838c6e9-239d-13e4-fd19-9e5d97774dfa testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.298Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering candidate state: node="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Candidate]" term=3 testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.298Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering candidate state: node="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Candidate]" term=3 testLoggerAdapter verbose: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.298Z [WARN] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: rejecting vote request since we have a leader: from=ae580321-9259-4b53-4ab2-b11f6c17aaa8 leader=0838c6e9-239d-13e4-fd19-9e5d97774dfa testing.go:149: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.298Z [WARN] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: rejecting vote request since we have a leader: from=ae580321-9259-4b53-4ab2-b11f6c17aaa8 leader=0838c6e9-239d-13e4-fd19-9e5d97774dfa testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.307Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:28.307Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.308Z [WARN] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: heartbeat timeout reached, starting election: last-leader=0838c6e9-239d-13e4-fd19-9e5d97774dfa testing.go:149: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.308Z [WARN] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: heartbeat timeout reached, starting election: last-leader=0838c6e9-239d-13e4-fd19-9e5d97774dfa testLoggerAdapter verbose: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.308Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: entering candidate state: node="Node at c677d058-06e6-6ad9-91fa-9babc09d6d1c [Candidate]" term=3 testing.go:149: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.308Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: entering candidate state: node="Node at c677d058-06e6-6ad9-91fa-9babc09d6d1c [Candidate]" term=3 testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.308Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: duplicate requestVote for same term: term=3 testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.308Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.348Z [WARN] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: Election timeout reached, restarting election testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.348Z [WARN] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: Election timeout reached, restarting election testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.348Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering candidate state: node="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Candidate]" term=4 testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.348Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering candidate state: node="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Candidate]" term=4 testLoggerAdapter verbose: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.349Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: entering follower state: follower="Node at c677d058-06e6-6ad9-91fa-9babc09d6d1c [Follower]" leader= testing.go:149: server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: 2025-02-09T06:52:28.349Z [INFO] server-c677d058-06e6-6ad9-91fa-9babc09d6d1c: entering follower state: follower="Node at c677d058-06e6-6ad9-91fa-9babc09d6d1c [Follower]" leader= testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: election won: tally=2 testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: election won: tally=2 testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering leader state: leader="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Leader]" testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: entering leader state: leader="Node at ae580321-9259-4b53-4ab2-b11f6c17aaa8 [Leader]" testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: added peer, starting replication: peer=server-c677d058-06e6-6ad9-91fa-9babc09d6d1c testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: added peer, starting replication: peer=server-c677d058-06e6-6ad9-91fa-9babc09d6d1c testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: pipelining replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.349Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: pipelining replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.454Z [INFO] stable state for Leader reached at 2025-02-09 06:52:28.3490993 +0000 UTC m=+9.327908075 (1 nodes), 41.599008ms from start of poll, 413.849475ms from cluster start. Timeout at 2025-02-09 06:52:28.454773371 +0000 UTC m=+9.433582124, 105.674049ms after stability testing.go:149: cluster: 2025-02-09T06:52:28.454Z [INFO] stable state for Leader reached at 2025-02-09 06:52:28.3490993 +0000 UTC m=+9.327908075 (1 nodes), 41.599008ms from start of poll, 413.849475ms from cluster start. Timeout at 2025-02-09 06:52:28.454773371 +0000 UTC m=+9.433582124, 105.674049ms after stability testLoggerAdapter verbose: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.560Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: aborting pipeline replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" testing.go:149: server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: 2025-02-09T06:52:28.560Z [INFO] server-ae580321-9259-4b53-4ab2-b11f6c17aaa8: aborting pipeline replication: peer="{Voter server-c677d058-06e6-6ad9-91fa-9babc09d6d1c c677d058-06e6-6ad9-91fa-9babc09d6d1c}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.561Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.561Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.561Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering follower state: follower="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Follower]" leader= testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.561Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering follower state: follower="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Follower]" leader= testLoggerAdapter verbose: server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: 2025-02-09T06:52:28.561Z [INFO] server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testing.go:149: server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: 2025-02-09T06:52:28.561Z [INFO] server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testLoggerAdapter verbose: server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: 2025-02-09T06:52:28.561Z [INFO] server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: entering follower state: follower="Node at f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 [Follower]" leader= testing.go:149: server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: 2025-02-09T06:52:28.561Z [INFO] server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: entering follower state: follower="Node at f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 [Follower]" leader= testLoggerAdapter verbose: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.561Z [INFO] server-bb88d534-4229-922a-decb-be8f58a65ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testing.go:149: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.561Z [INFO] server-bb88d534-4229-922a-decb-be8f58a65ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testLoggerAdapter verbose: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.561Z [INFO] server-bb88d534-4229-922a-decb-be8f58a65ec8: entering follower state: follower="Node at bb88d534-4229-922a-decb-be8f58a65ec8 [Follower]" leader= testing.go:149: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.561Z [INFO] server-bb88d534-4229-922a-decb-be8f58a65ec8: entering follower state: follower="Node at bb88d534-4229-922a-decb-be8f58a65ec8 [Follower]" leader= testLoggerAdapter verbose: server-73d42892-293a-5762-0208-8b9d8e465179: 2025-02-09T06:52:28.561Z [INFO] server-73d42892-293a-5762-0208-8b9d8e465179: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testing.go:149: server-73d42892-293a-5762-0208-8b9d8e465179: 2025-02-09T06:52:28.561Z [INFO] server-73d42892-293a-5762-0208-8b9d8e465179: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8} {Suffrage:Voter ID:server-73d42892-293a-5762-0208-8b9d8e465179 Address:73d42892-293a-5762-0208-8b9d8e465179}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.561Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:28.561Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-73d42892-293a-5762-0208-8b9d8e465179: 2025-02-09T06:52:28.561Z [INFO] server-73d42892-293a-5762-0208-8b9d8e465179: entering follower state: follower="Node at 73d42892-293a-5762-0208-8b9d8e465179 [Follower]" leader= testing.go:149: server-73d42892-293a-5762-0208-8b9d8e465179: 2025-02-09T06:52:28.561Z [INFO] server-73d42892-293a-5762-0208-8b9d8e465179: entering follower state: follower="Node at 73d42892-293a-5762-0208-8b9d8e465179 [Follower]" leader= testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering candidate state: node="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Candidate]" term=2 testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering candidate state: node="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Candidate]" term=2 testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: election won: tally=3 testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: election won: tally=3 testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering leader state: leader="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Leader]" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering leader state: leader="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Leader]" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: added peer, starting replication: peer=server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: added peer, starting replication: peer=server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: added peer, starting replication: peer=server-bb88d534-4229-922a-decb-be8f58a65ec8 testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: added peer, starting replication: peer=server-bb88d534-4229-922a-decb-be8f58a65ec8 testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: added peer, starting replication: peer=server-73d42892-293a-5762-0208-8b9d8e465179 testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: added peer, starting replication: peer=server-73d42892-293a-5762-0208-8b9d8e465179 testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: pipelining replication: peer="{Voter server-73d42892-293a-5762-0208-8b9d8e465179 73d42892-293a-5762-0208-8b9d8e465179}" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: pipelining replication: peer="{Voter server-73d42892-293a-5762-0208-8b9d8e465179 73d42892-293a-5762-0208-8b9d8e465179}" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: pipelining replication: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: pipelining replication: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: pipelining replication: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.624Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: pipelining replication: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.729Z [INFO] stable state for Leader reached at 2025-02-09 06:52:28.62439515 +0000 UTC m=+9.603203925 (1 nodes), 62.875896ms from start of poll, 63.319865ms from cluster start. Timeout at 2025-02-09 06:52:28.729943959 +0000 UTC m=+9.708752730, 105.548805ms after stability testing.go:149: cluster: 2025-02-09T06:52:28.729Z [INFO] stable state for Leader reached at 2025-02-09 06:52:28.62439515 +0000 UTC m=+9.603203925 (1 nodes), 62.875896ms from start of poll, 63.319865ms from cluster start. Timeout at 2025-02-09 06:52:28.729943959 +0000 UTC m=+9.708752730, 105.548805ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.835Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:28.835Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.940Z [INFO] stable state for Follower reached at 2025-02-09 06:52:28.835471061 +0000 UTC m=+9.814279827 (3 nodes), 908ns from start of poll, 274.395767ms from cluster start. Timeout at 2025-02-09 06:52:28.94065374 +0000 UTC m=+9.919462497, 105.18267ms after stability testing.go:149: cluster: 2025-02-09T06:52:28.940Z [INFO] stable state for Follower reached at 2025-02-09 06:52:28.835471061 +0000 UTC m=+9.814279827 (3 nodes), 908ns from start of poll, 274.395767ms from cluster start. Timeout at 2025-02-09 06:52:28.94065374 +0000 UTC m=+9.919462497, 105.18267ms after stability testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: updating configuration: command=RemoveServer server-id=server-73d42892-293a-5762-0208-8b9d8e465179 server-addr= servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8}]" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: updating configuration: command=RemoveServer server-id=server-73d42892-293a-5762-0208-8b9d8e465179 server-addr= servers="[{Suffrage:Voter ID:server-11d28c8d-42bd-914e-6e33-db5fae092536 Address:11d28c8d-42bd-914e-6e33-db5fae092536} {Suffrage:Voter ID:server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 Address:f9713e63-dadc-5a41-10b5-a8cb7d3e02c2} {Suffrage:Voter ID:server-bb88d534-4229-922a-decb-be8f58a65ec8 Address:bb88d534-4229-922a-decb-be8f58a65ec8}]" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: removed peer, stopping replication: peer=server-73d42892-293a-5762-0208-8b9d8e465179 last-index=3 testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: removed peer, stopping replication: peer=server-73d42892-293a-5762-0208-8b9d8e465179 last-index=3 testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to pipeline appendEntries: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="append pipeline closed" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to pipeline appendEntries: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="append pipeline closed" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: aborting pipeline replication: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: aborting pipeline replication: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to pipeline appendEntries: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="append pipeline closed" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to pipeline appendEntries: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="append pipeline closed" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: aborting pipeline replication: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: aborting pipeline replication: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: aborting pipeline replication: peer="{Voter server-73d42892-293a-5762-0208-8b9d8e465179 73d42892-293a-5762-0208-8b9d8e465179}" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.940Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: aborting pipeline replication: peer="{Voter server-73d42892-293a-5762-0208-8b9d8e465179 73d42892-293a-5762-0208-8b9d8e465179}" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.945Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.945Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.945Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=bb88d534-4229-922a-decb-be8f58a65ec8 error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.945Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=bb88d534-4229-922a-decb-be8f58a65ec8 error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.950Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.950Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.950Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.950Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.962Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.962Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.965Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=bb88d534-4229-922a-decb-be8f58a65ec8 error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.965Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=bb88d534-4229-922a-decb-be8f58a65ec8 error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.967Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.967Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.968Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.968Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.979Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.979Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.985Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=bb88d534-4229-922a-decb-be8f58a65ec8 error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.985Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to heartbeat to: peer=bb88d534-4229-922a-decb-be8f58a65ec8 error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.985Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.985Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.985Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact: server-id=server-bb88d534-4229-922a-decb-be8f58a65ec8 time=50.464849ms testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.985Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact: server-id=server-bb88d534-4229-922a-decb-be8f58a65ec8 time=50.464849ms testLoggerAdapter verbose: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.987Z [WARN] server-bb88d534-4229-922a-decb-be8f58a65ec8: heartbeat timeout reached, starting election: last-leader=11d28c8d-42bd-914e-6e33-db5fae092536 testing.go:149: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.987Z [WARN] server-bb88d534-4229-922a-decb-be8f58a65ec8: heartbeat timeout reached, starting election: last-leader=11d28c8d-42bd-914e-6e33-db5fae092536 testLoggerAdapter verbose: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.987Z [INFO] server-bb88d534-4229-922a-decb-be8f58a65ec8: entering candidate state: node="Node at bb88d534-4229-922a-decb-be8f58a65ec8 [Candidate]" term=3 testing.go:149: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.987Z [INFO] server-bb88d534-4229-922a-decb-be8f58a65ec8: entering candidate state: node="Node at bb88d534-4229-922a-decb-be8f58a65ec8 [Candidate]" term=3 testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.987Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.987Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testLoggerAdapter verbose: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.988Z [ERROR] server-bb88d534-4229-922a-decb-be8f58a65ec8: failed to make requestVote RPC: target="{Voter server-73d42892-293a-5762-0208-8b9d8e465179 73d42892-293a-5762-0208-8b9d8e465179}" error="failed to connect to peer: 73d42892-293a-5762-0208-8b9d8e465179" testing.go:149: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.988Z [ERROR] server-bb88d534-4229-922a-decb-be8f58a65ec8: failed to make requestVote RPC: target="{Voter server-73d42892-293a-5762-0208-8b9d8e465179 73d42892-293a-5762-0208-8b9d8e465179}" error="failed to connect to peer: 73d42892-293a-5762-0208-8b9d8e465179" testLoggerAdapter verbose: server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: 2025-02-09T06:52:28.988Z [WARN] server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: rejecting vote request since we have a leader: from=bb88d534-4229-922a-decb-be8f58a65ec8 leader=11d28c8d-42bd-914e-6e33-db5fae092536 testLoggerAdapter verbose: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.988Z [ERROR] server-bb88d534-4229-922a-decb-be8f58a65ec8: failed to make requestVote RPC: target="{Voter server-11d28c8d-42bd-914e-6e33-db5fae092536 11d28c8d-42bd-914e-6e33-db5fae092536}" error="failed to connect to peer: 11d28c8d-42bd-914e-6e33-db5fae092536" testing.go:149: server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: 2025-02-09T06:52:28.988Z [WARN] server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2: rejecting vote request since we have a leader: from=bb88d534-4229-922a-decb-be8f58a65ec8 leader=11d28c8d-42bd-914e-6e33-db5fae092536 testing.go:149: server-bb88d534-4229-922a-decb-be8f58a65ec8: 2025-02-09T06:52:28.988Z [ERROR] server-bb88d534-4229-922a-decb-be8f58a65ec8: failed to make requestVote RPC: target="{Voter server-11d28c8d-42bd-914e-6e33-db5fae092536 11d28c8d-42bd-914e-6e33-db5fae092536}" error="failed to connect to peer: 11d28c8d-42bd-914e-6e33-db5fae092536" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact: server-id=server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 time=56.938455ms testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact: server-id=server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 time=56.938455ms testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact: server-id=server-bb88d534-4229-922a-decb-be8f58a65ec8 time=61.232702ms testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact: server-id=server-bb88d534-4229-922a-decb-be8f58a65ec8 time=61.232702ms testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact quorum of nodes, stepping down testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [WARN] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering follower state: follower="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Follower]" leader= testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [INFO] server-11d28c8d-42bd-914e-6e33-db5fae092536: entering follower state: follower="Node at 11d28c8d-42bd-914e-6e33-db5fae092536 [Follower]" leader= testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-f9713e63-dadc-5a41-10b5-a8cb7d3e02c2 f9713e63-dadc-5a41-10b5-a8cb7d3e02c2}" error="failed to connect to peer: f9713e63-dadc-5a41-10b5-a8cb7d3e02c2" testLoggerAdapter verbose: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" testing.go:149: server-11d28c8d-42bd-914e-6e33-db5fae092536: 2025-02-09T06:52:28.996Z [ERROR] server-11d28c8d-42bd-914e-6e33-db5fae092536: failed to appendEntries to: peer="{Voter server-bb88d534-4229-922a-decb-be8f58a65ec8 bb88d534-4229-922a-decb-be8f58a65ec8}" error="failed to connect to peer: bb88d534-4229-922a-decb-be8f58a65ec8" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:28.997Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testing.go:149: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:28.997Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testLoggerAdapter verbose: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:28.997Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: entering follower state: follower="Node at 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c [Follower]" leader= testing.go:149: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:28.997Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: entering follower state: follower="Node at 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c [Follower]" leader= testLoggerAdapter verbose: server-fd343c0f-d0c9-813c-8b62-827445bc927e: 2025-02-09T06:52:28.997Z [INFO] server-fd343c0f-d0c9-813c-8b62-827445bc927e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testing.go:149: server-fd343c0f-d0c9-813c-8b62-827445bc927e: 2025-02-09T06:52:28.997Z [INFO] server-fd343c0f-d0c9-813c-8b62-827445bc927e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testLoggerAdapter verbose: server-fd343c0f-d0c9-813c-8b62-827445bc927e: 2025-02-09T06:52:28.997Z [INFO] server-fd343c0f-d0c9-813c-8b62-827445bc927e: entering follower state: follower="Node at fd343c0f-d0c9-813c-8b62-827445bc927e [Follower]" leader= testing.go:149: server-fd343c0f-d0c9-813c-8b62-827445bc927e: 2025-02-09T06:52:28.997Z [INFO] server-fd343c0f-d0c9-813c-8b62-827445bc927e: entering follower state: follower="Node at fd343c0f-d0c9-813c-8b62-827445bc927e [Follower]" leader= testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:28.998Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:28.998Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:28.998Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:28.998Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:28.998Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: entering follower state: follower="Node at bf17a376-c094-ea84-3489-52ad4bf33ff0 [Follower]" leader= testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:28.998Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: entering follower state: follower="Node at bf17a376-c094-ea84-3489-52ad4bf33ff0 [Follower]" leader= testLoggerAdapter verbose: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.058Z [WARN] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [WARN] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.058Z [WARN] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.058Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: entering candidate state: node="Node at 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c [Candidate]" term=2 testing.go:149: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.058Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: entering candidate state: node="Node at 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c [Candidate]" term=2 testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [WARN] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: entering candidate state: node="Node at bf17a376-c094-ea84-3489-52ad4bf33ff0 [Candidate]" term=2 testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: entering candidate state: node="Node at bf17a376-c094-ea84-3489-52ad4bf33ff0 [Candidate]" term=2 testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.058Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: duplicate requestVote for same term: term=2 testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: duplicate requestVote for same term: term=2 testing.go:149: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.058Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: election won: tally=2 testLoggerAdapter verbose: server-fd343c0f-d0c9-813c-8b62-827445bc927e: 2025-02-09T06:52:29.058Z [INFO] server-fd343c0f-d0c9-813c-8b62-827445bc927e: duplicate requestVote for same term: term=2 testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: election won: tally=2 testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: entering leader state: leader="Node at bf17a376-c094-ea84-3489-52ad4bf33ff0 [Leader]" testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: entering leader state: leader="Node at bf17a376-c094-ea84-3489-52ad4bf33ff0 [Leader]" testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: added peer, starting replication: peer=server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.058Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: added peer, starting replication: peer=server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.059Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: added peer, starting replication: peer=server-fd343c0f-d0c9-813c-8b62-827445bc927e testing.go:149: server-fd343c0f-d0c9-813c-8b62-827445bc927e: 2025-02-09T06:52:29.058Z [INFO] server-fd343c0f-d0c9-813c-8b62-827445bc927e: duplicate requestVote for same term: term=2 testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.059Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: added peer, starting replication: peer=server-fd343c0f-d0c9-813c-8b62-827445bc927e testLoggerAdapter verbose: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.059Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: entering follower state: follower="Node at 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c [Follower]" leader=bf17a376-c094-ea84-3489-52ad4bf33ff0 testing.go:149: server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: 2025-02-09T06:52:29.059Z [INFO] server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c: entering follower state: follower="Node at 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c [Follower]" leader=bf17a376-c094-ea84-3489-52ad4bf33ff0 testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.059Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: pipelining replication: peer="{Voter server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c}" testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.059Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: pipelining replication: peer="{Voter server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c}" testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.059Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: pipelining replication: peer="{Voter server-fd343c0f-d0c9-813c-8b62-827445bc927e fd343c0f-d0c9-813c-8b62-827445bc927e}" testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.059Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: pipelining replication: peer="{Voter server-fd343c0f-d0c9-813c-8b62-827445bc927e fd343c0f-d0c9-813c-8b62-827445bc927e}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.164Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.059073297 +0000 UTC m=+10.037882148 (1 nodes), 60.980653ms from start of poll, 61.357755ms from cluster start. Timeout at 2025-02-09 06:52:29.164080528 +0000 UTC m=+10.142889295, 105.007147ms after stability testing.go:149: cluster: 2025-02-09T06:52:29.164Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.059073297 +0000 UTC m=+10.037882148 (1 nodes), 60.980653ms from start of poll, 61.357755ms from cluster start. Timeout at 2025-02-09 06:52:29.164080528 +0000 UTC m=+10.142889295, 105.007147ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.164Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:29.164Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.269Z [INFO] stable state for Follower reached at 2025-02-09 06:52:29.164164019 +0000 UTC m=+10.142972799 (2 nodes), 707ns from start of poll, 166.448406ms from cluster start. Timeout at 2025-02-09 06:52:29.269947495 +0000 UTC m=+10.248756262, 105.783463ms after stability testing.go:149: cluster: 2025-02-09T06:52:29.269Z [INFO] stable state for Follower reached at 2025-02-09 06:52:29.164164019 +0000 UTC m=+10.142972799 (2 nodes), 707ns from start of poll, 166.448406ms from cluster start. Timeout at 2025-02-09 06:52:29.269947495 +0000 UTC m=+10.248756262, 105.783463ms after stability testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.270Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: updating configuration: command=AddStaging server-id=server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c server-addr=2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.270Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: updating configuration: command=AddStaging server-id=server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c server-addr=2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c servers="[{Suffrage:Voter ID:server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c Address:2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c} {Suffrage:Voter ID:server-fd343c0f-d0c9-813c-8b62-827445bc927e Address:fd343c0f-d0c9-813c-8b62-827445bc927e} {Suffrage:Voter ID:server-bf17a376-c094-ea84-3489-52ad4bf33ff0 Address:bf17a376-c094-ea84-3489-52ad4bf33ff0}]" testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.270Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: aborting pipeline replication: peer="{Voter server-fd343c0f-d0c9-813c-8b62-827445bc927e fd343c0f-d0c9-813c-8b62-827445bc927e}" testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.270Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: aborting pipeline replication: peer="{Voter server-fd343c0f-d0c9-813c-8b62-827445bc927e fd343c0f-d0c9-813c-8b62-827445bc927e}" testLoggerAdapter verbose: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.270Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: aborting pipeline replication: peer="{Voter server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c}" testing.go:149: server-bf17a376-c094-ea84-3489-52ad4bf33ff0: 2025-02-09T06:52:29.270Z [INFO] server-bf17a376-c094-ea84-3489-52ad4bf33ff0: aborting pipeline replication: peer="{Voter server-2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c 2f6e6abf-f2b1-4fb9-b4d6-1bc6cbb7315c}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-f73e93b4-e099-1e0b-8a78-0822efda467b: 2025-02-09T06:52:29.271Z [INFO] server-f73e93b4-e099-1e0b-8a78-0822efda467b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testing.go:149: server-f73e93b4-e099-1e0b-8a78-0822efda467b: 2025-02-09T06:52:29.271Z [INFO] server-f73e93b4-e099-1e0b-8a78-0822efda467b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testLoggerAdapter verbose: server-f73e93b4-e099-1e0b-8a78-0822efda467b: 2025-02-09T06:52:29.271Z [INFO] server-f73e93b4-e099-1e0b-8a78-0822efda467b: entering follower state: follower="Node at f73e93b4-e099-1e0b-8a78-0822efda467b [Follower]" leader= testing.go:149: server-f73e93b4-e099-1e0b-8a78-0822efda467b: 2025-02-09T06:52:29.271Z [INFO] server-f73e93b4-e099-1e0b-8a78-0822efda467b: entering follower state: follower="Node at f73e93b4-e099-1e0b-8a78-0822efda467b [Follower]" leader= testLoggerAdapter verbose: server-dcbaa381-966b-94be-6265-3e90ce55e628: 2025-02-09T06:52:29.271Z [INFO] server-dcbaa381-966b-94be-6265-3e90ce55e628: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testing.go:149: server-dcbaa381-966b-94be-6265-3e90ce55e628: 2025-02-09T06:52:29.271Z [INFO] server-dcbaa381-966b-94be-6265-3e90ce55e628: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testLoggerAdapter verbose: server-dcbaa381-966b-94be-6265-3e90ce55e628: 2025-02-09T06:52:29.271Z [INFO] server-dcbaa381-966b-94be-6265-3e90ce55e628: entering follower state: follower="Node at dcbaa381-966b-94be-6265-3e90ce55e628 [Follower]" leader= testing.go:149: server-dcbaa381-966b-94be-6265-3e90ce55e628: 2025-02-09T06:52:29.271Z [INFO] server-dcbaa381-966b-94be-6265-3e90ce55e628: entering follower state: follower="Node at dcbaa381-966b-94be-6265-3e90ce55e628 [Follower]" leader= testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.271Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.271Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.271Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:29.271Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.271Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: entering follower state: follower="Node at da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 [Follower]" leader= testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.271Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: entering follower state: follower="Node at da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 [Follower]" leader= testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [WARN] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [WARN] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: entering candidate state: node="Node at da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 [Candidate]" term=2 testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: entering candidate state: node="Node at da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 [Candidate]" term=2 testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: election won: tally=2 testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: election won: tally=2 testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: entering leader state: leader="Node at da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 [Leader]" testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: entering leader state: leader="Node at da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 [Leader]" testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: added peer, starting replication: peer=server-f73e93b4-e099-1e0b-8a78-0822efda467b testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: added peer, starting replication: peer=server-f73e93b4-e099-1e0b-8a78-0822efda467b testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: added peer, starting replication: peer=server-dcbaa381-966b-94be-6265-3e90ce55e628 testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: added peer, starting replication: peer=server-dcbaa381-966b-94be-6265-3e90ce55e628 testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: pipelining replication: peer="{Voter server-f73e93b4-e099-1e0b-8a78-0822efda467b f73e93b4-e099-1e0b-8a78-0822efda467b}" testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: pipelining replication: peer="{Voter server-f73e93b4-e099-1e0b-8a78-0822efda467b f73e93b4-e099-1e0b-8a78-0822efda467b}" testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: pipelining replication: peer="{Voter server-dcbaa381-966b-94be-6265-3e90ce55e628 dcbaa381-966b-94be-6265-3e90ce55e628}" testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.356Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: pipelining replication: peer="{Voter server-dcbaa381-966b-94be-6265-3e90ce55e628 dcbaa381-966b-94be-6265-3e90ce55e628}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.462Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.356946146 +0000 UTC m=+10.335754914 (1 nodes), 85.581788ms from start of poll, 85.925401ms from cluster start. Timeout at 2025-02-09 06:52:29.462875108 +0000 UTC m=+10.441683868, 105.928954ms after stability testing.go:149: cluster: 2025-02-09T06:52:29.462Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.356946146 +0000 UTC m=+10.335754914 (1 nodes), 85.581788ms from start of poll, 85.925401ms from cluster start. Timeout at 2025-02-09 06:52:29.462875108 +0000 UTC m=+10.441683868, 105.928954ms after stability testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.462Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: updating configuration: command=RemoveServer server-id=a3ad6b08-52dd-0964-96b0-003eb423b83a server-addr= servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.462Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: updating configuration: command=RemoveServer server-id=a3ad6b08-52dd-0964-96b0-003eb423b83a server-addr= servers="[{Suffrage:Voter ID:server-f73e93b4-e099-1e0b-8a78-0822efda467b Address:f73e93b4-e099-1e0b-8a78-0822efda467b} {Suffrage:Voter ID:server-dcbaa381-966b-94be-6265-3e90ce55e628 Address:dcbaa381-966b-94be-6265-3e90ce55e628} {Suffrage:Voter ID:server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76 Address:da29ab7d-22f8-62ba-ec4c-32a6ac3cec76}]" testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.463Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: aborting pipeline replication: peer="{Voter server-dcbaa381-966b-94be-6265-3e90ce55e628 dcbaa381-966b-94be-6265-3e90ce55e628}" testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.463Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: aborting pipeline replication: peer="{Voter server-dcbaa381-966b-94be-6265-3e90ce55e628 dcbaa381-966b-94be-6265-3e90ce55e628}" testLoggerAdapter verbose: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.463Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: aborting pipeline replication: peer="{Voter server-f73e93b4-e099-1e0b-8a78-0822efda467b f73e93b4-e099-1e0b-8a78-0822efda467b}" testing.go:149: server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: 2025-02-09T06:52:29.463Z [INFO] server-da29ab7d-22f8-62ba-ec4c-32a6ac3cec76: aborting pipeline replication: peer="{Voter server-f73e93b4-e099-1e0b-8a78-0822efda467b f73e93b4-e099-1e0b-8a78-0822efda467b}" --- PASS: TestRaft_RemoveUnknownPeer (0.19s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.463Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85476a8a-6632-bf20-7fe8-c1a95944264a Address:85476a8a-6632-bf20-7fe8-c1a95944264a}]" testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.463Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85476a8a-6632-bf20-7fe8-c1a95944264a Address:85476a8a-6632-bf20-7fe8-c1a95944264a}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.463Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:29.463Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.463Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: entering follower state: follower="Node at 85476a8a-6632-bf20-7fe8-c1a95944264a [Follower]" leader= testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.463Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: entering follower state: follower="Node at 85476a8a-6632-bf20-7fe8-c1a95944264a [Follower]" leader= testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [WARN] server-85476a8a-6632-bf20-7fe8-c1a95944264a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [WARN] server-85476a8a-6632-bf20-7fe8-c1a95944264a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: entering candidate state: node="Node at 85476a8a-6632-bf20-7fe8-c1a95944264a [Candidate]" term=2 testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: entering candidate state: node="Node at 85476a8a-6632-bf20-7fe8-c1a95944264a [Candidate]" term=2 testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: election won: tally=1 testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: election won: tally=1 testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: entering leader state: leader="Node at 85476a8a-6632-bf20-7fe8-c1a95944264a [Leader]" testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.552Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: entering leader state: leader="Node at 85476a8a-6632-bf20-7fe8-c1a95944264a [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.658Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.552957648 +0000 UTC m=+10.531766414 (1 nodes), 89.071099ms from start of poll, 89.201011ms from cluster start. Timeout at 2025-02-09 06:52:29.658842003 +0000 UTC m=+10.637650771, 105.884357ms after stability testing.go:149: cluster: 2025-02-09T06:52:29.658Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.552957648 +0000 UTC m=+10.531766414 (1 nodes), 89.071099ms from start of poll, 89.201011ms from cluster start. Timeout at 2025-02-09 06:52:29.658842003 +0000 UTC m=+10.637650771, 105.884357ms after stability testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.659Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: starting snapshot up to: index=102 testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.659Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: starting snapshot up to: index=102 testing.go:153: 2025-02-09T06:52:29.659Z [INFO] creating new snapshot: path=/tmp/raft1788383974/snapshots/2-102-1739083949659.tmp testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.743Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: compacting logs: from=1 to=92 testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.743Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.743Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: snapshot complete up to: index=102 testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.743Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: snapshot complete up to: index=102 testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.745Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: restored from snapshot: id=2-102-1739083949659 testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.745Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: restored from snapshot: id=2-102-1739083949659 testLoggerAdapter verbose: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.745Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85476a8a-6632-bf20-7fe8-c1a95944264a Address:85476a8a-6632-bf20-7fe8-c1a95944264a}]" testing.go:149: server-85476a8a-6632-bf20-7fe8-c1a95944264a: 2025-02-09T06:52:29.745Z [INFO] server-85476a8a-6632-bf20-7fe8-c1a95944264a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85476a8a-6632-bf20-7fe8-c1a95944264a Address:85476a8a-6632-bf20-7fe8-c1a95944264a}]" --- PASS: TestRaft_SnapshotRestore (0.28s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.746Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98400f3d-bf44-68b0-12a6-9a833d849c4b Address:98400f3d-bf44-68b0-12a6-9a833d849c4b}]" testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.746Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98400f3d-bf44-68b0-12a6-9a833d849c4b Address:98400f3d-bf44-68b0-12a6-9a833d849c4b}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.746Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:29.746Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.746Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering follower state: follower="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Follower]" leader= testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.746Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering follower state: follower="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Follower]" leader= testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.812Z [WARN] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.812Z [WARN] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.812Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering candidate state: node="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Candidate]" term=2 testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.812Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering candidate state: node="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Candidate]" term=2 testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.813Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: election won: tally=1 testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.813Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: election won: tally=1 testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.813Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering leader state: leader="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Leader]" testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.813Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering leader state: leader="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.919Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.813087994 +0000 UTC m=+10.791896756 (1 nodes), 66.917737ms from start of poll, 67.114562ms from cluster start. Timeout at 2025-02-09 06:52:29.919018331 +0000 UTC m=+10.897827100, 105.930344ms after stability testing.go:149: cluster: 2025-02-09T06:52:29.919Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.813087994 +0000 UTC m=+10.791896756 (1 nodes), 66.917737ms from start of poll, 67.114562ms from cluster start. Timeout at 2025-02-09 06:52:29.919018331 +0000 UTC m=+10.897827100, 105.930344ms after stability testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.919Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: starting snapshot up to: index=102 testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.919Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: starting snapshot up to: index=102 testing.go:153: 2025-02-09T06:52:29.919Z [INFO] creating new snapshot: path=/tmp/raft930944203/snapshots/2-102-1739083949919.tmp testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.923Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: compacting logs: from=1 to=92 testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.923Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.923Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: snapshot complete up to: index=102 testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.923Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: snapshot complete up to: index=102 testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.923Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98400f3d-bf44-68b0-12a6-9a833d849c4b Address:98400f3d-bf44-68b0-12a6-9a833d849c4b}]" testing.go:149: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.923Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98400f3d-bf44-68b0-12a6-9a833d849c4b Address:98400f3d-bf44-68b0-12a6-9a833d849c4b}]" --- PASS: TestRaft_NoRestoreOnStart (0.18s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:29.923Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering follower state: follower="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Follower]" leader= testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.924Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a Address:a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a}]" testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.924Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a Address:a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:29.924Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:29.924Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.924Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: entering follower state: follower="Node at a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a [Follower]" leader= testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.924Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: entering follower state: follower="Node at a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a [Follower]" leader= testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [WARN] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [WARN] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: entering candidate state: node="Node at a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a [Candidate]" term=2 testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: entering candidate state: node="Node at a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a [Candidate]" term=2 testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: election won: tally=1 testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: election won: tally=1 testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: entering leader state: leader="Node at a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a [Leader]" testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:29.981Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: entering leader state: leader="Node at a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a [Leader]" testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:30.002Z [WARN] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:30.002Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering candidate state: node="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Candidate]" term=3 testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:30.002Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: election won: tally=1 testLoggerAdapter verbose: server-98400f3d-bf44-68b0-12a6-9a833d849c4b: 2025-02-09T06:52:30.002Z [INFO] server-98400f3d-bf44-68b0-12a6-9a833d849c4b: entering leader state: leader="Node at 98400f3d-bf44-68b0-12a6-9a833d849c4b [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.087Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.981581866 +0000 UTC m=+10.960390645 (1 nodes), 57.239174ms from start of poll, 57.376655ms from cluster start. Timeout at 2025-02-09 06:52:30.087036947 +0000 UTC m=+11.065845715, 105.45507ms after stability testing.go:149: cluster: 2025-02-09T06:52:30.087Z [INFO] stable state for Leader reached at 2025-02-09 06:52:29.981581866 +0000 UTC m=+10.960390645 (1 nodes), 57.239174ms from start of poll, 57.376655ms from cluster start. Timeout at 2025-02-09 06:52:30.087036947 +0000 UTC m=+11.065845715, 105.45507ms after stability testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.087Z [ERROR] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.087Z [ERROR] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.087Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: starting snapshot up to: index=12 testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.087Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: starting snapshot up to: index=12 testing.go:153: 2025-02-09T06:52:30.087Z [INFO] creating new snapshot: path=/tmp/raft276410526/snapshots/2-12-1739083950087.tmp testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.091Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: compacting logs: from=1 to=2 testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.091Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.091Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: snapshot complete up to: index=12 testing.go:149: server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: 2025-02-09T06:52:30.091Z [INFO] server-a9d432a2-42a5-0fb4-e0ca-5ff591e6cc7a: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.17s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.092Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90870093-0964-9a2e-1b19-e08e62de00f7 Address:90870093-0964-9a2e-1b19-e08e62de00f7} {Suffrage:Voter ID:server-776ce372-c6a9-bb52-cddd-a76589382d7b Address:776ce372-c6a9-bb52-cddd-a76589382d7b} {Suffrage:Voter ID:server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 Address:c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}]" testing.go:149: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.092Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90870093-0964-9a2e-1b19-e08e62de00f7 Address:90870093-0964-9a2e-1b19-e08e62de00f7} {Suffrage:Voter ID:server-776ce372-c6a9-bb52-cddd-a76589382d7b Address:776ce372-c6a9-bb52-cddd-a76589382d7b} {Suffrage:Voter ID:server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 Address:c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}]" testLoggerAdapter verbose: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.092Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: entering follower state: follower="Node at 90870093-0964-9a2e-1b19-e08e62de00f7 [Follower]" leader= testing.go:149: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.092Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: entering follower state: follower="Node at 90870093-0964-9a2e-1b19-e08e62de00f7 [Follower]" leader= testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.092Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90870093-0964-9a2e-1b19-e08e62de00f7 Address:90870093-0964-9a2e-1b19-e08e62de00f7} {Suffrage:Voter ID:server-776ce372-c6a9-bb52-cddd-a76589382d7b Address:776ce372-c6a9-bb52-cddd-a76589382d7b} {Suffrage:Voter ID:server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 Address:c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}]" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.092Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90870093-0964-9a2e-1b19-e08e62de00f7 Address:90870093-0964-9a2e-1b19-e08e62de00f7} {Suffrage:Voter ID:server-776ce372-c6a9-bb52-cddd-a76589382d7b Address:776ce372-c6a9-bb52-cddd-a76589382d7b} {Suffrage:Voter ID:server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 Address:c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}]" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.092Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: entering follower state: follower="Node at 776ce372-c6a9-bb52-cddd-a76589382d7b [Follower]" leader= testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.092Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: entering follower state: follower="Node at 776ce372-c6a9-bb52-cddd-a76589382d7b [Follower]" leader= testLoggerAdapter verbose: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.092Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90870093-0964-9a2e-1b19-e08e62de00f7 Address:90870093-0964-9a2e-1b19-e08e62de00f7} {Suffrage:Voter ID:server-776ce372-c6a9-bb52-cddd-a76589382d7b Address:776ce372-c6a9-bb52-cddd-a76589382d7b} {Suffrage:Voter ID:server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 Address:c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}]" testing.go:149: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.092Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90870093-0964-9a2e-1b19-e08e62de00f7 Address:90870093-0964-9a2e-1b19-e08e62de00f7} {Suffrage:Voter ID:server-776ce372-c6a9-bb52-cddd-a76589382d7b Address:776ce372-c6a9-bb52-cddd-a76589382d7b} {Suffrage:Voter ID:server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 Address:c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.092Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:30.092Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.092Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: entering follower state: follower="Node at c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 [Follower]" leader= testing.go:149: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.092Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: entering follower state: follower="Node at c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 [Follower]" leader= testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.175Z [WARN] server-776ce372-c6a9-bb52-cddd-a76589382d7b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.175Z [WARN] server-776ce372-c6a9-bb52-cddd-a76589382d7b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: entering candidate state: node="Node at 776ce372-c6a9-bb52-cddd-a76589382d7b [Candidate]" term=2 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: entering candidate state: node="Node at 776ce372-c6a9-bb52-cddd-a76589382d7b [Candidate]" term=2 testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: election won: tally=2 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: election won: tally=2 testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: entering leader state: leader="Node at 776ce372-c6a9-bb52-cddd-a76589382d7b [Leader]" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: entering leader state: leader="Node at 776ce372-c6a9-bb52-cddd-a76589382d7b [Leader]" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: added peer, starting replication: peer=server-90870093-0964-9a2e-1b19-e08e62de00f7 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: added peer, starting replication: peer=server-90870093-0964-9a2e-1b19-e08e62de00f7 testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: added peer, starting replication: peer=server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: added peer, starting replication: peer=server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.176Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.281Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.176121262 +0000 UTC m=+11.154930049 (1 nodes), 83.447919ms from start of poll, 83.794836ms from cluster start. Timeout at 2025-02-09 06:52:30.281967019 +0000 UTC m=+11.260775792, 105.845743ms after stability testing.go:149: cluster: 2025-02-09T06:52:30.281Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.176121262 +0000 UTC m=+11.154930049 (1 nodes), 83.447919ms from start of poll, 83.794836ms from cluster start. Timeout at 2025-02-09 06:52:30.281967019 +0000 UTC m=+11.260775792, 105.845743ms after stability testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.282Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: starting snapshot up to: index=12 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.282Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: starting snapshot up to: index=12 testing.go:153: 2025-02-09T06:52:30.282Z [INFO] creating new snapshot: path=/tmp/raft472498701/snapshots/2-12-1739083950282.tmp testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.286Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: snapshot complete up to: index=12 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.286Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: snapshot complete up to: index=12 testing.go:153: 2025-02-09T06:52:30.287Z [INFO] creating new snapshot: path=/tmp/raft472498701/snapshots/2-23-1739083950287.tmp testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: copied to local snapshot: bytes=71 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: restored user snapshot: index=1 testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: restored user snapshot: index=1 testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.290Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.298Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.298Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:30.298Z [INFO] creating new snapshot: path=/tmp/raft1122326152/snapshots/2-23-1739083950298.tmp testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.300Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.300Z [ERROR] server-776ce372-c6a9-bb52-cddd-a76589382d7b: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:30.300Z [INFO] creating new snapshot: path=/tmp/raft1536740690/snapshots/2-23-1739083950300.tmp testLoggerAdapter verbose: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.302Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: copied to local snapshot: bytes=71 testing.go:149: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.302Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.302Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: Installed remote snapshot testing.go:149: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.302Z [INFO] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: Installed remote snapshot testLoggerAdapter verbose: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.302Z [WARN] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: skipping application of old log: index=22 testing.go:149: server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: 2025-02-09T06:52:30.302Z [WARN] server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920: skipping application of old log: index=22 testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.302Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.302Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testLoggerAdapter verbose: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.303Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: copied to local snapshot: bytes=71 testing.go:149: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.303Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.304Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: Installed remote snapshot testing.go:149: server-90870093-0964-9a2e-1b19-e08e62de00f7: 2025-02-09T06:52:30.304Z [INFO] server-90870093-0964-9a2e-1b19-e08e62de00f7: Installed remote snapshot testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.304Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.304Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: pipelining replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.320Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.320Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-c7f5cb9b-efca-1d3f-e539-e4b29ae2e920 c7f5cb9b-efca-1d3f-e539-e4b29ae2e920}" testLoggerAdapter verbose: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.320Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testing.go:149: server-776ce372-c6a9-bb52-cddd-a76589382d7b: 2025-02-09T06:52:30.320Z [INFO] server-776ce372-c6a9-bb52-cddd-a76589382d7b: aborting pipeline replication: peer="{Voter server-90870093-0964-9a2e-1b19-e08e62de00f7 90870093-0964-9a2e-1b19-e08e62de00f7}" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.321Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-520eb645-077f-ef5e-bceb-d7cdf8ac9494 Address:520eb645-077f-ef5e-bceb-d7cdf8ac9494} {Suffrage:Voter ID:server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b Address:d1334e8b-1f25-e18a-3f36-adaa88c6c26b} {Suffrage:Voter ID:server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 Address:087cc8b9-c1b2-c264-9d20-5034fb4069a6}]" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.321Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-520eb645-077f-ef5e-bceb-d7cdf8ac9494 Address:520eb645-077f-ef5e-bceb-d7cdf8ac9494} {Suffrage:Voter ID:server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b Address:d1334e8b-1f25-e18a-3f36-adaa88c6c26b} {Suffrage:Voter ID:server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 Address:087cc8b9-c1b2-c264-9d20-5034fb4069a6}]" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.321Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: entering follower state: follower="Node at 520eb645-077f-ef5e-bceb-d7cdf8ac9494 [Follower]" leader= testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.321Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: entering follower state: follower="Node at 520eb645-077f-ef5e-bceb-d7cdf8ac9494 [Follower]" leader= testLoggerAdapter verbose: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.321Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-520eb645-077f-ef5e-bceb-d7cdf8ac9494 Address:520eb645-077f-ef5e-bceb-d7cdf8ac9494} {Suffrage:Voter ID:server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b Address:d1334e8b-1f25-e18a-3f36-adaa88c6c26b} {Suffrage:Voter ID:server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 Address:087cc8b9-c1b2-c264-9d20-5034fb4069a6}]" testing.go:149: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.321Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-520eb645-077f-ef5e-bceb-d7cdf8ac9494 Address:520eb645-077f-ef5e-bceb-d7cdf8ac9494} {Suffrage:Voter ID:server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b Address:d1334e8b-1f25-e18a-3f36-adaa88c6c26b} {Suffrage:Voter ID:server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 Address:087cc8b9-c1b2-c264-9d20-5034fb4069a6}]" testLoggerAdapter verbose: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.321Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: entering follower state: follower="Node at d1334e8b-1f25-e18a-3f36-adaa88c6c26b [Follower]" leader= testing.go:149: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.321Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: entering follower state: follower="Node at d1334e8b-1f25-e18a-3f36-adaa88c6c26b [Follower]" leader= testLoggerAdapter verbose: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.321Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-520eb645-077f-ef5e-bceb-d7cdf8ac9494 Address:520eb645-077f-ef5e-bceb-d7cdf8ac9494} {Suffrage:Voter ID:server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b Address:d1334e8b-1f25-e18a-3f36-adaa88c6c26b} {Suffrage:Voter ID:server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 Address:087cc8b9-c1b2-c264-9d20-5034fb4069a6}]" testing.go:149: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.321Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-520eb645-077f-ef5e-bceb-d7cdf8ac9494 Address:520eb645-077f-ef5e-bceb-d7cdf8ac9494} {Suffrage:Voter ID:server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b Address:d1334e8b-1f25-e18a-3f36-adaa88c6c26b} {Suffrage:Voter ID:server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 Address:087cc8b9-c1b2-c264-9d20-5034fb4069a6}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.321Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:30.321Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.321Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: entering follower state: follower="Node at 087cc8b9-c1b2-c264-9d20-5034fb4069a6 [Follower]" leader= testing.go:149: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.321Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: entering follower state: follower="Node at 087cc8b9-c1b2-c264-9d20-5034fb4069a6 [Follower]" leader= testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [WARN] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [WARN] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: entering candidate state: node="Node at 520eb645-077f-ef5e-bceb-d7cdf8ac9494 [Candidate]" term=2 testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: entering candidate state: node="Node at 520eb645-077f-ef5e-bceb-d7cdf8ac9494 [Candidate]" term=2 testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: election won: tally=2 testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: election won: tally=2 testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: entering leader state: leader="Node at 520eb645-077f-ef5e-bceb-d7cdf8ac9494 [Leader]" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: entering leader state: leader="Node at 520eb645-077f-ef5e-bceb-d7cdf8ac9494 [Leader]" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: added peer, starting replication: peer=server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: added peer, starting replication: peer=server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: added peer, starting replication: peer=server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: added peer, starting replication: peer=server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.375Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.376Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.376Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.481Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.375917432 +0000 UTC m=+11.354726214 (1 nodes), 54.405307ms from start of poll, 54.768132ms from cluster start. Timeout at 2025-02-09 06:52:30.48174604 +0000 UTC m=+11.460554809, 105.828595ms after stability testing.go:149: cluster: 2025-02-09T06:52:30.481Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.375917432 +0000 UTC m=+11.354726214 (1 nodes), 54.405307ms from start of poll, 54.768132ms from cluster start. Timeout at 2025-02-09 06:52:30.48174604 +0000 UTC m=+11.460554809, 105.828595ms after stability testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.481Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: starting snapshot up to: index=12 testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.481Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: starting snapshot up to: index=12 testing.go:153: 2025-02-09T06:52:30.482Z [INFO] creating new snapshot: path=/tmp/raft3653764569/snapshots/2-12-1739083950482.tmp testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.485Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: snapshot complete up to: index=12 testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.485Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: snapshot complete up to: index=12 testing.go:153: 2025-02-09T06:52:30.485Z [INFO] creating new snapshot: path=/tmp/raft3653764569/snapshots/2-23-1739083950485.tmp testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.489Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: copied to local snapshot: bytes=71 testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.489Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.489Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: restored user snapshot: index=1 testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.489Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: restored user snapshot: index=1 testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.490Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.500Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.500Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.500Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.500Z [ERROR] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:30.500Z [INFO] creating new snapshot: path=/tmp/raft3837035945/snapshots/2-23-1739083950500.tmp testing.go:153: 2025-02-09T06:52:30.500Z [INFO] creating new snapshot: path=/tmp/raft1414319403/snapshots/2-23-1739083950500.tmp testLoggerAdapter verbose: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.504Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: copied to local snapshot: bytes=71 testing.go:149: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.504Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.504Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: Installed remote snapshot testing.go:149: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.504Z [INFO] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: Installed remote snapshot testLoggerAdapter verbose: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.504Z [WARN] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: skipping application of old log: index=22 testing.go:149: server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: 2025-02-09T06:52:30.504Z [WARN] server-087cc8b9-c1b2-c264-9d20-5034fb4069a6: skipping application of old log: index=22 testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.504Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.504Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testLoggerAdapter verbose: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.504Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: copied to local snapshot: bytes=71 testing.go:149: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.504Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.504Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: Installed remote snapshot testing.go:149: server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: 2025-02-09T06:52:30.504Z [INFO] server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b: Installed remote snapshot testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.504Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.504Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: pipelining replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.520Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.520Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-087cc8b9-c1b2-c264-9d20-5034fb4069a6 087cc8b9-c1b2-c264-9d20-5034fb4069a6}" testLoggerAdapter verbose: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.520Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testing.go:149: server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: 2025-02-09T06:52:30.520Z [INFO] server-520eb645-077f-ef5e-bceb-d7cdf8ac9494: aborting pipeline replication: peer="{Voter server-d1334e8b-1f25-e18a-3f36-adaa88c6c26b d1334e8b-1f25-e18a-3f36-adaa88c6c26b}" testLoggerAdapter verbose: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.521Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d77a21b-f17d-6ef4-3c55-355704340786 Address:9d77a21b-f17d-6ef4-3c55-355704340786} {Suffrage:Voter ID:server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 Address:42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0} {Suffrage:Voter ID:server-97d5b64c-70f6-ee6b-cef7-b89b31861f93 Address:97d5b64c-70f6-ee6b-cef7-b89b31861f93}]" testing.go:149: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.521Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d77a21b-f17d-6ef4-3c55-355704340786 Address:9d77a21b-f17d-6ef4-3c55-355704340786} {Suffrage:Voter ID:server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 Address:42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0} {Suffrage:Voter ID:server-97d5b64c-70f6-ee6b-cef7-b89b31861f93 Address:97d5b64c-70f6-ee6b-cef7-b89b31861f93}]" testLoggerAdapter verbose: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.522Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: entering follower state: follower="Node at 9d77a21b-f17d-6ef4-3c55-355704340786 [Follower]" leader= testing.go:149: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.522Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: entering follower state: follower="Node at 9d77a21b-f17d-6ef4-3c55-355704340786 [Follower]" leader= testLoggerAdapter verbose: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.522Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d77a21b-f17d-6ef4-3c55-355704340786 Address:9d77a21b-f17d-6ef4-3c55-355704340786} {Suffrage:Voter ID:server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 Address:42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0} {Suffrage:Voter ID:server-97d5b64c-70f6-ee6b-cef7-b89b31861f93 Address:97d5b64c-70f6-ee6b-cef7-b89b31861f93}]" testing.go:149: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.522Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d77a21b-f17d-6ef4-3c55-355704340786 Address:9d77a21b-f17d-6ef4-3c55-355704340786} {Suffrage:Voter ID:server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 Address:42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0} {Suffrage:Voter ID:server-97d5b64c-70f6-ee6b-cef7-b89b31861f93 Address:97d5b64c-70f6-ee6b-cef7-b89b31861f93}]" testLoggerAdapter verbose: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.522Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: entering follower state: follower="Node at 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 [Follower]" leader= testing.go:149: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.522Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: entering follower state: follower="Node at 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 [Follower]" leader= testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.522Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d77a21b-f17d-6ef4-3c55-355704340786 Address:9d77a21b-f17d-6ef4-3c55-355704340786} {Suffrage:Voter ID:server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 Address:42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0} {Suffrage:Voter ID:server-97d5b64c-70f6-ee6b-cef7-b89b31861f93 Address:97d5b64c-70f6-ee6b-cef7-b89b31861f93}]" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.522Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d77a21b-f17d-6ef4-3c55-355704340786 Address:9d77a21b-f17d-6ef4-3c55-355704340786} {Suffrage:Voter ID:server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 Address:42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0} {Suffrage:Voter ID:server-97d5b64c-70f6-ee6b-cef7-b89b31861f93 Address:97d5b64c-70f6-ee6b-cef7-b89b31861f93}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.522Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:30.522Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.522Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: entering follower state: follower="Node at 97d5b64c-70f6-ee6b-cef7-b89b31861f93 [Follower]" leader= testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.522Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: entering follower state: follower="Node at 97d5b64c-70f6-ee6b-cef7-b89b31861f93 [Follower]" leader= testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.577Z [WARN] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.577Z [WARN] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.577Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: entering candidate state: node="Node at 97d5b64c-70f6-ee6b-cef7-b89b31861f93 [Candidate]" term=2 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.577Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: entering candidate state: node="Node at 97d5b64c-70f6-ee6b-cef7-b89b31861f93 [Candidate]" term=2 testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: election won: tally=2 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: election won: tally=2 testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: entering leader state: leader="Node at 97d5b64c-70f6-ee6b-cef7-b89b31861f93 [Leader]" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: entering leader state: leader="Node at 97d5b64c-70f6-ee6b-cef7-b89b31861f93 [Leader]" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: added peer, starting replication: peer=server-9d77a21b-f17d-6ef4-3c55-355704340786 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: added peer, starting replication: peer=server-9d77a21b-f17d-6ef4-3c55-355704340786 testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: added peer, starting replication: peer=server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: added peer, starting replication: peer=server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.578Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.683Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.578113266 +0000 UTC m=+11.556922031 (1 nodes), 55.694498ms from start of poll, 56.203714ms from cluster start. Timeout at 2025-02-09 06:52:30.683384756 +0000 UTC m=+11.662193524, 105.271493ms after stability testing.go:149: cluster: 2025-02-09T06:52:30.683Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.578113266 +0000 UTC m=+11.556922031 (1 nodes), 55.694498ms from start of poll, 56.203714ms from cluster start. Timeout at 2025-02-09 06:52:30.683384756 +0000 UTC m=+11.662193524, 105.271493ms after stability testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.683Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: starting snapshot up to: index=12 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.683Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: starting snapshot up to: index=12 testing.go:153: 2025-02-09T06:52:30.683Z [INFO] creating new snapshot: path=/tmp/raft1583009783/snapshots/2-12-1739083950683.tmp testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.688Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: snapshot complete up to: index=12 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.688Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: snapshot complete up to: index=12 testing.go:153: 2025-02-09T06:52:30.688Z [INFO] creating new snapshot: path=/tmp/raft1583009783/snapshots/2-23-1739083950688.tmp testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: copied to local snapshot: bytes=71 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: restored user snapshot: index=1 testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: restored user snapshot: index=1 testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.691Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.698Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.698Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:30.698Z [INFO] creating new snapshot: path=/tmp/raft2258289372/snapshots/2-23-1739083950698.tmp testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.701Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.701Z [ERROR] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:30.702Z [INFO] creating new snapshot: path=/tmp/raft2603878710/snapshots/2-23-1739083950702.tmp testLoggerAdapter verbose: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.702Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: copied to local snapshot: bytes=71 testing.go:149: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.702Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.702Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: Installed remote snapshot testing.go:149: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.702Z [INFO] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: Installed remote snapshot testLoggerAdapter verbose: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.702Z [WARN] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: skipping application of old log: index=22 testing.go:149: server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: 2025-02-09T06:52:30.702Z [WARN] server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0: skipping application of old log: index=22 testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.702Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.702Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testLoggerAdapter verbose: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.705Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: copied to local snapshot: bytes=71 testing.go:149: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.705Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.705Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: Installed remote snapshot testing.go:149: server-9d77a21b-f17d-6ef4-3c55-355704340786: 2025-02-09T06:52:30.705Z [INFO] server-9d77a21b-f17d-6ef4-3c55-355704340786: Installed remote snapshot testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.705Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.705Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: pipelining replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.719Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.719Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0 42fd7b7b-283c-3ea8-cadf-d9cba3b80ec0}" testLoggerAdapter verbose: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.719Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testing.go:149: server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: 2025-02-09T06:52:30.719Z [INFO] server-97d5b64c-70f6-ee6b-cef7-b89b31861f93: aborting pipeline replication: peer="{Voter server-9d77a21b-f17d-6ef4-3c55-355704340786 9d77a21b-f17d-6ef4-3c55-355704340786}" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.720Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f893d015-09a3-921d-e2f9-eeb74c1e09a1 Address:f893d015-09a3-921d-e2f9-eeb74c1e09a1} {Suffrage:Voter ID:server-7ba6a560-8794-659b-71b8-e26c4a258a13 Address:7ba6a560-8794-659b-71b8-e26c4a258a13} {Suffrage:Voter ID:server-fc863720-0f1f-7639-7491-7b3c2d787261 Address:fc863720-0f1f-7639-7491-7b3c2d787261}]" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.720Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f893d015-09a3-921d-e2f9-eeb74c1e09a1 Address:f893d015-09a3-921d-e2f9-eeb74c1e09a1} {Suffrage:Voter ID:server-7ba6a560-8794-659b-71b8-e26c4a258a13 Address:7ba6a560-8794-659b-71b8-e26c4a258a13} {Suffrage:Voter ID:server-fc863720-0f1f-7639-7491-7b3c2d787261 Address:fc863720-0f1f-7639-7491-7b3c2d787261}]" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.720Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: entering follower state: follower="Node at f893d015-09a3-921d-e2f9-eeb74c1e09a1 [Follower]" leader= testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.720Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: entering follower state: follower="Node at f893d015-09a3-921d-e2f9-eeb74c1e09a1 [Follower]" leader= testLoggerAdapter verbose: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.720Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f893d015-09a3-921d-e2f9-eeb74c1e09a1 Address:f893d015-09a3-921d-e2f9-eeb74c1e09a1} {Suffrage:Voter ID:server-7ba6a560-8794-659b-71b8-e26c4a258a13 Address:7ba6a560-8794-659b-71b8-e26c4a258a13} {Suffrage:Voter ID:server-fc863720-0f1f-7639-7491-7b3c2d787261 Address:fc863720-0f1f-7639-7491-7b3c2d787261}]" testing.go:149: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.720Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f893d015-09a3-921d-e2f9-eeb74c1e09a1 Address:f893d015-09a3-921d-e2f9-eeb74c1e09a1} {Suffrage:Voter ID:server-7ba6a560-8794-659b-71b8-e26c4a258a13 Address:7ba6a560-8794-659b-71b8-e26c4a258a13} {Suffrage:Voter ID:server-fc863720-0f1f-7639-7491-7b3c2d787261 Address:fc863720-0f1f-7639-7491-7b3c2d787261}]" testLoggerAdapter verbose: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.720Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: entering follower state: follower="Node at 7ba6a560-8794-659b-71b8-e26c4a258a13 [Follower]" leader= testing.go:149: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.720Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: entering follower state: follower="Node at 7ba6a560-8794-659b-71b8-e26c4a258a13 [Follower]" leader= testLoggerAdapter verbose: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.720Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f893d015-09a3-921d-e2f9-eeb74c1e09a1 Address:f893d015-09a3-921d-e2f9-eeb74c1e09a1} {Suffrage:Voter ID:server-7ba6a560-8794-659b-71b8-e26c4a258a13 Address:7ba6a560-8794-659b-71b8-e26c4a258a13} {Suffrage:Voter ID:server-fc863720-0f1f-7639-7491-7b3c2d787261 Address:fc863720-0f1f-7639-7491-7b3c2d787261}]" testing.go:149: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.720Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f893d015-09a3-921d-e2f9-eeb74c1e09a1 Address:f893d015-09a3-921d-e2f9-eeb74c1e09a1} {Suffrage:Voter ID:server-7ba6a560-8794-659b-71b8-e26c4a258a13 Address:7ba6a560-8794-659b-71b8-e26c4a258a13} {Suffrage:Voter ID:server-fc863720-0f1f-7639-7491-7b3c2d787261 Address:fc863720-0f1f-7639-7491-7b3c2d787261}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.720Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:30.720Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.720Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: entering follower state: follower="Node at fc863720-0f1f-7639-7491-7b3c2d787261 [Follower]" leader= testing.go:149: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.720Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: entering follower state: follower="Node at fc863720-0f1f-7639-7491-7b3c2d787261 [Follower]" leader= testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [WARN] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [WARN] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: entering candidate state: node="Node at f893d015-09a3-921d-e2f9-eeb74c1e09a1 [Candidate]" term=2 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: entering candidate state: node="Node at f893d015-09a3-921d-e2f9-eeb74c1e09a1 [Candidate]" term=2 testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: election won: tally=2 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: election won: tally=2 testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: entering leader state: leader="Node at f893d015-09a3-921d-e2f9-eeb74c1e09a1 [Leader]" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: entering leader state: leader="Node at f893d015-09a3-921d-e2f9-eeb74c1e09a1 [Leader]" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: added peer, starting replication: peer=server-7ba6a560-8794-659b-71b8-e26c4a258a13 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: added peer, starting replication: peer=server-7ba6a560-8794-659b-71b8-e26c4a258a13 testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: added peer, starting replication: peer=server-fc863720-0f1f-7639-7491-7b3c2d787261 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: added peer, starting replication: peer=server-fc863720-0f1f-7639-7491-7b3c2d787261 testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.788Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.894Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.788344401 +0000 UTC m=+11.767153179 (1 nodes), 67.737571ms from start of poll, 68.081779ms from cluster start. Timeout at 2025-02-09 06:52:30.894032045 +0000 UTC m=+11.872840804, 105.687625ms after stability testing.go:149: cluster: 2025-02-09T06:52:30.894Z [INFO] stable state for Leader reached at 2025-02-09 06:52:30.788344401 +0000 UTC m=+11.767153179 (1 nodes), 67.737571ms from start of poll, 68.081779ms from cluster start. Timeout at 2025-02-09 06:52:30.894032045 +0000 UTC m=+11.872840804, 105.687625ms after stability testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.894Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: starting snapshot up to: index=12 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.894Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: starting snapshot up to: index=12 testing.go:153: 2025-02-09T06:52:30.894Z [INFO] creating new snapshot: path=/tmp/raft2966269005/snapshots/2-12-1739083950894.tmp testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.898Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: snapshot complete up to: index=12 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.898Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: snapshot complete up to: index=12 testing.go:153: 2025-02-09T06:52:30.898Z [INFO] creating new snapshot: path=/tmp/raft2966269005/snapshots/2-113-1739083950898.tmp testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: copied to local snapshot: bytes=71 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: restored user snapshot: index=1 testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: restored user snapshot: index=1 testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.902Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.903Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.903Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.909Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.909Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:30.910Z [INFO] creating new snapshot: path=/tmp/raft517204571/snapshots/2-113-1739083950910.tmp testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.912Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.912Z [ERROR] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:30.913Z [INFO] creating new snapshot: path=/tmp/raft447112047/snapshots/2-113-1739083950913.tmp testLoggerAdapter verbose: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.914Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: copied to local snapshot: bytes=71 testing.go:149: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.914Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.914Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: Installed remote snapshot testing.go:149: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.914Z [INFO] server-fc863720-0f1f-7639-7491-7b3c2d787261: Installed remote snapshot testLoggerAdapter verbose: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.914Z [WARN] server-fc863720-0f1f-7639-7491-7b3c2d787261: skipping application of old log: index=22 testing.go:149: server-fc863720-0f1f-7639-7491-7b3c2d787261: 2025-02-09T06:52:30.914Z [WARN] server-fc863720-0f1f-7639-7491-7b3c2d787261: skipping application of old log: index=22 testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.914Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.914Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testLoggerAdapter verbose: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.916Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: copied to local snapshot: bytes=71 testing.go:149: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.916Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.916Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: Installed remote snapshot testing.go:149: server-7ba6a560-8794-659b-71b8-e26c4a258a13: 2025-02-09T06:52:30.916Z [INFO] server-7ba6a560-8794-659b-71b8-e26c4a258a13: Installed remote snapshot testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.916Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.916Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: pipelining replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.936Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.936Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-fc863720-0f1f-7639-7491-7b3c2d787261 fc863720-0f1f-7639-7491-7b3c2d787261}" testLoggerAdapter verbose: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.936Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testing.go:149: server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: 2025-02-09T06:52:30.936Z [INFO] server-f893d015-09a3-921d-e2f9-eeb74c1e09a1: aborting pipeline replication: peer="{Voter server-7ba6a560-8794-659b-71b8-e26c4a258a13 7ba6a560-8794-659b-71b8-e26c4a258a13}" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.937Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b Address:3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b} {Suffrage:Voter ID:server-993631aa-85a8-0845-55f3-9c16d10b6eca Address:993631aa-85a8-0845-55f3-9c16d10b6eca} {Suffrage:Voter ID:server-abd23e07-12ed-2b70-1e64-334ea4b1055d Address:abd23e07-12ed-2b70-1e64-334ea4b1055d}]" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.937Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b Address:3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b} {Suffrage:Voter ID:server-993631aa-85a8-0845-55f3-9c16d10b6eca Address:993631aa-85a8-0845-55f3-9c16d10b6eca} {Suffrage:Voter ID:server-abd23e07-12ed-2b70-1e64-334ea4b1055d Address:abd23e07-12ed-2b70-1e64-334ea4b1055d}]" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.937Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: entering follower state: follower="Node at 3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b [Follower]" leader= testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.937Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: entering follower state: follower="Node at 3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b [Follower]" leader= testLoggerAdapter verbose: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:30.937Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b Address:3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b} {Suffrage:Voter ID:server-993631aa-85a8-0845-55f3-9c16d10b6eca Address:993631aa-85a8-0845-55f3-9c16d10b6eca} {Suffrage:Voter ID:server-abd23e07-12ed-2b70-1e64-334ea4b1055d Address:abd23e07-12ed-2b70-1e64-334ea4b1055d}]" testing.go:149: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:30.937Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b Address:3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b} {Suffrage:Voter ID:server-993631aa-85a8-0845-55f3-9c16d10b6eca Address:993631aa-85a8-0845-55f3-9c16d10b6eca} {Suffrage:Voter ID:server-abd23e07-12ed-2b70-1e64-334ea4b1055d Address:abd23e07-12ed-2b70-1e64-334ea4b1055d}]" testLoggerAdapter verbose: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:30.938Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: entering follower state: follower="Node at 993631aa-85a8-0845-55f3-9c16d10b6eca [Follower]" leader= testing.go:149: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:30.938Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: entering follower state: follower="Node at 993631aa-85a8-0845-55f3-9c16d10b6eca [Follower]" leader= testLoggerAdapter verbose: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:30.938Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b Address:3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b} {Suffrage:Voter ID:server-993631aa-85a8-0845-55f3-9c16d10b6eca Address:993631aa-85a8-0845-55f3-9c16d10b6eca} {Suffrage:Voter ID:server-abd23e07-12ed-2b70-1e64-334ea4b1055d Address:abd23e07-12ed-2b70-1e64-334ea4b1055d}]" testing.go:149: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:30.938Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b Address:3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b} {Suffrage:Voter ID:server-993631aa-85a8-0845-55f3-9c16d10b6eca Address:993631aa-85a8-0845-55f3-9c16d10b6eca} {Suffrage:Voter ID:server-abd23e07-12ed-2b70-1e64-334ea4b1055d Address:abd23e07-12ed-2b70-1e64-334ea4b1055d}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:30.938Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:30.938Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:30.938Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: entering follower state: follower="Node at abd23e07-12ed-2b70-1e64-334ea4b1055d [Follower]" leader= testing.go:149: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:30.938Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: entering follower state: follower="Node at abd23e07-12ed-2b70-1e64-334ea4b1055d [Follower]" leader= testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.999Z [WARN] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.999Z [WARN] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.999Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: entering candidate state: node="Node at 3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b [Candidate]" term=2 testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:30.999Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: entering candidate state: node="Node at 3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b [Candidate]" term=2 testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: election won: tally=2 testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: election won: tally=2 testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: entering leader state: leader="Node at 3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b [Leader]" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: entering leader state: leader="Node at 3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b [Leader]" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: added peer, starting replication: peer=server-993631aa-85a8-0845-55f3-9c16d10b6eca testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: added peer, starting replication: peer=server-993631aa-85a8-0845-55f3-9c16d10b6eca testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: added peer, starting replication: peer=server-abd23e07-12ed-2b70-1e64-334ea4b1055d testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: added peer, starting replication: peer=server-abd23e07-12ed-2b70-1e64-334ea4b1055d testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.000Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.105Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.000116399 +0000 UTC m=+11.978925213 (1 nodes), 61.993958ms from start of poll, 62.355225ms from cluster start. Timeout at 2025-02-09 06:52:31.105498884 +0000 UTC m=+12.084307643, 105.38243ms after stability testing.go:149: cluster: 2025-02-09T06:52:31.105Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.000116399 +0000 UTC m=+11.978925213 (1 nodes), 61.993958ms from start of poll, 62.355225ms from cluster start. Timeout at 2025-02-09 06:52:31.105498884 +0000 UTC m=+12.084307643, 105.38243ms after stability testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.105Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: starting snapshot up to: index=12 testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.105Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: starting snapshot up to: index=12 testing.go:153: 2025-02-09T06:52:31.105Z [INFO] creating new snapshot: path=/tmp/raft3164157557/snapshots/2-12-1739083951105.tmp testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.111Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: snapshot complete up to: index=12 testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.111Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: snapshot complete up to: index=12 testing.go:153: 2025-02-09T06:52:31.111Z [INFO] creating new snapshot: path=/tmp/raft3164157557/snapshots/2-1013-1739083951111.tmp testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: copied to local snapshot: bytes=71 testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: restored user snapshot: index=1 testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: restored user snapshot: index=1 testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.114Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.121Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.121Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:31.122Z [INFO] creating new snapshot: path=/tmp/raft3180562438/snapshots/2-1013-1739083951122.tmp testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.122Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.122Z [ERROR] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:31.123Z [INFO] creating new snapshot: path=/tmp/raft124009344/snapshots/2-1013-1739083951123.tmp testLoggerAdapter verbose: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:31.125Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: copied to local snapshot: bytes=71 testing.go:149: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:31.125Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:31.125Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: Installed remote snapshot testing.go:149: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:31.125Z [INFO] server-993631aa-85a8-0845-55f3-9c16d10b6eca: Installed remote snapshot testLoggerAdapter verbose: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:31.125Z [WARN] server-993631aa-85a8-0845-55f3-9c16d10b6eca: skipping application of old log: index=22 testing.go:149: server-993631aa-85a8-0845-55f3-9c16d10b6eca: 2025-02-09T06:52:31.125Z [WARN] server-993631aa-85a8-0845-55f3-9c16d10b6eca: skipping application of old log: index=22 testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.125Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.125Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testLoggerAdapter verbose: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:31.126Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: copied to local snapshot: bytes=71 testing.go:149: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:31.126Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:31.126Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: Installed remote snapshot testing.go:149: server-abd23e07-12ed-2b70-1e64-334ea4b1055d: 2025-02-09T06:52:31.126Z [INFO] server-abd23e07-12ed-2b70-1e64-334ea4b1055d: Installed remote snapshot testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.127Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.127Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: pipelining replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.142Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.142Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-abd23e07-12ed-2b70-1e64-334ea4b1055d abd23e07-12ed-2b70-1e64-334ea4b1055d}" testLoggerAdapter verbose: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.142Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testing.go:149: server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: 2025-02-09T06:52:31.142Z [INFO] server-3224f6d6-ff0a-ab2c-f705-fd0479ee6a7b: aborting pipeline replication: peer="{Voter server-993631aa-85a8-0845-55f3-9c16d10b6eca 993631aa-85a8-0845-55f3-9c16d10b6eca}" testLoggerAdapter verbose: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.143Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a429f1-aec4-9684-97eb-cebe43f05f2c Address:30a429f1-aec4-9684-97eb-cebe43f05f2c} {Suffrage:Voter ID:server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e Address:f1c4879e-7dc6-3a38-864b-453bc6cabe5e} {Suffrage:Voter ID:server-a7056529-d904-7f8e-13b7-db02fda90a80 Address:a7056529-d904-7f8e-13b7-db02fda90a80}]" testing.go:149: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.143Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a429f1-aec4-9684-97eb-cebe43f05f2c Address:30a429f1-aec4-9684-97eb-cebe43f05f2c} {Suffrage:Voter ID:server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e Address:f1c4879e-7dc6-3a38-864b-453bc6cabe5e} {Suffrage:Voter ID:server-a7056529-d904-7f8e-13b7-db02fda90a80 Address:a7056529-d904-7f8e-13b7-db02fda90a80}]" testLoggerAdapter verbose: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.143Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: entering follower state: follower="Node at 30a429f1-aec4-9684-97eb-cebe43f05f2c [Follower]" leader= testing.go:149: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.143Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: entering follower state: follower="Node at 30a429f1-aec4-9684-97eb-cebe43f05f2c [Follower]" leader= testLoggerAdapter verbose: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.143Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a429f1-aec4-9684-97eb-cebe43f05f2c Address:30a429f1-aec4-9684-97eb-cebe43f05f2c} {Suffrage:Voter ID:server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e Address:f1c4879e-7dc6-3a38-864b-453bc6cabe5e} {Suffrage:Voter ID:server-a7056529-d904-7f8e-13b7-db02fda90a80 Address:a7056529-d904-7f8e-13b7-db02fda90a80}]" testing.go:149: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.143Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a429f1-aec4-9684-97eb-cebe43f05f2c Address:30a429f1-aec4-9684-97eb-cebe43f05f2c} {Suffrage:Voter ID:server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e Address:f1c4879e-7dc6-3a38-864b-453bc6cabe5e} {Suffrage:Voter ID:server-a7056529-d904-7f8e-13b7-db02fda90a80 Address:a7056529-d904-7f8e-13b7-db02fda90a80}]" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.143Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a429f1-aec4-9684-97eb-cebe43f05f2c Address:30a429f1-aec4-9684-97eb-cebe43f05f2c} {Suffrage:Voter ID:server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e Address:f1c4879e-7dc6-3a38-864b-453bc6cabe5e} {Suffrage:Voter ID:server-a7056529-d904-7f8e-13b7-db02fda90a80 Address:a7056529-d904-7f8e-13b7-db02fda90a80}]" testLoggerAdapter verbose: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.143Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: entering follower state: follower="Node at f1c4879e-7dc6-3a38-864b-453bc6cabe5e [Follower]" leader= testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.143Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a429f1-aec4-9684-97eb-cebe43f05f2c Address:30a429f1-aec4-9684-97eb-cebe43f05f2c} {Suffrage:Voter ID:server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e Address:f1c4879e-7dc6-3a38-864b-453bc6cabe5e} {Suffrage:Voter ID:server-a7056529-d904-7f8e-13b7-db02fda90a80 Address:a7056529-d904-7f8e-13b7-db02fda90a80}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.143Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.143Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: entering follower state: follower="Node at f1c4879e-7dc6-3a38-864b-453bc6cabe5e [Follower]" leader= testing.go:149: cluster: 2025-02-09T06:52:31.143Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.143Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: entering follower state: follower="Node at a7056529-d904-7f8e-13b7-db02fda90a80 [Follower]" leader= testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.143Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: entering follower state: follower="Node at a7056529-d904-7f8e-13b7-db02fda90a80 [Follower]" leader= testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [WARN] server-a7056529-d904-7f8e-13b7-db02fda90a80: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [WARN] server-a7056529-d904-7f8e-13b7-db02fda90a80: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: entering candidate state: node="Node at a7056529-d904-7f8e-13b7-db02fda90a80 [Candidate]" term=2 testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: entering candidate state: node="Node at a7056529-d904-7f8e-13b7-db02fda90a80 [Candidate]" term=2 testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: election won: tally=2 testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: election won: tally=2 testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: entering leader state: leader="Node at a7056529-d904-7f8e-13b7-db02fda90a80 [Leader]" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: entering leader state: leader="Node at a7056529-d904-7f8e-13b7-db02fda90a80 [Leader]" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: added peer, starting replication: peer=server-30a429f1-aec4-9684-97eb-cebe43f05f2c testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: added peer, starting replication: peer=server-30a429f1-aec4-9684-97eb-cebe43f05f2c testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: added peer, starting replication: peer=server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: added peer, starting replication: peer=server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.198Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.304Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.198513698 +0000 UTC m=+12.177322462 (1 nodes), 54.621058ms from start of poll, 54.956851ms from cluster start. Timeout at 2025-02-09 06:52:31.304065433 +0000 UTC m=+12.282874228, 105.551766ms after stability testing.go:149: cluster: 2025-02-09T06:52:31.304Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.198513698 +0000 UTC m=+12.177322462 (1 nodes), 54.621058ms from start of poll, 54.956851ms from cluster start. Timeout at 2025-02-09 06:52:31.304065433 +0000 UTC m=+12.282874228, 105.551766ms after stability testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.304Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: starting snapshot up to: index=12 testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.304Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: starting snapshot up to: index=12 testing.go:153: 2025-02-09T06:52:31.304Z [INFO] creating new snapshot: path=/tmp/raft3427325543/snapshots/2-12-1739083951304.tmp testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.307Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: snapshot complete up to: index=12 testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.307Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: snapshot complete up to: index=12 testing.go:153: 2025-02-09T06:52:31.308Z [INFO] creating new snapshot: path=/tmp/raft3427325543/snapshots/2-10013-1739083951308.tmp testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: copied to local snapshot: bytes=71 testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: restored user snapshot: index=1 testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: restored user snapshot: index=1 testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.311Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.319Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.319Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.319Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.319Z [ERROR] server-a7056529-d904-7f8e-13b7-db02fda90a80: failed to get log: index=23 error="log not found" testing.go:153: 2025-02-09T06:52:31.319Z [INFO] creating new snapshot: path=/tmp/raft4195242853/snapshots/2-10013-1739083951319.tmp testing.go:153: 2025-02-09T06:52:31.319Z [INFO] creating new snapshot: path=/tmp/raft2597715895/snapshots/2-10013-1739083951319.tmp testLoggerAdapter verbose: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.323Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: copied to local snapshot: bytes=71 testing.go:149: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.323Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.323Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: Installed remote snapshot testing.go:149: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.323Z [INFO] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: Installed remote snapshot testLoggerAdapter verbose: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.323Z [WARN] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: skipping application of old log: index=22 testing.go:149: server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: 2025-02-09T06:52:31.323Z [WARN] server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e: skipping application of old log: index=22 testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.323Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.323Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testLoggerAdapter verbose: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.323Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: copied to local snapshot: bytes=71 testing.go:149: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.323Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.323Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: Installed remote snapshot testing.go:149: server-30a429f1-aec4-9684-97eb-cebe43f05f2c: 2025-02-09T06:52:31.323Z [INFO] server-30a429f1-aec4-9684-97eb-cebe43f05f2c: Installed remote snapshot testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.323Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.323Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: pipelining replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.340Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.340Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-f1c4879e-7dc6-3a38-864b-453bc6cabe5e f1c4879e-7dc6-3a38-864b-453bc6cabe5e}" testLoggerAdapter verbose: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.340Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" testing.go:149: server-a7056529-d904-7f8e-13b7-db02fda90a80: 2025-02-09T06:52:31.340Z [INFO] server-a7056529-d904-7f8e-13b7-db02fda90a80: aborting pipeline replication: peer="{Voter server-30a429f1-aec4-9684-97eb-cebe43f05f2c 30a429f1-aec4-9684-97eb-cebe43f05f2c}" --- PASS: TestRaft_UserRestore (1.25s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.341Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27e3834-1e25-de4f-7758-0769fcf30041 Address:f27e3834-1e25-de4f-7758-0769fcf30041} {Suffrage:Voter ID:server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f Address:9a864b8b-b5cb-2054-a3b5-4d03564f8f1f} {Suffrage:Voter ID:server-2948bb85-9435-564c-6883-21ba28fb987b Address:2948bb85-9435-564c-6883-21ba28fb987b}]" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.341Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27e3834-1e25-de4f-7758-0769fcf30041 Address:f27e3834-1e25-de4f-7758-0769fcf30041} {Suffrage:Voter ID:server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f Address:9a864b8b-b5cb-2054-a3b5-4d03564f8f1f} {Suffrage:Voter ID:server-2948bb85-9435-564c-6883-21ba28fb987b Address:2948bb85-9435-564c-6883-21ba28fb987b}]" testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.342Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: entering follower state: follower="Node at f27e3834-1e25-de4f-7758-0769fcf30041 [Follower]" leader= testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.342Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: entering follower state: follower="Node at f27e3834-1e25-de4f-7758-0769fcf30041 [Follower]" leader= testLoggerAdapter verbose: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.342Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27e3834-1e25-de4f-7758-0769fcf30041 Address:f27e3834-1e25-de4f-7758-0769fcf30041} {Suffrage:Voter ID:server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f Address:9a864b8b-b5cb-2054-a3b5-4d03564f8f1f} {Suffrage:Voter ID:server-2948bb85-9435-564c-6883-21ba28fb987b Address:2948bb85-9435-564c-6883-21ba28fb987b}]" testing.go:149: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.342Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27e3834-1e25-de4f-7758-0769fcf30041 Address:f27e3834-1e25-de4f-7758-0769fcf30041} {Suffrage:Voter ID:server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f Address:9a864b8b-b5cb-2054-a3b5-4d03564f8f1f} {Suffrage:Voter ID:server-2948bb85-9435-564c-6883-21ba28fb987b Address:2948bb85-9435-564c-6883-21ba28fb987b}]" testLoggerAdapter verbose: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.342Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: entering follower state: follower="Node at 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f [Follower]" leader= testing.go:149: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.342Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: entering follower state: follower="Node at 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f [Follower]" leader= testLoggerAdapter verbose: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.342Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27e3834-1e25-de4f-7758-0769fcf30041 Address:f27e3834-1e25-de4f-7758-0769fcf30041} {Suffrage:Voter ID:server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f Address:9a864b8b-b5cb-2054-a3b5-4d03564f8f1f} {Suffrage:Voter ID:server-2948bb85-9435-564c-6883-21ba28fb987b Address:2948bb85-9435-564c-6883-21ba28fb987b}]" testing.go:149: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.342Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27e3834-1e25-de4f-7758-0769fcf30041 Address:f27e3834-1e25-de4f-7758-0769fcf30041} {Suffrage:Voter ID:server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f Address:9a864b8b-b5cb-2054-a3b5-4d03564f8f1f} {Suffrage:Voter ID:server-2948bb85-9435-564c-6883-21ba28fb987b Address:2948bb85-9435-564c-6883-21ba28fb987b}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.342Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:31.342Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.342Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: entering follower state: follower="Node at 2948bb85-9435-564c-6883-21ba28fb987b [Follower]" leader= testing.go:149: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.342Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: entering follower state: follower="Node at 2948bb85-9435-564c-6883-21ba28fb987b [Follower]" leader= testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [WARN] server-f27e3834-1e25-de4f-7758-0769fcf30041: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [WARN] server-f27e3834-1e25-de4f-7758-0769fcf30041: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: entering candidate state: node="Node at f27e3834-1e25-de4f-7758-0769fcf30041 [Candidate]" term=2 testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: entering candidate state: node="Node at f27e3834-1e25-de4f-7758-0769fcf30041 [Candidate]" term=2 testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: election won: tally=2 testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: election won: tally=2 testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: entering leader state: leader="Node at f27e3834-1e25-de4f-7758-0769fcf30041 [Leader]" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: entering leader state: leader="Node at f27e3834-1e25-de4f-7758-0769fcf30041 [Leader]" testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: added peer, starting replication: peer=server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: added peer, starting replication: peer=server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: added peer, starting replication: peer=server-2948bb85-9435-564c-6883-21ba28fb987b testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: added peer, starting replication: peer=server-2948bb85-9435-564c-6883-21ba28fb987b testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: pipelining replication: peer="{Voter server-2948bb85-9435-564c-6883-21ba28fb987b 2948bb85-9435-564c-6883-21ba28fb987b}" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: pipelining replication: peer="{Voter server-2948bb85-9435-564c-6883-21ba28fb987b 2948bb85-9435-564c-6883-21ba28fb987b}" testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: pipelining replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.406Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: pipelining replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.511Z [INFO] stable state for Follower reached at 2025-02-09 06:52:31.406548301 +0000 UTC m=+12.385357065 (2 nodes), 64.2697ms from start of poll, 64.654679ms from cluster start. Timeout at 2025-02-09 06:52:31.511859974 +0000 UTC m=+12.490668741, 105.311676ms after stability testing.go:149: cluster: 2025-02-09T06:52:31.511Z [INFO] stable state for Follower reached at 2025-02-09 06:52:31.406548301 +0000 UTC m=+12.385357065 (2 nodes), 64.2697ms from start of poll, 64.654679ms from cluster start. Timeout at 2025-02-09 06:52:31.511859974 +0000 UTC m=+12.490668741, 105.311676ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.511Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:31.511Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.617Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.511933975 +0000 UTC m=+12.490742738 (1 nodes), 481ns from start of poll, 170.040352ms from cluster start. Timeout at 2025-02-09 06:52:31.617750979 +0000 UTC m=+12.596559739, 105.817001ms after stability testing.go:149: cluster: 2025-02-09T06:52:31.617Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.511933975 +0000 UTC m=+12.490742738 (1 nodes), 481ns from start of poll, 170.040352ms from cluster start. Timeout at 2025-02-09 06:52:31.617750979 +0000 UTC m=+12.596559739, 105.817001ms after stability testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.617Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to pipeline appendEntries: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" error="append pipeline closed" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.617Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to pipeline appendEntries: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" error="append pipeline closed" testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.617Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: aborting pipeline replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.617Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: aborting pipeline replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.617Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to appendEntries to: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" error="failed to connect to peer: 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.617Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to appendEntries to: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" error="failed to connect to peer: 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.619Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: starting snapshot up to: index=102 testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.619Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: starting snapshot up to: index=102 testing.go:153: 2025-02-09T06:52:31.619Z [INFO] creating new snapshot: path=/tmp/raft4257923744/snapshots/2-102-1739083951619.tmp testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.623Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: compacting logs: from=1 to=92 testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.623Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.623Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: snapshot complete up to: index=102 testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.623Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: snapshot complete up to: index=102 testLoggerAdapter verbose: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.623Z [ERROR] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.623Z [ERROR] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.623Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: starting snapshot up to: index=91 testing.go:149: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.623Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: starting snapshot up to: index=91 testing.go:153: 2025-02-09T06:52:31.623Z [INFO] creating new snapshot: path=/tmp/raft2201791858/snapshots/2-91-1739083951623.tmp testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.624Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to heartbeat to: peer=9a864b8b-b5cb-2054-a3b5-4d03564f8f1f error="failed to connect to peer: 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.624Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to heartbeat to: peer=9a864b8b-b5cb-2054-a3b5-4d03564f8f1f error="failed to connect to peer: 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f" testLoggerAdapter verbose: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.625Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: compacting logs: from=1 to=91 testing.go:149: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.625Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: compacting logs: from=1 to=91 testLoggerAdapter verbose: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.625Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: snapshot complete up to: index=91 testing.go:149: server-2948bb85-9435-564c-6883-21ba28fb987b: 2025-02-09T06:52:31.625Z [INFO] server-2948bb85-9435-564c-6883-21ba28fb987b: snapshot complete up to: index=91 testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.628Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to get log: index=2 error="log not found" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.628Z [ERROR] server-f27e3834-1e25-de4f-7758-0769fcf30041: failed to get log: index=2 error="log not found" testing.go:153: 2025-02-09T06:52:31.629Z [INFO] creating new snapshot: path=/tmp/raft1433449189/snapshots/2-102-1739083951629.tmp testLoggerAdapter verbose: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.632Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: copied to local snapshot: bytes=693 testing.go:149: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.632Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.632Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: Installed remote snapshot testing.go:149: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.632Z [INFO] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: Installed remote snapshot testLoggerAdapter verbose: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.632Z [WARN] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: skipping application of old log: index=102 testing.go:149: server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: 2025-02-09T06:52:31.632Z [WARN] server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f: skipping application of old log: index=102 testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.632Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: pipelining replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.632Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: pipelining replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.636Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: aborting pipeline replication: peer="{Voter server-2948bb85-9435-564c-6883-21ba28fb987b 2948bb85-9435-564c-6883-21ba28fb987b}" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.636Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: aborting pipeline replication: peer="{Voter server-2948bb85-9435-564c-6883-21ba28fb987b 2948bb85-9435-564c-6883-21ba28fb987b}" testLoggerAdapter verbose: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.636Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: aborting pipeline replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" testing.go:149: server-f27e3834-1e25-de4f-7758-0769fcf30041: 2025-02-09T06:52:31.636Z [INFO] server-f27e3834-1e25-de4f-7758-0769fcf30041: aborting pipeline replication: peer="{Voter server-9a864b8b-b5cb-2054-a3b5-4d03564f8f1f 9a864b8b-b5cb-2054-a3b5-4d03564f8f1f}" --- PASS: TestRaft_SendSnapshotFollower (0.30s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.637Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a7755d-3460-4753-8090-799f13ced7da Address:16a7755d-3460-4753-8090-799f13ced7da} {Suffrage:Voter ID:server-113f0681-7acf-736a-2a6e-1cfb35ac2cce Address:113f0681-7acf-736a-2a6e-1cfb35ac2cce} {Suffrage:Voter ID:server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 Address:0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}]" testing.go:149: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.637Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a7755d-3460-4753-8090-799f13ced7da Address:16a7755d-3460-4753-8090-799f13ced7da} {Suffrage:Voter ID:server-113f0681-7acf-736a-2a6e-1cfb35ac2cce Address:113f0681-7acf-736a-2a6e-1cfb35ac2cce} {Suffrage:Voter ID:server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 Address:0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}]" testLoggerAdapter verbose: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.637Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: entering follower state: follower="Node at 16a7755d-3460-4753-8090-799f13ced7da [Follower]" leader= testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.637Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a7755d-3460-4753-8090-799f13ced7da Address:16a7755d-3460-4753-8090-799f13ced7da} {Suffrage:Voter ID:server-113f0681-7acf-736a-2a6e-1cfb35ac2cce Address:113f0681-7acf-736a-2a6e-1cfb35ac2cce} {Suffrage:Voter ID:server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 Address:0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}]" testing.go:149: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.637Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: entering follower state: follower="Node at 16a7755d-3460-4753-8090-799f13ced7da [Follower]" leader= testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.637Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a7755d-3460-4753-8090-799f13ced7da Address:16a7755d-3460-4753-8090-799f13ced7da} {Suffrage:Voter ID:server-113f0681-7acf-736a-2a6e-1cfb35ac2cce Address:113f0681-7acf-736a-2a6e-1cfb35ac2cce} {Suffrage:Voter ID:server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 Address:0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}]" testLoggerAdapter verbose: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.638Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a7755d-3460-4753-8090-799f13ced7da Address:16a7755d-3460-4753-8090-799f13ced7da} {Suffrage:Voter ID:server-113f0681-7acf-736a-2a6e-1cfb35ac2cce Address:113f0681-7acf-736a-2a6e-1cfb35ac2cce} {Suffrage:Voter ID:server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 Address:0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}]" testing.go:149: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.638Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16a7755d-3460-4753-8090-799f13ced7da Address:16a7755d-3460-4753-8090-799f13ced7da} {Suffrage:Voter ID:server-113f0681-7acf-736a-2a6e-1cfb35ac2cce Address:113f0681-7acf-736a-2a6e-1cfb35ac2cce} {Suffrage:Voter ID:server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 Address:0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.638Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:31.638Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.638Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: entering follower state: follower="Node at 113f0681-7acf-736a-2a6e-1cfb35ac2cce [Follower]" leader= testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.638Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: entering follower state: follower="Node at 113f0681-7acf-736a-2a6e-1cfb35ac2cce [Follower]" leader= testLoggerAdapter verbose: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.638Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: entering follower state: follower="Node at 0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 [Follower]" leader= testing.go:149: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.638Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: entering follower state: follower="Node at 0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 [Follower]" leader= testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [WARN] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [WARN] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: entering candidate state: node="Node at 113f0681-7acf-736a-2a6e-1cfb35ac2cce [Candidate]" term=2 testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: entering candidate state: node="Node at 113f0681-7acf-736a-2a6e-1cfb35ac2cce [Candidate]" term=2 testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: election won: tally=2 testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: election won: tally=2 testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: entering leader state: leader="Node at 113f0681-7acf-736a-2a6e-1cfb35ac2cce [Leader]" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: entering leader state: leader="Node at 113f0681-7acf-736a-2a6e-1cfb35ac2cce [Leader]" testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: added peer, starting replication: peer=server-16a7755d-3460-4753-8090-799f13ced7da testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: added peer, starting replication: peer=server-16a7755d-3460-4753-8090-799f13ced7da testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: added peer, starting replication: peer=server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: added peer, starting replication: peer=server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: pipelining replication: peer="{Voter server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: pipelining replication: peer="{Voter server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}" testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: pipelining replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.691Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: pipelining replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.797Z [INFO] stable state for Follower reached at 2025-02-09 06:52:31.691773644 +0000 UTC m=+12.670582415 (2 nodes), 53.590529ms from start of poll, 54.090753ms from cluster start. Timeout at 2025-02-09 06:52:31.79767778 +0000 UTC m=+12.776486537, 105.904122ms after stability testing.go:149: cluster: 2025-02-09T06:52:31.797Z [INFO] stable state for Follower reached at 2025-02-09 06:52:31.691773644 +0000 UTC m=+12.670582415 (2 nodes), 53.590529ms from start of poll, 54.090753ms from cluster start. Timeout at 2025-02-09 06:52:31.79767778 +0000 UTC m=+12.776486537, 105.904122ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.797Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:31.797Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.903Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.797761631 +0000 UTC m=+12.776570404 (1 nodes), 529ns from start of poll, 160.078742ms from cluster start. Timeout at 2025-02-09 06:52:31.903304496 +0000 UTC m=+12.882113254, 105.54285ms after stability testing.go:149: cluster: 2025-02-09T06:52:31.903Z [INFO] stable state for Leader reached at 2025-02-09 06:52:31.797761631 +0000 UTC m=+12.776570404 (1 nodes), 529ns from start of poll, 160.078742ms from cluster start. Timeout at 2025-02-09 06:52:31.903304496 +0000 UTC m=+12.882113254, 105.54285ms after stability testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.903Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to pipeline appendEntries: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" error="append pipeline closed" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.903Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to pipeline appendEntries: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" error="append pipeline closed" testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.903Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: aborting pipeline replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.903Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: aborting pipeline replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.903Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to appendEntries to: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" error="failed to connect to peer: 16a7755d-3460-4753-8090-799f13ced7da" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.903Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to appendEntries to: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" error="failed to connect to peer: 16a7755d-3460-4753-8090-799f13ced7da" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.904Z [ERROR] server-16a7755d-3460-4753-8090-799f13ced7da: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.904Z [ERROR] server-16a7755d-3460-4753-8090-799f13ced7da: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.904Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: starting snapshot up to: index=102 testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.904Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: starting snapshot up to: index=102 testing.go:153: 2025-02-09T06:52:31.904Z [INFO] creating new snapshot: path=/tmp/raft403308337/snapshots/2-102-1739083951904.tmp testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.908Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: compacting logs: from=1 to=92 testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.908Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.908Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: snapshot complete up to: index=102 testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.908Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: snapshot complete up to: index=102 testLoggerAdapter verbose: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.908Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: starting snapshot up to: index=98 testing.go:149: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.908Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: starting snapshot up to: index=98 testing.go:153: 2025-02-09T06:52:31.908Z [INFO] creating new snapshot: path=/tmp/raft3275006271/snapshots/2-98-1739083951908.tmp testLoggerAdapter verbose: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.910Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: compacting logs: from=1 to=92 testing.go:149: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.910Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.910Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: snapshot complete up to: index=98 testing.go:149: server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: 2025-02-09T06:52:31.910Z [INFO] server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54: snapshot complete up to: index=98 testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.911Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to heartbeat to: peer=16a7755d-3460-4753-8090-799f13ced7da error="failed to connect to peer: 16a7755d-3460-4753-8090-799f13ced7da" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.911Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to heartbeat to: peer=16a7755d-3460-4753-8090-799f13ced7da error="failed to connect to peer: 16a7755d-3460-4753-8090-799f13ced7da" raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.914Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to get log: index=2 error="log not found" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.914Z [ERROR] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: failed to get log: index=2 error="log not found" testing.go:153: 2025-02-09T06:52:31.914Z [INFO] creating new snapshot: path=/tmp/raft1873730546/snapshots/2-102-1739083951914.tmp testLoggerAdapter verbose: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.917Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: copied to local snapshot: bytes=693 testing.go:149: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.917Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.917Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: Installed remote snapshot testing.go:149: server-16a7755d-3460-4753-8090-799f13ced7da: 2025-02-09T06:52:31.917Z [INFO] server-16a7755d-3460-4753-8090-799f13ced7da: Installed remote snapshot testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.917Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: pipelining replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.917Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: pipelining replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.923Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: aborting pipeline replication: peer="{Voter server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.923Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: aborting pipeline replication: peer="{Voter server-0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54 0f3aeac5-6e53-cb1e-55ef-d477f4dbcc54}" testLoggerAdapter verbose: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.923Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: aborting pipeline replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" testing.go:149: server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: 2025-02-09T06:52:31.923Z [INFO] server-113f0681-7acf-736a-2a6e-1cfb35ac2cce: aborting pipeline replication: peer="{Voter server-16a7755d-3460-4753-8090-799f13ced7da 16a7755d-3460-4753-8090-799f13ced7da}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:31.924Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7} {Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testing.go:149: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:31.924Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7} {Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testLoggerAdapter verbose: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:31.925Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: entering follower state: follower="Node at 60273cff-8d8e-d83e-9886-d08cd479f1e7 [Follower]" leader= testing.go:149: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:31.925Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: entering follower state: follower="Node at 60273cff-8d8e-d83e-9886-d08cd479f1e7 [Follower]" leader= testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:31.925Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7} {Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:31.925Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7} {Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:31.925Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering follower state: follower="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Follower]" leader= testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:31.925Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering follower state: follower="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Follower]" leader= testLoggerAdapter verbose: server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: 2025-02-09T06:52:31.925Z [INFO] server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7} {Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testing.go:149: server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: 2025-02-09T06:52:31.925Z [INFO] server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7} {Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:31.925Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:31.925Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: 2025-02-09T06:52:31.925Z [INFO] server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: entering follower state: follower="Node at 7db1d716-1992-8bf9-f1c0-322e0890f7f2 [Follower]" leader= testing.go:149: server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: 2025-02-09T06:52:31.925Z [INFO] server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: entering follower state: follower="Node at 7db1d716-1992-8bf9-f1c0-322e0890f7f2 [Follower]" leader= testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering candidate state: node="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Candidate]" term=2 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering candidate state: node="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Candidate]" term=2 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: election won: tally=2 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: election won: tally=2 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering leader state: leader="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Leader]" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering leader state: leader="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Leader]" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.013Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.118Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.013160956 +0000 UTC m=+12.991969741 (1 nodes), 87.761598ms from start of poll, 88.340621ms from cluster start. Timeout at 2025-02-09 06:52:32.118886188 +0000 UTC m=+13.097694947, 105.725206ms after stability testing.go:149: cluster: 2025-02-09T06:52:32.118Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.013160956 +0000 UTC m=+12.991969741 (1 nodes), 87.761598ms from start of poll, 88.340621ms from cluster start. Timeout at 2025-02-09 06:52:32.118886188 +0000 UTC m=+13.097694947, 105.725206ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.124Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:32.124Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.229Z [INFO] stable state for Follower reached at 2025-02-09 06:52:32.124503987 +0000 UTC m=+13.103312765 (2 nodes), 699ns from start of poll, 199.683645ms from cluster start. Timeout at 2025-02-09 06:52:32.229749769 +0000 UTC m=+13.208558541, 105.245776ms after stability testing.go:149: cluster: 2025-02-09T06:52:32.229Z [INFO] stable state for Follower reached at 2025-02-09 06:52:32.124503987 +0000 UTC m=+13.103312765 (2 nodes), 699ns from start of poll, 199.683645ms from cluster start. Timeout at 2025-02-09 06:52:32.229749769 +0000 UTC m=+13.208558541, 105.245776ms after stability testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.229Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: updating configuration: command=RemoveServer server-id=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 server-addr= servers="[{Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.229Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: updating configuration: command=RemoveServer server-id=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 server-addr= servers="[{Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2}]" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.229Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: removed peer, stopping replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 last-index=3 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.229Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: removed peer, stopping replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 last-index=3 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.229Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.229Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testLoggerAdapter verbose: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.335Z [WARN] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: heartbeat timeout reached, starting election: last-leader=8de14cca-22ff-e432-6df6-16f277011295 testing.go:149: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.335Z [WARN] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: heartbeat timeout reached, starting election: last-leader=8de14cca-22ff-e432-6df6-16f277011295 testLoggerAdapter verbose: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.335Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: entering candidate state: node="Node at 60273cff-8d8e-d83e-9886-d08cd479f1e7 [Candidate]" term=3 testing.go:149: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.335Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: entering candidate state: node="Node at 60273cff-8d8e-d83e-9886-d08cd479f1e7 [Candidate]" term=3 testLoggerAdapter verbose: server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: 2025-02-09T06:52:32.335Z [WARN] server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: rejecting vote request since we have a leader: from=60273cff-8d8e-d83e-9886-d08cd479f1e7 leader=8de14cca-22ff-e432-6df6-16f277011295 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: rejecting vote request since we have a leader: from=60273cff-8d8e-d83e-9886-d08cd479f1e7 leader=8de14cca-22ff-e432-6df6-16f277011295 testing.go:149: server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: 2025-02-09T06:52:32.335Z [WARN] server-7db1d716-1992-8bf9-f1c0-322e0890f7f2: rejecting vote request since we have a leader: from=60273cff-8d8e-d83e-9886-d08cd479f1e7 leader=8de14cca-22ff-e432-6df6-16f277011295 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: rejecting vote request since we have a leader: from=60273cff-8d8e-d83e-9886-d08cd479f1e7 leader=8de14cca-22ff-e432-6df6-16f277011295 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: updating configuration: command=AddStaging server-id=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 server-addr=60273cff-8d8e-d83e-9886-d08cd479f1e7 servers="[{Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2} {Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7}]" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: updating configuration: command=AddStaging server-id=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 server-addr=60273cff-8d8e-d83e-9886-d08cd479f1e7 servers="[{Suffrage:Voter ID:server-8de14cca-22ff-e432-6df6-16f277011295 Address:8de14cca-22ff-e432-6df6-16f277011295} {Suffrage:Voter ID:server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 Address:7db1d716-1992-8bf9-f1c0-322e0890f7f2} {Suffrage:Voter ID:server-60273cff-8d8e-d83e-9886-d08cd479f1e7 Address:60273cff-8d8e-d83e-9886-d08cd479f1e7}]" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [ERROR] server-8de14cca-22ff-e432-6df6-16f277011295: peer has newer term, stopping replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [ERROR] server-8de14cca-22ff-e432-6df6-16f277011295: peer has newer term, stopping replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.335Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering follower state: follower="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Follower]" leader= testing.go:149: cluster: 2025-02-09T06:52:32.335Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering follower state: follower="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Follower]" leader= testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.335Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.425Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.425Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering candidate state: node="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Candidate]" term=3 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering candidate state: node="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Candidate]" term=3 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: election won: tally=2 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: election won: tally=2 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering leader state: leader="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Leader]" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: entering leader state: leader="Node at 8de14cca-22ff-e432-6df6-16f277011295 [Leader]" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: added peer, starting replication: peer=server-60273cff-8d8e-d83e-9886-d08cd479f1e7 testLoggerAdapter verbose: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.426Z [WARN] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.426Z [WARN] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.426Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: entering follower state: follower="Node at 60273cff-8d8e-d83e-9886-d08cd479f1e7 [Follower]" leader=8de14cca-22ff-e432-6df6-16f277011295 testing.go:149: server-60273cff-8d8e-d83e-9886-d08cd479f1e7: 2025-02-09T06:52:32.426Z [INFO] server-60273cff-8d8e-d83e-9886-d08cd479f1e7: entering follower state: follower="Node at 60273cff-8d8e-d83e-9886-d08cd479f1e7 [Follower]" leader=8de14cca-22ff-e432-6df6-16f277011295 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: appendEntries rejected, sending older logs: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" next=4 testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [WARN] server-8de14cca-22ff-e432-6df6-16f277011295: appendEntries rejected, sending older logs: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" next=4 testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.426Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: pipelining replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.532Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.426209818 +0000 UTC m=+13.405018581 (1 nodes), 90.227403ms from start of poll, 501.389461ms from cluster start. Timeout at 2025-02-09 06:52:32.532046595 +0000 UTC m=+13.510855353, 105.836772ms after stability testing.go:149: cluster: 2025-02-09T06:52:32.532Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.426209818 +0000 UTC m=+13.405018581 (1 nodes), 90.227403ms from start of poll, 501.389461ms from cluster start. Timeout at 2025-02-09 06:52:32.532046595 +0000 UTC m=+13.510855353, 105.836772ms after stability testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.532Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.532Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-60273cff-8d8e-d83e-9886-d08cd479f1e7 60273cff-8d8e-d83e-9886-d08cd479f1e7}" testLoggerAdapter verbose: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.532Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" testing.go:149: server-8de14cca-22ff-e432-6df6-16f277011295: 2025-02-09T06:52:32.532Z [INFO] server-8de14cca-22ff-e432-6df6-16f277011295: aborting pipeline replication: peer="{Voter server-7db1d716-1992-8bf9-f1c0-322e0890f7f2 7db1d716-1992-8bf9-f1c0-322e0890f7f2}" --- PASS: TestRaft_ReJoinFollower (0.61s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.533Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f7294cd-deaa-11d5-cabd-b4cfb810832f Address:2f7294cd-deaa-11d5-cabd-b4cfb810832f} {Suffrage:Voter ID:server-b85161ed-805a-30a5-72d2-168ba2464710 Address:b85161ed-805a-30a5-72d2-168ba2464710}]" testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.533Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f7294cd-deaa-11d5-cabd-b4cfb810832f Address:2f7294cd-deaa-11d5-cabd-b4cfb810832f} {Suffrage:Voter ID:server-b85161ed-805a-30a5-72d2-168ba2464710 Address:b85161ed-805a-30a5-72d2-168ba2464710}]" testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.533Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering follower state: follower="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Follower]" leader= testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.533Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering follower state: follower="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Follower]" leader= testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.533Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f7294cd-deaa-11d5-cabd-b4cfb810832f Address:2f7294cd-deaa-11d5-cabd-b4cfb810832f} {Suffrage:Voter ID:server-b85161ed-805a-30a5-72d2-168ba2464710 Address:b85161ed-805a-30a5-72d2-168ba2464710}]" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.533Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f7294cd-deaa-11d5-cabd-b4cfb810832f Address:2f7294cd-deaa-11d5-cabd-b4cfb810832f} {Suffrage:Voter ID:server-b85161ed-805a-30a5-72d2-168ba2464710 Address:b85161ed-805a-30a5-72d2-168ba2464710}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.533Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:32.533Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.533Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering follower state: follower="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Follower]" leader= testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.533Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering follower state: follower="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Follower]" leader= testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.619Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.619Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=2 testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=2 testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: election won: tally=2 testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: election won: tally=2 testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering leader state: leader="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Leader]" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering leader state: leader="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Leader]" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: added peer, starting replication: peer=server-2f7294cd-deaa-11d5-cabd-b4cfb810832f testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: added peer, starting replication: peer=server-2f7294cd-deaa-11d5-cabd-b4cfb810832f testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: pipelining replication: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.620Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: pipelining replication: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.725Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.62016912 +0000 UTC m=+13.598977897 (1 nodes), 86.907775ms from start of poll, 87.214905ms from cluster start. Timeout at 2025-02-09 06:52:32.725517666 +0000 UTC m=+13.704326424, 105.348527ms after stability testing.go:149: cluster: 2025-02-09T06:52:32.725Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.62016912 +0000 UTC m=+13.598977897 (1 nodes), 86.907775ms from start of poll, 87.214905ms from cluster start. Timeout at 2025-02-09 06:52:32.725517666 +0000 UTC m=+13.704326424, 105.348527ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.730Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:32.730Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.836Z [INFO] stable state for Follower reached at 2025-02-09 06:52:32.730997312 +0000 UTC m=+13.709806077 (1 nodes), 442ns from start of poll, 198.043085ms from cluster start. Timeout at 2025-02-09 06:52:32.836008408 +0000 UTC m=+13.814817176, 105.011099ms after stability testing.go:149: cluster: 2025-02-09T06:52:32.836Z [INFO] stable state for Follower reached at 2025-02-09 06:52:32.730997312 +0000 UTC m=+13.709806077 (1 nodes), 442ns from start of poll, 198.043085ms from cluster start. Timeout at 2025-02-09 06:52:32.836008408 +0000 UTC m=+13.814817176, 105.011099ms after stability raft_test.go:1492: [INFO] Disconnecting Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Follower] testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.840Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to heartbeat to: peer=2f7294cd-deaa-11d5-cabd-b4cfb810832f error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.840Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to heartbeat to: peer=2f7294cd-deaa-11d5-cabd-b4cfb810832f error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.840Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to pipeline appendEntries: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="append pipeline closed" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.840Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to pipeline appendEntries: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="append pipeline closed" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.840Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: aborting pipeline replication: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.840Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: aborting pipeline replication: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.847Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to appendEntries to: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.847Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to appendEntries to: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.858Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to heartbeat to: peer=2f7294cd-deaa-11d5-cabd-b4cfb810832f error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.858Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to heartbeat to: peer=2f7294cd-deaa-11d5-cabd-b4cfb810832f error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.866Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to appendEntries to: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.866Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to appendEntries to: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.873Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to heartbeat to: peer=2f7294cd-deaa-11d5-cabd-b4cfb810832f error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.873Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to heartbeat to: peer=2f7294cd-deaa-11d5-cabd-b4cfb810832f error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.881Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to contact: server-id=server-2f7294cd-deaa-11d5-cabd-b4cfb810832f time=50.588458ms testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.881Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to contact: server-id=server-2f7294cd-deaa-11d5-cabd-b4cfb810832f time=50.588458ms testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.881Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to contact quorum of nodes, stepping down testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.881Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.881Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering follower state: follower="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Follower]" leader= testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.881Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering follower state: follower="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T06:52:32.881Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:32.881Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.882Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to appendEntries to: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.882Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to appendEntries to: peer="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.908Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: heartbeat timeout reached, starting election: last-leader=b85161ed-805a-30a5-72d2-168ba2464710 testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.908Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: heartbeat timeout reached, starting election: last-leader=b85161ed-805a-30a5-72d2-168ba2464710 testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.908Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=3 testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.908Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=3 testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.908Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.908Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.972Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.972Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.972Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=3 testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.972Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=3 testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.972Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to make requestVote RPC: target="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:32.972Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to make requestVote RPC: target="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.980Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: Election timeout reached, restarting election testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.980Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: Election timeout reached, restarting election testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.980Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=4 testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.980Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=4 testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.980Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:32.980Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.064Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: Election timeout reached, restarting election testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.064Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: Election timeout reached, restarting election testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.064Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=4 testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.064Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=4 testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.064Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to make requestVote RPC: target="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.064Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to make requestVote RPC: target="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.070Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: Election timeout reached, restarting election testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.070Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: Election timeout reached, restarting election testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.070Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=5 testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.070Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=5 testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.070Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.070Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.078Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.973002961 +0000 UTC m=+13.951811735 (0 nodes), 91.69049ms from start of poll, 440.048743ms from cluster start. Timeout at 2025-02-09 06:52:33.07841625 +0000 UTC m=+14.057225014, 105.413279ms after stability testing.go:149: cluster: 2025-02-09T06:52:33.078Z [INFO] stable state for Leader reached at 2025-02-09 06:52:32.973002961 +0000 UTC m=+13.951811735 (0 nodes), 91.69049ms from start of poll, 440.048743ms from cluster start. Timeout at 2025-02-09 06:52:33.07841625 +0000 UTC m=+14.057225014, 105.413279ms after stability testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.127Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: Election timeout reached, restarting election testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.127Z [WARN] server-b85161ed-805a-30a5-72d2-168ba2464710: Election timeout reached, restarting election testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.127Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=5 testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.127Z [INFO] server-b85161ed-805a-30a5-72d2-168ba2464710: entering candidate state: node="Node at b85161ed-805a-30a5-72d2-168ba2464710 [Candidate]" term=5 testLoggerAdapter verbose: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.127Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to make requestVote RPC: target="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testing.go:149: server-b85161ed-805a-30a5-72d2-168ba2464710: 2025-02-09T06:52:33.127Z [ERROR] server-b85161ed-805a-30a5-72d2-168ba2464710: failed to make requestVote RPC: target="{Voter server-2f7294cd-deaa-11d5-cabd-b4cfb810832f 2f7294cd-deaa-11d5-cabd-b4cfb810832f}" error="failed to connect to peer: 2f7294cd-deaa-11d5-cabd-b4cfb810832f" testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.149Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: Election timeout reached, restarting election testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.149Z [WARN] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: Election timeout reached, restarting election testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.149Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=6 testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.149Z [INFO] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: entering candidate state: node="Node at 2f7294cd-deaa-11d5-cabd-b4cfb810832f [Candidate]" term=6 testLoggerAdapter verbose: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.149Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" testing.go:149: server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: 2025-02-09T06:52:33.149Z [ERROR] server-2f7294cd-deaa-11d5-cabd-b4cfb810832f: failed to make requestVote RPC: target="{Voter server-b85161ed-805a-30a5-72d2-168ba2464710 b85161ed-805a-30a5-72d2-168ba2464710}" error="failed to connect to peer: b85161ed-805a-30a5-72d2-168ba2464710" --- PASS: TestRaft_LeaderLeaseExpire (0.65s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-93863f81-728f-8596-6af3-e3c555253058: 2025-02-09T06:52:33.184Z [INFO] server-93863f81-728f-8596-6af3-e3c555253058: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93863f81-728f-8596-6af3-e3c555253058 Address:93863f81-728f-8596-6af3-e3c555253058} {Suffrage:Voter ID:server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608 Address:06fa92f0-2d9d-a0ea-46ee-01ba1714c608} {Suffrage:Voter ID:server-414f9659-6384-ab8e-b782-684e9b49be12 Address:414f9659-6384-ab8e-b782-684e9b49be12}]" testing.go:149: server-93863f81-728f-8596-6af3-e3c555253058: 2025-02-09T06:52:33.184Z [INFO] server-93863f81-728f-8596-6af3-e3c555253058: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93863f81-728f-8596-6af3-e3c555253058 Address:93863f81-728f-8596-6af3-e3c555253058} {Suffrage:Voter ID:server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608 Address:06fa92f0-2d9d-a0ea-46ee-01ba1714c608} {Suffrage:Voter ID:server-414f9659-6384-ab8e-b782-684e9b49be12 Address:414f9659-6384-ab8e-b782-684e9b49be12}]" testLoggerAdapter verbose: server-93863f81-728f-8596-6af3-e3c555253058: 2025-02-09T06:52:33.184Z [INFO] server-93863f81-728f-8596-6af3-e3c555253058: entering follower state: follower="Node at 93863f81-728f-8596-6af3-e3c555253058 [Follower]" leader= testing.go:149: server-93863f81-728f-8596-6af3-e3c555253058: 2025-02-09T06:52:33.184Z [INFO] server-93863f81-728f-8596-6af3-e3c555253058: entering follower state: follower="Node at 93863f81-728f-8596-6af3-e3c555253058 [Follower]" leader= testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.184Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93863f81-728f-8596-6af3-e3c555253058 Address:93863f81-728f-8596-6af3-e3c555253058} {Suffrage:Voter ID:server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608 Address:06fa92f0-2d9d-a0ea-46ee-01ba1714c608} {Suffrage:Voter ID:server-414f9659-6384-ab8e-b782-684e9b49be12 Address:414f9659-6384-ab8e-b782-684e9b49be12}]" testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.184Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93863f81-728f-8596-6af3-e3c555253058 Address:93863f81-728f-8596-6af3-e3c555253058} {Suffrage:Voter ID:server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608 Address:06fa92f0-2d9d-a0ea-46ee-01ba1714c608} {Suffrage:Voter ID:server-414f9659-6384-ab8e-b782-684e9b49be12 Address:414f9659-6384-ab8e-b782-684e9b49be12}]" testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.184Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: entering follower state: follower="Node at 06fa92f0-2d9d-a0ea-46ee-01ba1714c608 [Follower]" leader= testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.184Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: entering follower state: follower="Node at 06fa92f0-2d9d-a0ea-46ee-01ba1714c608 [Follower]" leader= testLoggerAdapter verbose: server-414f9659-6384-ab8e-b782-684e9b49be12: 2025-02-09T06:52:33.184Z [INFO] server-414f9659-6384-ab8e-b782-684e9b49be12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93863f81-728f-8596-6af3-e3c555253058 Address:93863f81-728f-8596-6af3-e3c555253058} {Suffrage:Voter ID:server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608 Address:06fa92f0-2d9d-a0ea-46ee-01ba1714c608} {Suffrage:Voter ID:server-414f9659-6384-ab8e-b782-684e9b49be12 Address:414f9659-6384-ab8e-b782-684e9b49be12}]" testing.go:149: server-414f9659-6384-ab8e-b782-684e9b49be12: 2025-02-09T06:52:33.184Z [INFO] server-414f9659-6384-ab8e-b782-684e9b49be12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93863f81-728f-8596-6af3-e3c555253058 Address:93863f81-728f-8596-6af3-e3c555253058} {Suffrage:Voter ID:server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608 Address:06fa92f0-2d9d-a0ea-46ee-01ba1714c608} {Suffrage:Voter ID:server-414f9659-6384-ab8e-b782-684e9b49be12 Address:414f9659-6384-ab8e-b782-684e9b49be12}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.184Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:33.184Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-414f9659-6384-ab8e-b782-684e9b49be12: 2025-02-09T06:52:33.184Z [INFO] server-414f9659-6384-ab8e-b782-684e9b49be12: entering follower state: follower="Node at 414f9659-6384-ab8e-b782-684e9b49be12 [Follower]" leader= testing.go:149: server-414f9659-6384-ab8e-b782-684e9b49be12: 2025-02-09T06:52:33.184Z [INFO] server-414f9659-6384-ab8e-b782-684e9b49be12: entering follower state: follower="Node at 414f9659-6384-ab8e-b782-684e9b49be12 [Follower]" leader= testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [WARN] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [WARN] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: entering candidate state: node="Node at 06fa92f0-2d9d-a0ea-46ee-01ba1714c608 [Candidate]" term=2 testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: entering candidate state: node="Node at 06fa92f0-2d9d-a0ea-46ee-01ba1714c608 [Candidate]" term=2 testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: election won: tally=2 testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: election won: tally=2 testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: entering leader state: leader="Node at 06fa92f0-2d9d-a0ea-46ee-01ba1714c608 [Leader]" testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: entering leader state: leader="Node at 06fa92f0-2d9d-a0ea-46ee-01ba1714c608 [Leader]" testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: added peer, starting replication: peer=server-93863f81-728f-8596-6af3-e3c555253058 testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: added peer, starting replication: peer=server-93863f81-728f-8596-6af3-e3c555253058 testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: added peer, starting replication: peer=server-414f9659-6384-ab8e-b782-684e9b49be12 testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: added peer, starting replication: peer=server-414f9659-6384-ab8e-b782-684e9b49be12 testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: pipelining replication: peer="{Voter server-93863f81-728f-8596-6af3-e3c555253058 93863f81-728f-8596-6af3-e3c555253058}" testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: pipelining replication: peer="{Voter server-93863f81-728f-8596-6af3-e3c555253058 93863f81-728f-8596-6af3-e3c555253058}" testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: pipelining replication: peer="{Voter server-414f9659-6384-ab8e-b782-684e9b49be12 414f9659-6384-ab8e-b782-684e9b49be12}" testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.244Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: pipelining replication: peer="{Voter server-414f9659-6384-ab8e-b782-684e9b49be12 414f9659-6384-ab8e-b782-684e9b49be12}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.350Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.244565371 +0000 UTC m=+14.223374150 (1 nodes), 59.702197ms from start of poll, 60.23214ms from cluster start. Timeout at 2025-02-09 06:52:33.35020395 +0000 UTC m=+14.329012720, 105.63857ms after stability testing.go:149: cluster: 2025-02-09T06:52:33.350Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.244565371 +0000 UTC m=+14.223374150 (1 nodes), 59.702197ms from start of poll, 60.23214ms from cluster start. Timeout at 2025-02-09 06:52:33.35020395 +0000 UTC m=+14.329012720, 105.63857ms after stability testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.363Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: aborting pipeline replication: peer="{Voter server-414f9659-6384-ab8e-b782-684e9b49be12 414f9659-6384-ab8e-b782-684e9b49be12}" testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.363Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: aborting pipeline replication: peer="{Voter server-414f9659-6384-ab8e-b782-684e9b49be12 414f9659-6384-ab8e-b782-684e9b49be12}" testLoggerAdapter verbose: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.363Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: aborting pipeline replication: peer="{Voter server-93863f81-728f-8596-6af3-e3c555253058 93863f81-728f-8596-6af3-e3c555253058}" testing.go:149: server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: 2025-02-09T06:52:33.363Z [INFO] server-06fa92f0-2d9d-a0ea-46ee-01ba1714c608: aborting pipeline replication: peer="{Voter server-93863f81-728f-8596-6af3-e3c555253058 93863f81-728f-8596-6af3-e3c555253058}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.364Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b Address:7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b} {Suffrage:Voter ID:server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d Address:addd2fd6-4777-b1ca-3cfd-ebb7e113383d} {Suffrage:Voter ID:server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 Address:f43b44e5-55d6-fdf6-eb31-2935691dbb35}]" testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.364Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b Address:7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b} {Suffrage:Voter ID:server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d Address:addd2fd6-4777-b1ca-3cfd-ebb7e113383d} {Suffrage:Voter ID:server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 Address:f43b44e5-55d6-fdf6-eb31-2935691dbb35}]" testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.364Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: entering follower state: follower="Node at 7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b [Follower]" leader= testLoggerAdapter verbose: server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: 2025-02-09T06:52:33.364Z [INFO] server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b Address:7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b} {Suffrage:Voter ID:server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d Address:addd2fd6-4777-b1ca-3cfd-ebb7e113383d} {Suffrage:Voter ID:server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 Address:f43b44e5-55d6-fdf6-eb31-2935691dbb35}]" testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.364Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: entering follower state: follower="Node at 7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b [Follower]" leader= testing.go:149: server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: 2025-02-09T06:52:33.364Z [INFO] server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b Address:7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b} {Suffrage:Voter ID:server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d Address:addd2fd6-4777-b1ca-3cfd-ebb7e113383d} {Suffrage:Voter ID:server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 Address:f43b44e5-55d6-fdf6-eb31-2935691dbb35}]" testLoggerAdapter verbose: server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: 2025-02-09T06:52:33.365Z [INFO] server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: entering follower state: follower="Node at addd2fd6-4777-b1ca-3cfd-ebb7e113383d [Follower]" leader= testing.go:149: server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: 2025-02-09T06:52:33.365Z [INFO] server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d: entering follower state: follower="Node at addd2fd6-4777-b1ca-3cfd-ebb7e113383d [Follower]" leader= testLoggerAdapter verbose: server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: 2025-02-09T06:52:33.365Z [INFO] server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b Address:7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b} {Suffrage:Voter ID:server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d Address:addd2fd6-4777-b1ca-3cfd-ebb7e113383d} {Suffrage:Voter ID:server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 Address:f43b44e5-55d6-fdf6-eb31-2935691dbb35}]" testing.go:149: server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: 2025-02-09T06:52:33.365Z [INFO] server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b Address:7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b} {Suffrage:Voter ID:server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d Address:addd2fd6-4777-b1ca-3cfd-ebb7e113383d} {Suffrage:Voter ID:server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 Address:f43b44e5-55d6-fdf6-eb31-2935691dbb35}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.365Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:33.365Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: 2025-02-09T06:52:33.365Z [INFO] server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: entering follower state: follower="Node at f43b44e5-55d6-fdf6-eb31-2935691dbb35 [Follower]" leader= testing.go:149: server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: 2025-02-09T06:52:33.365Z [INFO] server-f43b44e5-55d6-fdf6-eb31-2935691dbb35: entering follower state: follower="Node at f43b44e5-55d6-fdf6-eb31-2935691dbb35 [Follower]" leader= testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [WARN] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [WARN] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: entering candidate state: node="Node at 7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b [Candidate]" term=2 testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: entering candidate state: node="Node at 7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b [Candidate]" term=2 testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: election won: tally=2 testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: election won: tally=2 testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: entering leader state: leader="Node at 7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b [Leader]" testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: entering leader state: leader="Node at 7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b [Leader]" testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: added peer, starting replication: peer=server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: added peer, starting replication: peer=server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: added peer, starting replication: peer=server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: added peer, starting replication: peer=server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: pipelining replication: peer="{Voter server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 f43b44e5-55d6-fdf6-eb31-2935691dbb35}" testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.421Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: pipelining replication: peer="{Voter server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 f43b44e5-55d6-fdf6-eb31-2935691dbb35}" testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.422Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: pipelining replication: peer="{Voter server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d addd2fd6-4777-b1ca-3cfd-ebb7e113383d}" testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.422Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: pipelining replication: peer="{Voter server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d addd2fd6-4777-b1ca-3cfd-ebb7e113383d}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.527Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.421891365 +0000 UTC m=+14.400700130 (1 nodes), 56.738011ms from start of poll, 57.103172ms from cluster start. Timeout at 2025-02-09 06:52:33.52789275 +0000 UTC m=+14.506701520, 106.00139ms after stability testing.go:149: cluster: 2025-02-09T06:52:33.527Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.421891365 +0000 UTC m=+14.400700130 (1 nodes), 56.738011ms from start of poll, 57.103172ms from cluster start. Timeout at 2025-02-09 06:52:33.52789275 +0000 UTC m=+14.506701520, 106.00139ms after stability testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.527Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: aborting pipeline replication: peer="{Voter server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 f43b44e5-55d6-fdf6-eb31-2935691dbb35}" testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.527Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: aborting pipeline replication: peer="{Voter server-f43b44e5-55d6-fdf6-eb31-2935691dbb35 f43b44e5-55d6-fdf6-eb31-2935691dbb35}" testLoggerAdapter verbose: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.527Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: aborting pipeline replication: peer="{Voter server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d addd2fd6-4777-b1ca-3cfd-ebb7e113383d}" testing.go:149: server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: 2025-02-09T06:52:33.527Z [INFO] server-7bf3ec9b-c077-5aa5-b3e7-004aa0f1520b: aborting pipeline replication: peer="{Voter server-addd2fd6-4777-b1ca-3cfd-ebb7e113383d addd2fd6-4777-b1ca-3cfd-ebb7e113383d}" --- PASS: TestRaft_VerifyLeader (0.16s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.528Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6420c583-5e98-6da3-4606-3a28e7c27404 Address:6420c583-5e98-6da3-4606-3a28e7c27404}]" testing.go:149: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.528Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6420c583-5e98-6da3-4606-3a28e7c27404 Address:6420c583-5e98-6da3-4606-3a28e7c27404}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.528Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:33.528Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.528Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: entering follower state: follower="Node at 6420c583-5e98-6da3-4606-3a28e7c27404 [Follower]" leader= testing.go:149: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.528Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: entering follower state: follower="Node at 6420c583-5e98-6da3-4606-3a28e7c27404 [Follower]" leader= testLoggerAdapter verbose: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [WARN] server-6420c583-5e98-6da3-4606-3a28e7c27404: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [WARN] server-6420c583-5e98-6da3-4606-3a28e7c27404: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: entering candidate state: node="Node at 6420c583-5e98-6da3-4606-3a28e7c27404 [Candidate]" term=2 testing.go:149: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: entering candidate state: node="Node at 6420c583-5e98-6da3-4606-3a28e7c27404 [Candidate]" term=2 testLoggerAdapter verbose: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: election won: tally=1 testing.go:149: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: election won: tally=1 testLoggerAdapter verbose: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: entering leader state: leader="Node at 6420c583-5e98-6da3-4606-3a28e7c27404 [Leader]" testing.go:149: server-6420c583-5e98-6da3-4606-3a28e7c27404: 2025-02-09T06:52:33.602Z [INFO] server-6420c583-5e98-6da3-4606-3a28e7c27404: entering leader state: leader="Node at 6420c583-5e98-6da3-4606-3a28e7c27404 [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.707Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.602260638 +0000 UTC m=+14.581069404 (1 nodes), 73.468308ms from start of poll, 73.613404ms from cluster start. Timeout at 2025-02-09 06:52:33.70748026 +0000 UTC m=+14.686289034, 105.21963ms after stability testing.go:149: cluster: 2025-02-09T06:52:33.707Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.602260638 +0000 UTC m=+14.581069404 (1 nodes), 73.468308ms from start of poll, 73.613404ms from cluster start. Timeout at 2025-02-09 06:52:33.70748026 +0000 UTC m=+14.686289034, 105.21963ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-9e342e83-73bf-5e69-a63f-526682b23fce: 2025-02-09T06:52:33.708Z [INFO] server-9e342e83-73bf-5e69-a63f-526682b23fce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e342e83-73bf-5e69-a63f-526682b23fce Address:9e342e83-73bf-5e69-a63f-526682b23fce} {Suffrage:Voter ID:server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4 Address:f7c6d453-f2b6-2581-3bc1-7202fbec38c4}]" testing.go:149: server-9e342e83-73bf-5e69-a63f-526682b23fce: 2025-02-09T06:52:33.708Z [INFO] server-9e342e83-73bf-5e69-a63f-526682b23fce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e342e83-73bf-5e69-a63f-526682b23fce Address:9e342e83-73bf-5e69-a63f-526682b23fce} {Suffrage:Voter ID:server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4 Address:f7c6d453-f2b6-2581-3bc1-7202fbec38c4}]" testLoggerAdapter verbose: server-9e342e83-73bf-5e69-a63f-526682b23fce: 2025-02-09T06:52:33.708Z [INFO] server-9e342e83-73bf-5e69-a63f-526682b23fce: entering follower state: follower="Node at 9e342e83-73bf-5e69-a63f-526682b23fce [Follower]" leader= testing.go:149: server-9e342e83-73bf-5e69-a63f-526682b23fce: 2025-02-09T06:52:33.708Z [INFO] server-9e342e83-73bf-5e69-a63f-526682b23fce: entering follower state: follower="Node at 9e342e83-73bf-5e69-a63f-526682b23fce [Follower]" leader= testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.708Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e342e83-73bf-5e69-a63f-526682b23fce Address:9e342e83-73bf-5e69-a63f-526682b23fce} {Suffrage:Voter ID:server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4 Address:f7c6d453-f2b6-2581-3bc1-7202fbec38c4}]" testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.708Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e342e83-73bf-5e69-a63f-526682b23fce Address:9e342e83-73bf-5e69-a63f-526682b23fce} {Suffrage:Voter ID:server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4 Address:f7c6d453-f2b6-2581-3bc1-7202fbec38c4}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.708Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:33.708Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.708Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering follower state: follower="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Follower]" leader= testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.708Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering follower state: follower="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Follower]" leader= testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [WARN] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [WARN] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering candidate state: node="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Candidate]" term=2 testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering candidate state: node="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Candidate]" term=2 testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: election won: tally=2 testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: election won: tally=2 testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering leader state: leader="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Leader]" testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering leader state: leader="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Leader]" testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: added peer, starting replication: peer=server-9e342e83-73bf-5e69-a63f-526682b23fce testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: added peer, starting replication: peer=server-9e342e83-73bf-5e69-a63f-526682b23fce testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: pipelining replication: peer="{Voter server-9e342e83-73bf-5e69-a63f-526682b23fce 9e342e83-73bf-5e69-a63f-526682b23fce}" testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.779Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: pipelining replication: peer="{Voter server-9e342e83-73bf-5e69-a63f-526682b23fce 9e342e83-73bf-5e69-a63f-526682b23fce}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.884Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.779450425 +0000 UTC m=+14.758259193 (1 nodes), 71.137839ms from start of poll, 71.385887ms from cluster start. Timeout at 2025-02-09 06:52:33.884900735 +0000 UTC m=+14.863709514, 105.450321ms after stability testing.go:149: cluster: 2025-02-09T06:52:33.884Z [INFO] stable state for Leader reached at 2025-02-09 06:52:33.779450425 +0000 UTC m=+14.758259193 (1 nodes), 71.137839ms from start of poll, 71.385887ms from cluster start. Timeout at 2025-02-09 06:52:33.884900735 +0000 UTC m=+14.863709514, 105.450321ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.884Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:33.884Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.990Z [INFO] stable state for Follower reached at 2025-02-09 06:52:33.884982604 +0000 UTC m=+14.863791369 (1 nodes), 444ns from start of poll, 176.918063ms from cluster start. Timeout at 2025-02-09 06:52:33.990525314 +0000 UTC m=+14.969334072, 105.542703ms after stability testing.go:149: cluster: 2025-02-09T06:52:33.990Z [INFO] stable state for Follower reached at 2025-02-09 06:52:33.884982604 +0000 UTC m=+14.863791369 (1 nodes), 444ns from start of poll, 176.918063ms from cluster start. Timeout at 2025-02-09 06:52:33.990525314 +0000 UTC m=+14.969334072, 105.542703ms after stability testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.990Z [WARN] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: new leader elected, stepping down testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.990Z [WARN] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: new leader elected, stepping down testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.990Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering follower state: follower="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Follower]" leader= testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.990Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: entering follower state: follower="Node at f7c6d453-f2b6-2581-3bc1-7202fbec38c4 [Follower]" leader= testLoggerAdapter verbose: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.990Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: aborting pipeline replication: peer="{Voter server-9e342e83-73bf-5e69-a63f-526682b23fce 9e342e83-73bf-5e69-a63f-526682b23fce}" testing.go:149: server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: 2025-02-09T06:52:33.990Z [INFO] server-f7c6d453-f2b6-2581-3bc1-7202fbec38c4: aborting pipeline replication: peer="{Voter server-9e342e83-73bf-5e69-a63f-526682b23fce 9e342e83-73bf-5e69-a63f-526682b23fce}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:33.991Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d1599f6-da3c-0851-04df-2cc5787ef026 Address:9d1599f6-da3c-0851-04df-2cc5787ef026} {Suffrage:Voter ID:server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 Address:407cc3b4-66c0-c34b-5a67-82fda2c709c1} {Suffrage:Voter ID:server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba Address:df9f2c72-97eb-acff-c8ad-75ebb50a00ba}]" testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:33.991Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d1599f6-da3c-0851-04df-2cc5787ef026 Address:9d1599f6-da3c-0851-04df-2cc5787ef026} {Suffrage:Voter ID:server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 Address:407cc3b4-66c0-c34b-5a67-82fda2c709c1} {Suffrage:Voter ID:server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba Address:df9f2c72-97eb-acff-c8ad-75ebb50a00ba}]" testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:33.991Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: entering follower state: follower="Node at 9d1599f6-da3c-0851-04df-2cc5787ef026 [Follower]" leader= testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:33.991Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: entering follower state: follower="Node at 9d1599f6-da3c-0851-04df-2cc5787ef026 [Follower]" leader= testLoggerAdapter verbose: server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: 2025-02-09T06:52:33.991Z [INFO] server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d1599f6-da3c-0851-04df-2cc5787ef026 Address:9d1599f6-da3c-0851-04df-2cc5787ef026} {Suffrage:Voter ID:server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 Address:407cc3b4-66c0-c34b-5a67-82fda2c709c1} {Suffrage:Voter ID:server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba Address:df9f2c72-97eb-acff-c8ad-75ebb50a00ba}]" testing.go:149: server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: 2025-02-09T06:52:33.991Z [INFO] server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d1599f6-da3c-0851-04df-2cc5787ef026 Address:9d1599f6-da3c-0851-04df-2cc5787ef026} {Suffrage:Voter ID:server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 Address:407cc3b4-66c0-c34b-5a67-82fda2c709c1} {Suffrage:Voter ID:server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba Address:df9f2c72-97eb-acff-c8ad-75ebb50a00ba}]" testLoggerAdapter verbose: server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: 2025-02-09T06:52:33.991Z [INFO] server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: entering follower state: follower="Node at 407cc3b4-66c0-c34b-5a67-82fda2c709c1 [Follower]" leader= testLoggerAdapter verbose: server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: 2025-02-09T06:52:33.991Z [INFO] server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d1599f6-da3c-0851-04df-2cc5787ef026 Address:9d1599f6-da3c-0851-04df-2cc5787ef026} {Suffrage:Voter ID:server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 Address:407cc3b4-66c0-c34b-5a67-82fda2c709c1} {Suffrage:Voter ID:server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba Address:df9f2c72-97eb-acff-c8ad-75ebb50a00ba}]" testing.go:149: server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: 2025-02-09T06:52:33.991Z [INFO] server-407cc3b4-66c0-c34b-5a67-82fda2c709c1: entering follower state: follower="Node at 407cc3b4-66c0-c34b-5a67-82fda2c709c1 [Follower]" leader= testing.go:149: server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: 2025-02-09T06:52:33.991Z [INFO] server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d1599f6-da3c-0851-04df-2cc5787ef026 Address:9d1599f6-da3c-0851-04df-2cc5787ef026} {Suffrage:Voter ID:server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 Address:407cc3b4-66c0-c34b-5a67-82fda2c709c1} {Suffrage:Voter ID:server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba Address:df9f2c72-97eb-acff-c8ad-75ebb50a00ba}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:33.991Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:33.991Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: 2025-02-09T06:52:33.991Z [INFO] server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: entering follower state: follower="Node at df9f2c72-97eb-acff-c8ad-75ebb50a00ba [Follower]" leader= testing.go:149: server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: 2025-02-09T06:52:33.991Z [INFO] server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba: entering follower state: follower="Node at df9f2c72-97eb-acff-c8ad-75ebb50a00ba [Follower]" leader= testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [WARN] server-9d1599f6-da3c-0851-04df-2cc5787ef026: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [WARN] server-9d1599f6-da3c-0851-04df-2cc5787ef026: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: entering candidate state: node="Node at 9d1599f6-da3c-0851-04df-2cc5787ef026 [Candidate]" term=2 testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: entering candidate state: node="Node at 9d1599f6-da3c-0851-04df-2cc5787ef026 [Candidate]" term=2 testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: election won: tally=2 testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: election won: tally=2 testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: entering leader state: leader="Node at 9d1599f6-da3c-0851-04df-2cc5787ef026 [Leader]" testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.049Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: entering leader state: leader="Node at 9d1599f6-da3c-0851-04df-2cc5787ef026 [Leader]" testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: added peer, starting replication: peer=server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: added peer, starting replication: peer=server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: added peer, starting replication: peer=server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: added peer, starting replication: peer=server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: pipelining replication: peer="{Voter server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba df9f2c72-97eb-acff-c8ad-75ebb50a00ba}" testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: pipelining replication: peer="{Voter server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba df9f2c72-97eb-acff-c8ad-75ebb50a00ba}" testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: pipelining replication: peer="{Voter server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 407cc3b4-66c0-c34b-5a67-82fda2c709c1}" testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.050Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: pipelining replication: peer="{Voter server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 407cc3b4-66c0-c34b-5a67-82fda2c709c1}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.155Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.050034059 +0000 UTC m=+15.028842824 (1 nodes), 58.233422ms from start of poll, 58.621104ms from cluster start. Timeout at 2025-02-09 06:52:34.155589992 +0000 UTC m=+15.134398751, 105.555927ms after stability testing.go:149: cluster: 2025-02-09T06:52:34.155Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.050034059 +0000 UTC m=+15.028842824 (1 nodes), 58.233422ms from start of poll, 58.621104ms from cluster start. Timeout at 2025-02-09 06:52:34.155589992 +0000 UTC m=+15.134398751, 105.555927ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.161Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:34.161Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.266Z [INFO] stable state for Follower reached at 2025-02-09 06:52:34.161116239 +0000 UTC m=+15.139925004 (2 nodes), 773ns from start of poll, 169.703284ms from cluster start. Timeout at 2025-02-09 06:52:34.266946128 +0000 UTC m=+15.245754896, 105.829892ms after stability testing.go:149: cluster: 2025-02-09T06:52:34.266Z [INFO] stable state for Follower reached at 2025-02-09 06:52:34.161116239 +0000 UTC m=+15.139925004 (2 nodes), 773ns from start of poll, 169.703284ms from cluster start. Timeout at 2025-02-09 06:52:34.266946128 +0000 UTC m=+15.245754896, 105.829892ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 407cc3b4-66c0-c34b-5a67-82fda2c709c1 [Follower] testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.267Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: aborting pipeline replication: peer="{Voter server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba df9f2c72-97eb-acff-c8ad-75ebb50a00ba}" testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.267Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: aborting pipeline replication: peer="{Voter server-df9f2c72-97eb-acff-c8ad-75ebb50a00ba df9f2c72-97eb-acff-c8ad-75ebb50a00ba}" testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.267Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: aborting pipeline replication: peer="{Voter server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 407cc3b4-66c0-c34b-5a67-82fda2c709c1}" testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.267Z [INFO] server-9d1599f6-da3c-0851-04df-2cc5787ef026: aborting pipeline replication: peer="{Voter server-407cc3b4-66c0-c34b-5a67-82fda2c709c1 407cc3b4-66c0-c34b-5a67-82fda2c709c1}" testLoggerAdapter verbose: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.267Z [ERROR] server-9d1599f6-da3c-0851-04df-2cc5787ef026: failed to heartbeat to: peer=407cc3b4-66c0-c34b-5a67-82fda2c709c1 error="failed to connect to peer: 407cc3b4-66c0-c34b-5a67-82fda2c709c1" testing.go:149: server-9d1599f6-da3c-0851-04df-2cc5787ef026: 2025-02-09T06:52:34.267Z [ERROR] server-9d1599f6-da3c-0851-04df-2cc5787ef026: failed to heartbeat to: peer=407cc3b4-66c0-c34b-5a67-82fda2c709c1 error="failed to connect to peer: 407cc3b4-66c0-c34b-5a67-82fda2c709c1" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: 2025-02-09T06:52:34.267Z [INFO] server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0 Address:ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0}]" testing.go:149: server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: 2025-02-09T06:52:34.267Z [INFO] server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0 Address:ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0}]" testLoggerAdapter verbose: server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: 2025-02-09T06:52:34.267Z [INFO] server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: entering leader state: leader="Node at ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0 [Leader]" testing.go:149: server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: 2025-02-09T06:52:34.267Z [INFO] server-ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0: entering leader state: leader="Node at ec3f17ef-df76-8d61-fd8b-fe45d9bd7cd0 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.268Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e510a56-65ae-c993-f684-401f9345e797 Address:0e510a56-65ae-c993-f684-401f9345e797}]" testing.go:149: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.268Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e510a56-65ae-c993-f684-401f9345e797 Address:0e510a56-65ae-c993-f684-401f9345e797}]" testLoggerAdapter verbose: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.268Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: entering follower state: follower="Node at 0e510a56-65ae-c993-f684-401f9345e797 [Follower]" leader= testing.go:149: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.268Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: entering follower state: follower="Node at 0e510a56-65ae-c993-f684-401f9345e797 [Follower]" leader= testLoggerAdapter verbose: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [WARN] server-0e510a56-65ae-c993-f684-401f9345e797: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [WARN] server-0e510a56-65ae-c993-f684-401f9345e797: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: entering candidate state: node="Node at 0e510a56-65ae-c993-f684-401f9345e797 [Candidate]" term=2 testing.go:149: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: entering candidate state: node="Node at 0e510a56-65ae-c993-f684-401f9345e797 [Candidate]" term=2 testLoggerAdapter verbose: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: election won: tally=1 testing.go:149: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: election won: tally=1 testLoggerAdapter verbose: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: entering leader state: leader="Node at 0e510a56-65ae-c993-f684-401f9345e797 [Leader]" testing.go:149: server-0e510a56-65ae-c993-f684-401f9345e797: 2025-02-09T06:52:34.359Z [INFO] server-0e510a56-65ae-c993-f684-401f9345e797: entering leader state: leader="Node at 0e510a56-65ae-c993-f684-401f9345e797 [Leader]" --- PASS: TestRaft_NotifyCh (0.09s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: 2025-02-09T06:52:34.360Z [INFO] server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c Address:6d608e61-8c13-b9ca-8ce4-87cfcc4de59c} {Suffrage:Voter ID:server-495f2503-a9c7-727c-2520-1b572acf664c Address:495f2503-a9c7-727c-2520-1b572acf664c} {Suffrage:Voter ID:server-283496e2-c19e-0c9b-c344-d497adf8e685 Address:283496e2-c19e-0c9b-c344-d497adf8e685}]" testing.go:149: server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: 2025-02-09T06:52:34.360Z [INFO] server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c Address:6d608e61-8c13-b9ca-8ce4-87cfcc4de59c} {Suffrage:Voter ID:server-495f2503-a9c7-727c-2520-1b572acf664c Address:495f2503-a9c7-727c-2520-1b572acf664c} {Suffrage:Voter ID:server-283496e2-c19e-0c9b-c344-d497adf8e685 Address:283496e2-c19e-0c9b-c344-d497adf8e685}]" testLoggerAdapter verbose: server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: 2025-02-09T06:52:34.360Z [INFO] server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: entering follower state: follower="Node at 6d608e61-8c13-b9ca-8ce4-87cfcc4de59c [Follower]" leader= testing.go:149: server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: 2025-02-09T06:52:34.360Z [INFO] server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: entering follower state: follower="Node at 6d608e61-8c13-b9ca-8ce4-87cfcc4de59c [Follower]" leader= testLoggerAdapter verbose: server-495f2503-a9c7-727c-2520-1b572acf664c: 2025-02-09T06:52:34.360Z [INFO] server-495f2503-a9c7-727c-2520-1b572acf664c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c Address:6d608e61-8c13-b9ca-8ce4-87cfcc4de59c} {Suffrage:Voter ID:server-495f2503-a9c7-727c-2520-1b572acf664c Address:495f2503-a9c7-727c-2520-1b572acf664c} {Suffrage:Voter ID:server-283496e2-c19e-0c9b-c344-d497adf8e685 Address:283496e2-c19e-0c9b-c344-d497adf8e685}]" testing.go:149: server-495f2503-a9c7-727c-2520-1b572acf664c: 2025-02-09T06:52:34.360Z [INFO] server-495f2503-a9c7-727c-2520-1b572acf664c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c Address:6d608e61-8c13-b9ca-8ce4-87cfcc4de59c} {Suffrage:Voter ID:server-495f2503-a9c7-727c-2520-1b572acf664c Address:495f2503-a9c7-727c-2520-1b572acf664c} {Suffrage:Voter ID:server-283496e2-c19e-0c9b-c344-d497adf8e685 Address:283496e2-c19e-0c9b-c344-d497adf8e685}]" testLoggerAdapter verbose: server-495f2503-a9c7-727c-2520-1b572acf664c: 2025-02-09T06:52:34.360Z [INFO] server-495f2503-a9c7-727c-2520-1b572acf664c: entering follower state: follower="Node at 495f2503-a9c7-727c-2520-1b572acf664c [Follower]" leader= testing.go:149: server-495f2503-a9c7-727c-2520-1b572acf664c: 2025-02-09T06:52:34.360Z [INFO] server-495f2503-a9c7-727c-2520-1b572acf664c: entering follower state: follower="Node at 495f2503-a9c7-727c-2520-1b572acf664c [Follower]" leader= testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.360Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c Address:6d608e61-8c13-b9ca-8ce4-87cfcc4de59c} {Suffrage:Voter ID:server-495f2503-a9c7-727c-2520-1b572acf664c Address:495f2503-a9c7-727c-2520-1b572acf664c} {Suffrage:Voter ID:server-283496e2-c19e-0c9b-c344-d497adf8e685 Address:283496e2-c19e-0c9b-c344-d497adf8e685}]" testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.360Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c Address:6d608e61-8c13-b9ca-8ce4-87cfcc4de59c} {Suffrage:Voter ID:server-495f2503-a9c7-727c-2520-1b572acf664c Address:495f2503-a9c7-727c-2520-1b572acf664c} {Suffrage:Voter ID:server-283496e2-c19e-0c9b-c344-d497adf8e685 Address:283496e2-c19e-0c9b-c344-d497adf8e685}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.360Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:34.360Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.360Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: entering follower state: follower="Node at 283496e2-c19e-0c9b-c344-d497adf8e685 [Follower]" leader= testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.360Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: entering follower state: follower="Node at 283496e2-c19e-0c9b-c344-d497adf8e685 [Follower]" leader= testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [WARN] server-283496e2-c19e-0c9b-c344-d497adf8e685: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [WARN] server-283496e2-c19e-0c9b-c344-d497adf8e685: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: entering candidate state: node="Node at 283496e2-c19e-0c9b-c344-d497adf8e685 [Candidate]" term=2 testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: entering candidate state: node="Node at 283496e2-c19e-0c9b-c344-d497adf8e685 [Candidate]" term=2 testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: election won: tally=2 testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: election won: tally=2 testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: entering leader state: leader="Node at 283496e2-c19e-0c9b-c344-d497adf8e685 [Leader]" testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: entering leader state: leader="Node at 283496e2-c19e-0c9b-c344-d497adf8e685 [Leader]" testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: added peer, starting replication: peer=server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: added peer, starting replication: peer=server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: added peer, starting replication: peer=server-495f2503-a9c7-727c-2520-1b572acf664c testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: added peer, starting replication: peer=server-495f2503-a9c7-727c-2520-1b572acf664c testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: pipelining replication: peer="{Voter server-495f2503-a9c7-727c-2520-1b572acf664c 495f2503-a9c7-727c-2520-1b572acf664c}" testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: pipelining replication: peer="{Voter server-495f2503-a9c7-727c-2520-1b572acf664c 495f2503-a9c7-727c-2520-1b572acf664c}" testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: pipelining replication: peer="{Voter server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c 6d608e61-8c13-b9ca-8ce4-87cfcc4de59c}" testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.430Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: pipelining replication: peer="{Voter server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c 6d608e61-8c13-b9ca-8ce4-87cfcc4de59c}" testLoggerAdapter verbose: server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: 2025-02-09T06:52:34.430Z [WARN] server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: rejecting vote request since our last term is greater: candidate=283496e2-c19e-0c9b-c344-d497adf8e685 last-term=2 last-candidate-term=1 testing.go:149: server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: 2025-02-09T06:52:34.430Z [WARN] server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c: rejecting vote request since our last term is greater: candidate=283496e2-c19e-0c9b-c344-d497adf8e685 last-term=2 last-candidate-term=1 testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.536Z [INFO] stable state for Follower reached at 2025-02-09 06:52:34.430802537 +0000 UTC m=+15.409611300 (2 nodes), 69.934801ms from start of poll, 70.31543ms from cluster start. Timeout at 2025-02-09 06:52:34.536531212 +0000 UTC m=+15.515339980, 105.72868ms after stability testing.go:149: cluster: 2025-02-09T06:52:34.536Z [INFO] stable state for Follower reached at 2025-02-09 06:52:34.430802537 +0000 UTC m=+15.409611300 (2 nodes), 69.934801ms from start of poll, 70.31543ms from cluster start. Timeout at 2025-02-09 06:52:34.536531212 +0000 UTC m=+15.515339980, 105.72868ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.536Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:34.536Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.642Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.536606742 +0000 UTC m=+15.515415506 (1 nodes), 477ns from start of poll, 176.119636ms from cluster start. Timeout at 2025-02-09 06:52:34.64236387 +0000 UTC m=+15.621172643, 105.757137ms after stability testing.go:149: cluster: 2025-02-09T06:52:34.642Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.536606742 +0000 UTC m=+15.515415506 (1 nodes), 477ns from start of poll, 176.119636ms from cluster start. Timeout at 2025-02-09 06:52:34.64236387 +0000 UTC m=+15.621172643, 105.757137ms after stability testLoggerAdapter verbose: server-495f2503-a9c7-727c-2520-1b572acf664c: 2025-02-09T06:52:34.642Z [WARN] server-495f2503-a9c7-727c-2520-1b572acf664c: rejecting vote request since we have a leader: from=6d608e61-8c13-b9ca-8ce4-87cfcc4de59c leader=283496e2-c19e-0c9b-c344-d497adf8e685 testing.go:149: server-495f2503-a9c7-727c-2520-1b572acf664c: 2025-02-09T06:52:34.642Z [WARN] server-495f2503-a9c7-727c-2520-1b572acf664c: rejecting vote request since we have a leader: from=6d608e61-8c13-b9ca-8ce4-87cfcc4de59c leader=283496e2-c19e-0c9b-c344-d497adf8e685 testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.642Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: aborting pipeline replication: peer="{Voter server-495f2503-a9c7-727c-2520-1b572acf664c 495f2503-a9c7-727c-2520-1b572acf664c}" testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.642Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: aborting pipeline replication: peer="{Voter server-495f2503-a9c7-727c-2520-1b572acf664c 495f2503-a9c7-727c-2520-1b572acf664c}" testLoggerAdapter verbose: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.642Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: aborting pipeline replication: peer="{Voter server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c 6d608e61-8c13-b9ca-8ce4-87cfcc4de59c}" testing.go:149: server-283496e2-c19e-0c9b-c344-d497adf8e685: 2025-02-09T06:52:34.642Z [INFO] server-283496e2-c19e-0c9b-c344-d497adf8e685: aborting pipeline replication: peer="{Voter server-6d608e61-8c13-b9ca-8ce4-87cfcc4de59c 6d608e61-8c13-b9ca-8ce4-87cfcc4de59c}" --- PASS: TestRaft_Voting (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-61297dd4-04b3-b836-70c7-64ffffa840fb: 2025-02-09T06:52:34.643Z [INFO] server-61297dd4-04b3-b836-70c7-64ffffa840fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61297dd4-04b3-b836-70c7-64ffffa840fb Address:61297dd4-04b3-b836-70c7-64ffffa840fb} {Suffrage:Voter ID:server-47382fa7-c90c-cefa-ea0f-586190dfd445 Address:47382fa7-c90c-cefa-ea0f-586190dfd445} {Suffrage:Voter ID:server-35de255a-c7a2-94c4-8446-aeabccc7ae3f Address:35de255a-c7a2-94c4-8446-aeabccc7ae3f}]" testing.go:149: server-61297dd4-04b3-b836-70c7-64ffffa840fb: 2025-02-09T06:52:34.643Z [INFO] server-61297dd4-04b3-b836-70c7-64ffffa840fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61297dd4-04b3-b836-70c7-64ffffa840fb Address:61297dd4-04b3-b836-70c7-64ffffa840fb} {Suffrage:Voter ID:server-47382fa7-c90c-cefa-ea0f-586190dfd445 Address:47382fa7-c90c-cefa-ea0f-586190dfd445} {Suffrage:Voter ID:server-35de255a-c7a2-94c4-8446-aeabccc7ae3f Address:35de255a-c7a2-94c4-8446-aeabccc7ae3f}]" testLoggerAdapter verbose: server-61297dd4-04b3-b836-70c7-64ffffa840fb: 2025-02-09T06:52:34.643Z [INFO] server-61297dd4-04b3-b836-70c7-64ffffa840fb: entering follower state: follower="Node at 61297dd4-04b3-b836-70c7-64ffffa840fb [Follower]" leader= testing.go:149: server-61297dd4-04b3-b836-70c7-64ffffa840fb: 2025-02-09T06:52:34.643Z [INFO] server-61297dd4-04b3-b836-70c7-64ffffa840fb: entering follower state: follower="Node at 61297dd4-04b3-b836-70c7-64ffffa840fb [Follower]" leader= testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.643Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61297dd4-04b3-b836-70c7-64ffffa840fb Address:61297dd4-04b3-b836-70c7-64ffffa840fb} {Suffrage:Voter ID:server-47382fa7-c90c-cefa-ea0f-586190dfd445 Address:47382fa7-c90c-cefa-ea0f-586190dfd445} {Suffrage:Voter ID:server-35de255a-c7a2-94c4-8446-aeabccc7ae3f Address:35de255a-c7a2-94c4-8446-aeabccc7ae3f}]" testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.643Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61297dd4-04b3-b836-70c7-64ffffa840fb Address:61297dd4-04b3-b836-70c7-64ffffa840fb} {Suffrage:Voter ID:server-47382fa7-c90c-cefa-ea0f-586190dfd445 Address:47382fa7-c90c-cefa-ea0f-586190dfd445} {Suffrage:Voter ID:server-35de255a-c7a2-94c4-8446-aeabccc7ae3f Address:35de255a-c7a2-94c4-8446-aeabccc7ae3f}]" testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.643Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: entering follower state: follower="Node at 47382fa7-c90c-cefa-ea0f-586190dfd445 [Follower]" leader= testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.643Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: entering follower state: follower="Node at 47382fa7-c90c-cefa-ea0f-586190dfd445 [Follower]" leader= testLoggerAdapter verbose: server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: 2025-02-09T06:52:34.643Z [INFO] server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61297dd4-04b3-b836-70c7-64ffffa840fb Address:61297dd4-04b3-b836-70c7-64ffffa840fb} {Suffrage:Voter ID:server-47382fa7-c90c-cefa-ea0f-586190dfd445 Address:47382fa7-c90c-cefa-ea0f-586190dfd445} {Suffrage:Voter ID:server-35de255a-c7a2-94c4-8446-aeabccc7ae3f Address:35de255a-c7a2-94c4-8446-aeabccc7ae3f}]" testing.go:149: server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: 2025-02-09T06:52:34.643Z [INFO] server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61297dd4-04b3-b836-70c7-64ffffa840fb Address:61297dd4-04b3-b836-70c7-64ffffa840fb} {Suffrage:Voter ID:server-47382fa7-c90c-cefa-ea0f-586190dfd445 Address:47382fa7-c90c-cefa-ea0f-586190dfd445} {Suffrage:Voter ID:server-35de255a-c7a2-94c4-8446-aeabccc7ae3f Address:35de255a-c7a2-94c4-8446-aeabccc7ae3f}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.643Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:34.643Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: 2025-02-09T06:52:34.643Z [INFO] server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: entering follower state: follower="Node at 35de255a-c7a2-94c4-8446-aeabccc7ae3f [Follower]" leader= testing.go:149: server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: 2025-02-09T06:52:34.643Z [INFO] server-35de255a-c7a2-94c4-8446-aeabccc7ae3f: entering follower state: follower="Node at 35de255a-c7a2-94c4-8446-aeabccc7ae3f [Follower]" leader= testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [WARN] server-47382fa7-c90c-cefa-ea0f-586190dfd445: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [WARN] server-47382fa7-c90c-cefa-ea0f-586190dfd445: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: entering candidate state: node="Node at 47382fa7-c90c-cefa-ea0f-586190dfd445 [Candidate]" term=2 testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: entering candidate state: node="Node at 47382fa7-c90c-cefa-ea0f-586190dfd445 [Candidate]" term=2 testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: election won: tally=2 testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: election won: tally=2 testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: entering leader state: leader="Node at 47382fa7-c90c-cefa-ea0f-586190dfd445 [Leader]" testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: entering leader state: leader="Node at 47382fa7-c90c-cefa-ea0f-586190dfd445 [Leader]" testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: added peer, starting replication: peer=server-61297dd4-04b3-b836-70c7-64ffffa840fb testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: added peer, starting replication: peer=server-61297dd4-04b3-b836-70c7-64ffffa840fb testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: added peer, starting replication: peer=server-35de255a-c7a2-94c4-8446-aeabccc7ae3f testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: added peer, starting replication: peer=server-35de255a-c7a2-94c4-8446-aeabccc7ae3f testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: pipelining replication: peer="{Voter server-61297dd4-04b3-b836-70c7-64ffffa840fb 61297dd4-04b3-b836-70c7-64ffffa840fb}" testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: pipelining replication: peer="{Voter server-61297dd4-04b3-b836-70c7-64ffffa840fb 61297dd4-04b3-b836-70c7-64ffffa840fb}" testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: pipelining replication: peer="{Voter server-35de255a-c7a2-94c4-8446-aeabccc7ae3f 35de255a-c7a2-94c4-8446-aeabccc7ae3f}" testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.696Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: pipelining replication: peer="{Voter server-35de255a-c7a2-94c4-8446-aeabccc7ae3f 35de255a-c7a2-94c4-8446-aeabccc7ae3f}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.802Z [INFO] stable state for Follower reached at 2025-02-09 06:52:34.696560386 +0000 UTC m=+15.675369161 (2 nodes), 52.910329ms from start of poll, 53.264005ms from cluster start. Timeout at 2025-02-09 06:52:34.802054063 +0000 UTC m=+15.780862835, 105.493674ms after stability testing.go:149: cluster: 2025-02-09T06:52:34.802Z [INFO] stable state for Follower reached at 2025-02-09 06:52:34.696560386 +0000 UTC m=+15.675369161 (2 nodes), 52.910329ms from start of poll, 53.264005ms from cluster start. Timeout at 2025-02-09 06:52:34.802054063 +0000 UTC m=+15.780862835, 105.493674ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.802Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:34.802Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.907Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.802115152 +0000 UTC m=+15.780923920 (1 nodes), 447ns from start of poll, 158.818764ms from cluster start. Timeout at 2025-02-09 06:52:34.907778965 +0000 UTC m=+15.886587725, 105.663805ms after stability testing.go:149: cluster: 2025-02-09T06:52:34.907Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.802115152 +0000 UTC m=+15.780923920 (1 nodes), 447ns from start of poll, 158.818764ms from cluster start. Timeout at 2025-02-09 06:52:34.907778965 +0000 UTC m=+15.886587725, 105.663805ms after stability testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.907Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: aborting pipeline replication: peer="{Voter server-35de255a-c7a2-94c4-8446-aeabccc7ae3f 35de255a-c7a2-94c4-8446-aeabccc7ae3f}" testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.907Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: aborting pipeline replication: peer="{Voter server-35de255a-c7a2-94c4-8446-aeabccc7ae3f 35de255a-c7a2-94c4-8446-aeabccc7ae3f}" testLoggerAdapter verbose: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.907Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: aborting pipeline replication: peer="{Voter server-61297dd4-04b3-b836-70c7-64ffffa840fb 61297dd4-04b3-b836-70c7-64ffffa840fb}" testing.go:149: server-47382fa7-c90c-cefa-ea0f-586190dfd445: 2025-02-09T06:52:34.907Z [INFO] server-47382fa7-c90c-cefa-ea0f-586190dfd445: aborting pipeline replication: peer="{Voter server-61297dd4-04b3-b836-70c7-64ffffa840fb 61297dd4-04b3-b836-70c7-64ffffa840fb}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.908Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba}]" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.908Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba}]" testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.908Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: entering follower state: follower="Node at e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d [Follower]" leader= testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.908Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: entering follower state: follower="Node at e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d [Follower]" leader= testLoggerAdapter verbose: d0462cea-3ad4-806d-b04c-dd014ba238ba: 2025-02-09T06:52:34.908Z [INFO] d0462cea-3ad4-806d-b04c-dd014ba238ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba}]" testing.go:149: d0462cea-3ad4-806d-b04c-dd014ba238ba: 2025-02-09T06:52:34.908Z [INFO] d0462cea-3ad4-806d-b04c-dd014ba238ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba}]" testLoggerAdapter verbose: d0462cea-3ad4-806d-b04c-dd014ba238ba: 2025-02-09T06:52:34.909Z [INFO] d0462cea-3ad4-806d-b04c-dd014ba238ba: entering follower state: follower="Node at d0462cea-3ad4-806d-b04c-dd014ba238ba [Follower]" leader= testing.go:149: d0462cea-3ad4-806d-b04c-dd014ba238ba: 2025-02-09T06:52:34.909Z [INFO] d0462cea-3ad4-806d-b04c-dd014ba238ba: entering follower state: follower="Node at d0462cea-3ad4-806d-b04c-dd014ba238ba [Follower]" leader= testLoggerAdapter verbose: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:34.909Z [INFO] 339b6473-3f55-1511-ad33-e1b79d113b2e: initial configuration: index=0 servers=[] testing.go:149: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:34.909Z [INFO] 339b6473-3f55-1511-ad33-e1b79d113b2e: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T06:52:34.909Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:34.909Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:34.909Z [INFO] 339b6473-3f55-1511-ad33-e1b79d113b2e: entering follower state: follower="Node at 339b6473-3f55-1511-ad33-e1b79d113b2e [Follower]" leader= testing.go:149: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:34.909Z [INFO] 339b6473-3f55-1511-ad33-e1b79d113b2e: entering follower state: follower="Node at 339b6473-3f55-1511-ad33-e1b79d113b2e [Follower]" leader= testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [WARN] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: heartbeat timeout reached, starting election: last-leader= testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [WARN] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: entering candidate state: node="Node at e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d [Candidate]" term=2 testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: entering candidate state: node="Node at e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d [Candidate]" term=2 testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: election won: tally=2 testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: election won: tally=2 testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: entering leader state: leader="Node at e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d [Leader]" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: entering leader state: leader="Node at e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d [Leader]" testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: added peer, starting replication: peer=d0462cea-3ad4-806d-b04c-dd014ba238ba testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: added peer, starting replication: peer=d0462cea-3ad4-806d-b04c-dd014ba238ba testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: pipelining replication: peer="{Voter d0462cea-3ad4-806d-b04c-dd014ba238ba d0462cea-3ad4-806d-b04c-dd014ba238ba}" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:34.962Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: pipelining replication: peer="{Voter d0462cea-3ad4-806d-b04c-dd014ba238ba d0462cea-3ad4-806d-b04c-dd014ba238ba}" testLoggerAdapter verbose: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:34.970Z [WARN] 339b6473-3f55-1511-ad33-e1b79d113b2e: no known peers, aborting election testing.go:149: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:34.970Z [WARN] 339b6473-3f55-1511-ad33-e1b79d113b2e: no known peers, aborting election testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.067Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.962565532 +0000 UTC m=+15.941374307 (1 nodes), 53.250734ms from start of poll, 53.791882ms from cluster start. Timeout at 2025-02-09 06:52:35.067881532 +0000 UTC m=+16.046690290, 105.315983ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.067Z [INFO] stable state for Leader reached at 2025-02-09 06:52:34.962565532 +0000 UTC m=+15.941374307 (1 nodes), 53.250734ms from start of poll, 53.791882ms from cluster start. Timeout at 2025-02-09 06:52:35.067881532 +0000 UTC m=+16.046690290, 105.315983ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.067Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:35.067Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.173Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.067943708 +0000 UTC m=+16.046752474 (1 nodes), 488ns from start of poll, 159.170049ms from cluster start. Timeout at 2025-02-09 06:52:35.173468354 +0000 UTC m=+16.152277108, 105.524634ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.173Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.067943708 +0000 UTC m=+16.046752474 (1 nodes), 488ns from start of poll, 159.170049ms from cluster start. Timeout at 2025-02-09 06:52:35.173468354 +0000 UTC m=+16.152277108, 105.524634ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.173Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:35.173Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.279Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.173540947 +0000 UTC m=+16.152349712 (1 nodes), 458ns from start of poll, 264.767287ms from cluster start. Timeout at 2025-02-09 06:52:35.279053353 +0000 UTC m=+16.257862121, 105.512409ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.279Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.173540947 +0000 UTC m=+16.152349712 (1 nodes), 458ns from start of poll, 264.767287ms from cluster start. Timeout at 2025-02-09 06:52:35.279053353 +0000 UTC m=+16.257862121, 105.512409ms after stability testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: updating configuration: command=AddStaging server-id=339b6473-3f55-1511-ad33-e1b79d113b2e server-addr=339b6473-3f55-1511-ad33-e1b79d113b2e servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba} {Suffrage:Voter ID:339b6473-3f55-1511-ad33-e1b79d113b2e Address:339b6473-3f55-1511-ad33-e1b79d113b2e}]" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: updating configuration: command=AddStaging server-id=339b6473-3f55-1511-ad33-e1b79d113b2e server-addr=339b6473-3f55-1511-ad33-e1b79d113b2e servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba} {Suffrage:Voter ID:339b6473-3f55-1511-ad33-e1b79d113b2e Address:339b6473-3f55-1511-ad33-e1b79d113b2e}]" testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: added peer, starting replication: peer=339b6473-3f55-1511-ad33-e1b79d113b2e testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: added peer, starting replication: peer=339b6473-3f55-1511-ad33-e1b79d113b2e testLoggerAdapter verbose: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:35.279Z [WARN] 339b6473-3f55-1511-ad33-e1b79d113b2e: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: 339b6473-3f55-1511-ad33-e1b79d113b2e: 2025-02-09T06:52:35.279Z [WARN] 339b6473-3f55-1511-ad33-e1b79d113b2e: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [WARN] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: appendEntries rejected, sending older logs: peer="{Voter 339b6473-3f55-1511-ad33-e1b79d113b2e 339b6473-3f55-1511-ad33-e1b79d113b2e}" next=1 testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [WARN] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: appendEntries rejected, sending older logs: peer="{Voter 339b6473-3f55-1511-ad33-e1b79d113b2e 339b6473-3f55-1511-ad33-e1b79d113b2e}" next=1 testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: pipelining replication: peer="{Voter 339b6473-3f55-1511-ad33-e1b79d113b2e 339b6473-3f55-1511-ad33-e1b79d113b2e}" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.279Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: pipelining replication: peer="{Voter 339b6473-3f55-1511-ad33-e1b79d113b2e 339b6473-3f55-1511-ad33-e1b79d113b2e}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.279Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:35.279Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.385Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.279407196 +0000 UTC m=+16.258215960 (1 nodes), 504ns from start of poll, 370.633535ms from cluster start. Timeout at 2025-02-09 06:52:35.385082758 +0000 UTC m=+16.363891517, 105.675557ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.385Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.279407196 +0000 UTC m=+16.258215960 (1 nodes), 504ns from start of poll, 370.633535ms from cluster start. Timeout at 2025-02-09 06:52:35.385082758 +0000 UTC m=+16.363891517, 105.675557ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.385Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:35.385Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.490Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.385181518 +0000 UTC m=+16.363990283 (1 nodes), 466ns from start of poll, 476.407858ms from cluster start. Timeout at 2025-02-09 06:52:35.490796279 +0000 UTC m=+16.469605048, 105.614765ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.490Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.385181518 +0000 UTC m=+16.363990283 (1 nodes), 466ns from start of poll, 476.407858ms from cluster start. Timeout at 2025-02-09 06:52:35.490796279 +0000 UTC m=+16.469605048, 105.614765ms after stability testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.490Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: updating configuration: command=RemoveServer server-id=339b6473-3f55-1511-ad33-e1b79d113b2e server-addr= servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba}]" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.490Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: updating configuration: command=RemoveServer server-id=339b6473-3f55-1511-ad33-e1b79d113b2e server-addr= servers="[{Suffrage:Voter ID:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d Address:e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d} {Suffrage:Voter ID:d0462cea-3ad4-806d-b04c-dd014ba238ba Address:d0462cea-3ad4-806d-b04c-dd014ba238ba}]" testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.490Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: removed peer, stopping replication: peer=339b6473-3f55-1511-ad33-e1b79d113b2e last-index=4 testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.490Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: removed peer, stopping replication: peer=339b6473-3f55-1511-ad33-e1b79d113b2e last-index=4 testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.491Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: aborting pipeline replication: peer="{Voter 339b6473-3f55-1511-ad33-e1b79d113b2e 339b6473-3f55-1511-ad33-e1b79d113b2e}" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.491Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: aborting pipeline replication: peer="{Voter 339b6473-3f55-1511-ad33-e1b79d113b2e 339b6473-3f55-1511-ad33-e1b79d113b2e}" testLoggerAdapter verbose: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.491Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: aborting pipeline replication: peer="{Voter d0462cea-3ad4-806d-b04c-dd014ba238ba d0462cea-3ad4-806d-b04c-dd014ba238ba}" testing.go:149: e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: 2025-02-09T06:52:35.491Z [INFO] e4741432-ae0e-b56f-7bb9-6d8b50e4cc7d: aborting pipeline replication: peer="{Voter d0462cea-3ad4-806d-b04c-dd014ba238ba d0462cea-3ad4-806d-b04c-dd014ba238ba}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.58s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: 2025-02-09T06:52:35.491Z [INFO] 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: initial configuration: index=1 servers="[{Suffrage:Voter ID:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d Address:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d} {Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1}]" testing.go:149: 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: 2025-02-09T06:52:35.491Z [INFO] 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: initial configuration: index=1 servers="[{Suffrage:Voter ID:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d Address:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d} {Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1}]" testLoggerAdapter verbose: 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: 2025-02-09T06:52:35.492Z [INFO] 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: entering follower state: follower="Node at 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d [Follower]" leader= testing.go:149: 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: 2025-02-09T06:52:35.492Z [INFO] 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d: entering follower state: follower="Node at 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d [Follower]" leader= testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.492Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d Address:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d} {Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1}]" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.492Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d Address:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d} {Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.493Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:35.493Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.493Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: entering follower state: follower="Node at 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 [Follower]" leader= testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.493Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: entering follower state: follower="Node at 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 [Follower]" leader= testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [WARN] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: heartbeat timeout reached, starting election: last-leader= testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [WARN] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: entering candidate state: node="Node at 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 [Candidate]" term=2 testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: entering candidate state: node="Node at 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 [Candidate]" term=2 testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: election won: tally=2 testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: election won: tally=2 testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: entering leader state: leader="Node at 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 [Leader]" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: entering leader state: leader="Node at 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 [Leader]" testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: added peer, starting replication: peer=0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.547Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: added peer, starting replication: peer=0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.548Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: pipelining replication: peer="{Voter 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d}" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.548Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: pipelining replication: peer="{Voter 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.653Z [INFO] stable state for Follower reached at 2025-02-09 06:52:35.547984666 +0000 UTC m=+16.526793452 (1 nodes), 54.84146ms from start of poll, 56.070103ms from cluster start. Timeout at 2025-02-09 06:52:35.653052271 +0000 UTC m=+16.631861041, 105.067589ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.653Z [INFO] stable state for Follower reached at 2025-02-09 06:52:35.547984666 +0000 UTC m=+16.526793452 (1 nodes), 54.84146ms from start of poll, 56.070103ms from cluster start. Timeout at 2025-02-09 06:52:35.653052271 +0000 UTC m=+16.631861041, 105.067589ms after stability testLoggerAdapter verbose: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.653Z [INFO] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: initial configuration: index=0 servers=[] testing.go:149: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.653Z [INFO] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.653Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.653Z [INFO] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: entering follower state: follower="Node at bdfe75b5-1662-6cf4-9646-fa7a9f725c92 [Follower]" leader= testing.go:149: cluster: 2025-02-09T06:52:35.653Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.653Z [INFO] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: entering follower state: follower="Node at bdfe75b5-1662-6cf4-9646-fa7a9f725c92 [Follower]" leader= testLoggerAdapter verbose: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.708Z [WARN] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: no known peers, aborting election testing.go:149: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.708Z [WARN] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: no known peers, aborting election testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.758Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.65347388 +0000 UTC m=+16.632282681 (1 nodes), 951ns from start of poll, 161.559332ms from cluster start. Timeout at 2025-02-09 06:52:35.758820084 +0000 UTC m=+16.737628844, 105.346163ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.758Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.65347388 +0000 UTC m=+16.632282681 (1 nodes), 951ns from start of poll, 161.559332ms from cluster start. Timeout at 2025-02-09 06:52:35.758820084 +0000 UTC m=+16.737628844, 105.346163ms after stability testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.758Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: updating configuration: command=AddStaging server-id=server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 server-addr=bdfe75b5-1662-6cf4-9646-fa7a9f725c92 servers="[{Suffrage:Voter ID:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d Address:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d} {Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1} {Suffrage:Voter ID:server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 Address:bdfe75b5-1662-6cf4-9646-fa7a9f725c92}]" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.758Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: updating configuration: command=AddStaging server-id=server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 server-addr=bdfe75b5-1662-6cf4-9646-fa7a9f725c92 servers="[{Suffrage:Voter ID:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d Address:0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d} {Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1} {Suffrage:Voter ID:server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 Address:bdfe75b5-1662-6cf4-9646-fa7a9f725c92}]" testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.758Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: added peer, starting replication: peer=server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.758Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: added peer, starting replication: peer=server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 testLoggerAdapter verbose: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.759Z [WARN] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: 2025-02-09T06:52:35.759Z [WARN] server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.759Z [WARN] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: appendEntries rejected, sending older logs: peer="{Voter server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 bdfe75b5-1662-6cf4-9646-fa7a9f725c92}" next=1 testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.759Z [WARN] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: appendEntries rejected, sending older logs: peer="{Voter server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 bdfe75b5-1662-6cf4-9646-fa7a9f725c92}" next=1 testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.759Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: pipelining replication: peer="{Voter server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 bdfe75b5-1662-6cf4-9646-fa7a9f725c92}" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.759Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: pipelining replication: peer="{Voter server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 bdfe75b5-1662-6cf4-9646-fa7a9f725c92}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.764Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:35.764Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.869Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.764172007 +0000 UTC m=+16.742980774 (1 nodes), 509ns from start of poll, 272.257425ms from cluster start. Timeout at 2025-02-09 06:52:35.869249677 +0000 UTC m=+16.848058435, 105.077661ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.869Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.764172007 +0000 UTC m=+16.742980774 (1 nodes), 509ns from start of poll, 272.257425ms from cluster start. Timeout at 2025-02-09 06:52:35.869249677 +0000 UTC m=+16.848058435, 105.077661ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.869Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:35.869Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.974Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.869317261 +0000 UTC m=+16.848126026 (1 nodes), 463ns from start of poll, 377.402677ms from cluster start. Timeout at 2025-02-09 06:52:35.974319964 +0000 UTC m=+16.953128728, 105.002702ms after stability testing.go:149: cluster: 2025-02-09T06:52:35.974Z [INFO] stable state for Leader reached at 2025-02-09 06:52:35.869317261 +0000 UTC m=+16.848126026 (1 nodes), 463ns from start of poll, 377.402677ms from cluster start. Timeout at 2025-02-09 06:52:35.974319964 +0000 UTC m=+16.953128728, 105.002702ms after stability testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: updating configuration: command=RemoveServer server-id=0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d server-addr= servers="[{Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1} {Suffrage:Voter ID:server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 Address:bdfe75b5-1662-6cf4-9646-fa7a9f725c92}]" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: updating configuration: command=RemoveServer server-id=0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d server-addr= servers="[{Suffrage:Voter ID:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1 Address:2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1} {Suffrage:Voter ID:server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 Address:bdfe75b5-1662-6cf4-9646-fa7a9f725c92}]" testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: removed peer, stopping replication: peer=0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d last-index=4 testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: removed peer, stopping replication: peer=0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d last-index=4 testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: aborting pipeline replication: peer="{Voter 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d}" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: aborting pipeline replication: peer="{Voter 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d 0b7d4ad4-ad8d-76a5-bf5d-c4d610f9b45d}" testLoggerAdapter verbose: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: aborting pipeline replication: peer="{Voter server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 bdfe75b5-1662-6cf4-9646-fa7a9f725c92}" testing.go:149: 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: 2025-02-09T06:52:35.974Z [INFO] 2a5e0774-93ab-3cc3-b4a3-b5f8da3f4cd1: aborting pipeline replication: peer="{Voter server-bdfe75b5-1662-6cf4-9646-fa7a9f725c92 bdfe75b5-1662-6cf4-9646-fa7a9f725c92}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.48s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:35.975Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 Address:2903c9a1-c302-a0a9-75da-aba72d8c7ea2} {Suffrage:Voter ID:server-742753e4-5df7-c5ee-04d2-b8e401f8c46f Address:742753e4-5df7-c5ee-04d2-b8e401f8c46f} {Suffrage:Voter ID:server-7abad551-1625-795a-768a-2502ef371445 Address:7abad551-1625-795a-768a-2502ef371445}]" testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:35.975Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 Address:2903c9a1-c302-a0a9-75da-aba72d8c7ea2} {Suffrage:Voter ID:server-742753e4-5df7-c5ee-04d2-b8e401f8c46f Address:742753e4-5df7-c5ee-04d2-b8e401f8c46f} {Suffrage:Voter ID:server-7abad551-1625-795a-768a-2502ef371445 Address:7abad551-1625-795a-768a-2502ef371445}]" testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:35.975Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: entering follower state: follower="Node at 2903c9a1-c302-a0a9-75da-aba72d8c7ea2 [Follower]" leader= testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:35.975Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: entering follower state: follower="Node at 2903c9a1-c302-a0a9-75da-aba72d8c7ea2 [Follower]" leader= testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:35.975Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 Address:2903c9a1-c302-a0a9-75da-aba72d8c7ea2} {Suffrage:Voter ID:server-742753e4-5df7-c5ee-04d2-b8e401f8c46f Address:742753e4-5df7-c5ee-04d2-b8e401f8c46f} {Suffrage:Voter ID:server-7abad551-1625-795a-768a-2502ef371445 Address:7abad551-1625-795a-768a-2502ef371445}]" testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:35.975Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 Address:2903c9a1-c302-a0a9-75da-aba72d8c7ea2} {Suffrage:Voter ID:server-742753e4-5df7-c5ee-04d2-b8e401f8c46f Address:742753e4-5df7-c5ee-04d2-b8e401f8c46f} {Suffrage:Voter ID:server-7abad551-1625-795a-768a-2502ef371445 Address:7abad551-1625-795a-768a-2502ef371445}]" testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:35.975Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering follower state: follower="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Follower]" leader= testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:35.975Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering follower state: follower="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Follower]" leader= testLoggerAdapter verbose: server-7abad551-1625-795a-768a-2502ef371445: 2025-02-09T06:52:35.975Z [INFO] server-7abad551-1625-795a-768a-2502ef371445: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 Address:2903c9a1-c302-a0a9-75da-aba72d8c7ea2} {Suffrage:Voter ID:server-742753e4-5df7-c5ee-04d2-b8e401f8c46f Address:742753e4-5df7-c5ee-04d2-b8e401f8c46f} {Suffrage:Voter ID:server-7abad551-1625-795a-768a-2502ef371445 Address:7abad551-1625-795a-768a-2502ef371445}]" testing.go:149: server-7abad551-1625-795a-768a-2502ef371445: 2025-02-09T06:52:35.975Z [INFO] server-7abad551-1625-795a-768a-2502ef371445: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 Address:2903c9a1-c302-a0a9-75da-aba72d8c7ea2} {Suffrage:Voter ID:server-742753e4-5df7-c5ee-04d2-b8e401f8c46f Address:742753e4-5df7-c5ee-04d2-b8e401f8c46f} {Suffrage:Voter ID:server-7abad551-1625-795a-768a-2502ef371445 Address:7abad551-1625-795a-768a-2502ef371445}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:35.975Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:35.975Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7abad551-1625-795a-768a-2502ef371445: 2025-02-09T06:52:35.975Z [INFO] server-7abad551-1625-795a-768a-2502ef371445: entering follower state: follower="Node at 7abad551-1625-795a-768a-2502ef371445 [Follower]" leader= testing.go:149: server-7abad551-1625-795a-768a-2502ef371445: 2025-02-09T06:52:35.975Z [INFO] server-7abad551-1625-795a-768a-2502ef371445: entering follower state: follower="Node at 7abad551-1625-795a-768a-2502ef371445 [Follower]" leader= testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [WARN] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [WARN] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering candidate state: node="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Candidate]" term=2 testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering candidate state: node="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Candidate]" term=2 testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: election won: tally=2 testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: election won: tally=2 testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering leader state: leader="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Leader]" testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering leader state: leader="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Leader]" testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: added peer, starting replication: peer=server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: added peer, starting replication: peer=server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: added peer, starting replication: peer=server-7abad551-1625-795a-768a-2502ef371445 testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: added peer, starting replication: peer=server-7abad551-1625-795a-768a-2502ef371445 testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: pipelining replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: pipelining replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: pipelining replication: peer="{Voter server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 2903c9a1-c302-a0a9-75da-aba72d8c7ea2}" testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.040Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: pipelining replication: peer="{Voter server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 2903c9a1-c302-a0a9-75da-aba72d8c7ea2}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.146Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.040531437 +0000 UTC m=+17.019340202 (1 nodes), 64.752728ms from start of poll, 65.082035ms from cluster start. Timeout at 2025-02-09 06:52:36.146242084 +0000 UTC m=+17.125050843, 105.710641ms after stability testing.go:149: cluster: 2025-02-09T06:52:36.146Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.040531437 +0000 UTC m=+17.019340202 (1 nodes), 64.752728ms from start of poll, 65.082035ms from cluster start. Timeout at 2025-02-09 06:52:36.146242084 +0000 UTC m=+17.125050843, 105.710641ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.146Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:36.146Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.251Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.146308148 +0000 UTC m=+17.125116913 (1 nodes), 481ns from start of poll, 170.858746ms from cluster start. Timeout at 2025-02-09 06:52:36.251499544 +0000 UTC m=+17.230308313, 105.1914ms after stability testing.go:149: cluster: 2025-02-09T06:52:36.251Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.146308148 +0000 UTC m=+17.125116913 (1 nodes), 481ns from start of poll, 170.858746ms from cluster start. Timeout at 2025-02-09 06:52:36.251499544 +0000 UTC m=+17.230308313, 105.1914ms after stability testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: entering candidate state: node="Node at 2903c9a1-c302-a0a9-75da-aba72d8c7ea2 [Candidate]" term=3 testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: entering candidate state: node="Node at 2903c9a1-c302-a0a9-75da-aba72d8c7ea2 [Candidate]" term=3 testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: election won: tally=2 testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: election won: tally=2 testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: entering leader state: leader="Node at 2903c9a1-c302-a0a9-75da-aba72d8c7ea2 [Leader]" testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: entering leader state: leader="Node at 2903c9a1-c302-a0a9-75da-aba72d8c7ea2 [Leader]" testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: added peer, starting replication: peer=server-742753e4-5df7-c5ee-04d2-b8e401f8c46f testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: added peer, starting replication: peer=server-742753e4-5df7-c5ee-04d2-b8e401f8c46f testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: added peer, starting replication: peer=server-7abad551-1625-795a-768a-2502ef371445 testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: added peer, starting replication: peer=server-7abad551-1625-795a-768a-2502ef371445 testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.251Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering follower state: follower="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Follower]" leader= testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.251Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.251Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: entering follower state: follower="Node at 742753e4-5df7-c5ee-04d2-b8e401f8c46f [Follower]" leader= testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: pipelining replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" testing.go:149: cluster: 2025-02-09T06:52:36.251Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.251Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: aborting pipeline replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: pipelining replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.251Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: aborting pipeline replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: pipelining replication: peer="{Voter server-742753e4-5df7-c5ee-04d2-b8e401f8c46f 742753e4-5df7-c5ee-04d2-b8e401f8c46f}" testLoggerAdapter verbose: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.251Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: aborting pipeline replication: peer="{Voter server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 2903c9a1-c302-a0a9-75da-aba72d8c7ea2}" testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.251Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: pipelining replication: peer="{Voter server-742753e4-5df7-c5ee-04d2-b8e401f8c46f 742753e4-5df7-c5ee-04d2-b8e401f8c46f}" testing.go:149: server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: 2025-02-09T06:52:36.251Z [INFO] server-742753e4-5df7-c5ee-04d2-b8e401f8c46f: aborting pipeline replication: peer="{Voter server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2 2903c9a1-c302-a0a9-75da-aba72d8c7ea2}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.357Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.251784547 +0000 UTC m=+17.230593320 (1 nodes), 11.096µs from start of poll, 276.335153ms from cluster start. Timeout at 2025-02-09 06:52:36.357094604 +0000 UTC m=+17.335903372, 105.310052ms after stability testing.go:149: cluster: 2025-02-09T06:52:36.357Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.251784547 +0000 UTC m=+17.230593320 (1 nodes), 11.096µs from start of poll, 276.335153ms from cluster start. Timeout at 2025-02-09 06:52:36.357094604 +0000 UTC m=+17.335903372, 105.310052ms after stability testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.357Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: aborting pipeline replication: peer="{Voter server-742753e4-5df7-c5ee-04d2-b8e401f8c46f 742753e4-5df7-c5ee-04d2-b8e401f8c46f}" testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.357Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: aborting pipeline replication: peer="{Voter server-742753e4-5df7-c5ee-04d2-b8e401f8c46f 742753e4-5df7-c5ee-04d2-b8e401f8c46f}" testLoggerAdapter verbose: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.357Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: aborting pipeline replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" testing.go:149: server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: 2025-02-09T06:52:36.357Z [INFO] server-2903c9a1-c302-a0a9-75da-aba72d8c7ea2: aborting pipeline replication: peer="{Voter server-7abad551-1625-795a-768a-2502ef371445 7abad551-1625-795a-768a-2502ef371445}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.357Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e Address:eb11ecf7-adbe-fb0e-7d9e-a51a1433868e}]" testing.go:149: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.357Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e Address:eb11ecf7-adbe-fb0e-7d9e-a51a1433868e}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.357Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:36.357Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.358Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: entering follower state: follower="Node at eb11ecf7-adbe-fb0e-7d9e-a51a1433868e [Follower]" leader= testing.go:149: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.358Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: entering follower state: follower="Node at eb11ecf7-adbe-fb0e-7d9e-a51a1433868e [Follower]" leader= testLoggerAdapter verbose: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [WARN] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [WARN] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: entering candidate state: node="Node at eb11ecf7-adbe-fb0e-7d9e-a51a1433868e [Candidate]" term=2 testing.go:149: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: entering candidate state: node="Node at eb11ecf7-adbe-fb0e-7d9e-a51a1433868e [Candidate]" term=2 testLoggerAdapter verbose: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: election won: tally=1 testing.go:149: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: election won: tally=1 testLoggerAdapter verbose: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: entering leader state: leader="Node at eb11ecf7-adbe-fb0e-7d9e-a51a1433868e [Leader]" testing.go:149: server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: 2025-02-09T06:52:36.412Z [INFO] server-eb11ecf7-adbe-fb0e-7d9e-a51a1433868e: entering leader state: leader="Node at eb11ecf7-adbe-fb0e-7d9e-a51a1433868e [Leader]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.518Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.412989292 +0000 UTC m=+17.391798057 (1 nodes), 54.976152ms from start of poll, 55.140166ms from cluster start. Timeout at 2025-02-09 06:52:36.518446607 +0000 UTC m=+17.497255379, 105.457322ms after stability testing.go:149: cluster: 2025-02-09T06:52:36.518Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.412989292 +0000 UTC m=+17.391798057 (1 nodes), 54.976152ms from start of poll, 55.140166ms from cluster start. Timeout at 2025-02-09 06:52:36.518446607 +0000 UTC m=+17.497255379, 105.457322ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.16s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.519Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.519Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.519Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: entering follower state: follower="Node at 9eb65806-9efa-c938-bc7e-a31eed882685 [Follower]" leader= testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.519Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: entering follower state: follower="Node at 9eb65806-9efa-c938-bc7e-a31eed882685 [Follower]" leader= testLoggerAdapter verbose: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.519Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testing.go:149: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.519Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testLoggerAdapter verbose: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.519Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: entering follower state: follower="Node at b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd [Follower]" leader= testing.go:149: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.519Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: entering follower state: follower="Node at b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd [Follower]" leader= testLoggerAdapter verbose: server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: 2025-02-09T06:52:36.519Z [INFO] server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testing.go:149: server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: 2025-02-09T06:52:36.519Z [INFO] server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testLoggerAdapter verbose: server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: 2025-02-09T06:52:36.519Z [INFO] server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: entering follower state: follower="Node at 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 [Follower]" leader= testing.go:149: server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: 2025-02-09T06:52:36.519Z [INFO] server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: entering follower state: follower="Node at 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 [Follower]" leader= testLoggerAdapter verbose: server-31718b08-787f-39a7-143e-bd02cb427cd2: 2025-02-09T06:52:36.520Z [INFO] server-31718b08-787f-39a7-143e-bd02cb427cd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testing.go:149: server-31718b08-787f-39a7-143e-bd02cb427cd2: 2025-02-09T06:52:36.520Z [INFO] server-31718b08-787f-39a7-143e-bd02cb427cd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testLoggerAdapter verbose: server-31718b08-787f-39a7-143e-bd02cb427cd2: 2025-02-09T06:52:36.520Z [INFO] server-31718b08-787f-39a7-143e-bd02cb427cd2: entering follower state: follower="Node at 31718b08-787f-39a7-143e-bd02cb427cd2 [Follower]" leader= testing.go:149: server-31718b08-787f-39a7-143e-bd02cb427cd2: 2025-02-09T06:52:36.520Z [INFO] server-31718b08-787f-39a7-143e-bd02cb427cd2: entering follower state: follower="Node at 31718b08-787f-39a7-143e-bd02cb427cd2 [Follower]" leader= testLoggerAdapter verbose: server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: 2025-02-09T06:52:36.520Z [INFO] server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testing.go:149: server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: 2025-02-09T06:52:36.520Z [INFO] server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testLoggerAdapter verbose: server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: 2025-02-09T06:52:36.520Z [INFO] server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: entering follower state: follower="Node at 8a069cab-9494-f9fa-96fb-2f80e9ef85ef [Follower]" leader= testing.go:149: server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: 2025-02-09T06:52:36.520Z [INFO] server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: entering follower state: follower="Node at 8a069cab-9494-f9fa-96fb-2f80e9ef85ef [Follower]" leader= testLoggerAdapter verbose: server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: 2025-02-09T06:52:36.520Z [INFO] server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testing.go:149: server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: 2025-02-09T06:52:36.520Z [INFO] server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testLoggerAdapter verbose: server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: 2025-02-09T06:52:36.520Z [INFO] server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: entering follower state: follower="Node at 7f28fa71-9f71-f154-6c0f-02ecacf2d840 [Follower]" leader= testing.go:149: server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: 2025-02-09T06:52:36.520Z [INFO] server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: entering follower state: follower="Node at 7f28fa71-9f71-f154-6c0f-02ecacf2d840 [Follower]" leader= testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.520Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.520Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9eb65806-9efa-c938-bc7e-a31eed882685 Address:9eb65806-9efa-c938-bc7e-a31eed882685} {Suffrage:Voter ID:server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd Address:b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd} {Suffrage:Voter ID:server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 Address:758cf40e-3688-7e45-04ad-ef7e8d7fe5c8} {Suffrage:Voter ID:server-31718b08-787f-39a7-143e-bd02cb427cd2 Address:31718b08-787f-39a7-143e-bd02cb427cd2} {Suffrage:Voter ID:server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef Address:8a069cab-9494-f9fa-96fb-2f80e9ef85ef} {Suffrage:Voter ID:server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 Address:7f28fa71-9f71-f154-6c0f-02ecacf2d840} {Suffrage:Voter ID:server-99c861f7-d592-33f2-7a9e-bccd0bd28952 Address:99c861f7-d592-33f2-7a9e-bccd0bd28952}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.520Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:36.520Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.520Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering follower state: follower="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Follower]" leader= testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.520Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering follower state: follower="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Follower]" leader= testLoggerAdapter verbose: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [WARN] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [WARN] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [WARN] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: entering candidate state: node="Node at b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd [Candidate]" term=2 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [WARN] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering candidate state: node="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Candidate]" term=2 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering candidate state: node="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Candidate]" term=2 testing.go:149: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: entering candidate state: node="Node at b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd [Candidate]" term=2 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: election won: tally=4 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: election won: tally=4 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering leader state: leader="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Leader]" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering leader state: leader="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Leader]" testLoggerAdapter verbose: server-31718b08-787f-39a7-143e-bd02cb427cd2: 2025-02-09T06:52:36.586Z [INFO] server-31718b08-787f-39a7-143e-bd02cb427cd2: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: 2025-02-09T06:52:36.586Z [INFO] server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-9eb65806-9efa-c938-bc7e-a31eed882685 testing.go:149: server-31718b08-787f-39a7-143e-bd02cb427cd2: 2025-02-09T06:52:36.586Z [INFO] server-31718b08-787f-39a7-143e-bd02cb427cd2: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.586Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: 2025-02-09T06:52:36.586Z [INFO] server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: 2025-02-09T06:52:36.586Z [INFO] server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: duplicate requestVote for same term: term=2 testing.go:149: server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: 2025-02-09T06:52:36.586Z [INFO] server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef: duplicate requestVote for same term: term=2 testing.go:149: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: duplicate requestVote for same term: term=2 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-9eb65806-9efa-c938-bc7e-a31eed882685 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-31718b08-787f-39a7-143e-bd02cb427cd2 testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.586Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: entering follower state: follower="Node at b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd [Follower]" leader=99c861f7-d592-33f2-7a9e-bccd0bd28952 testing.go:149: server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: 2025-02-09T06:52:36.586Z [INFO] server-7f28fa71-9f71-f154-6c0f-02ecacf2d840: duplicate requestVote for same term: term=2 testing.go:149: server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: 2025-02-09T06:52:36.586Z [INFO] server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8: duplicate requestVote for same term: term=2 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-31718b08-787f-39a7-143e-bd02cb427cd2 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: added peer, starting replication: peer=server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [WARN] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: rejecting vote request since we have a leader: from=b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd leader=99c861f7-d592-33f2-7a9e-bccd0bd28952 testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [WARN] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: rejecting vote request since we have a leader: from=b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd leader=99c861f7-d592-33f2-7a9e-bccd0bd28952 testing.go:149: server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: 2025-02-09T06:52:36.586Z [INFO] server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd: entering follower state: follower="Node at b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd [Follower]" leader=99c861f7-d592-33f2-7a9e-bccd0bd28952 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-9eb65806-9efa-c938-bc7e-a31eed882685 9eb65806-9efa-c938-bc7e-a31eed882685}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-9eb65806-9efa-c938-bc7e-a31eed882685 9eb65806-9efa-c938-bc7e-a31eed882685}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.587Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.587Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.587Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.587Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.586Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: pipelining replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.692Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.587016707 +0000 UTC m=+17.565825471 (1 nodes), 66.567044ms from start of poll, 67.438302ms from cluster start. Timeout at 2025-02-09 06:52:36.692038475 +0000 UTC m=+17.670847232, 105.021761ms after stability testing.go:149: cluster: 2025-02-09T06:52:36.692Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.587016707 +0000 UTC m=+17.565825471 (1 nodes), 66.567044ms from start of poll, 67.438302ms from cluster start. Timeout at 2025-02-09 06:52:36.692038475 +0000 UTC m=+17.670847232, 105.021761ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.692Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:36.692Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.797Z [INFO] stable state for Follower reached at 2025-02-09 06:52:36.692107714 +0000 UTC m=+17.670916482 (6 nodes), 1.158µs from start of poll, 172.529313ms from cluster start. Timeout at 2025-02-09 06:52:36.797237756 +0000 UTC m=+17.776046513, 105.130031ms after stability testing.go:149: cluster: 2025-02-09T06:52:36.797Z [INFO] stable state for Follower reached at 2025-02-09 06:52:36.692107714 +0000 UTC m=+17.670916482 (6 nodes), 1.158µs from start of poll, 172.529313ms from cluster start. Timeout at 2025-02-09 06:52:36.797237756 +0000 UTC m=+17.776046513, 105.130031ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.797Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:36.797Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.902Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.797301925 +0000 UTC m=+17.776110699 (1 nodes), 520ns from start of poll, 277.72353ms from cluster start. Timeout at 2025-02-09 06:52:36.902450091 +0000 UTC m=+17.881258851, 105.148152ms after stability testing.go:149: cluster: 2025-02-09T06:52:36.902Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.797301925 +0000 UTC m=+17.776110699 (1 nodes), 520ns from start of poll, 277.72353ms from cluster start. Timeout at 2025-02-09 06:52:36.902450091 +0000 UTC m=+17.881258851, 105.148152ms after stability testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: entering candidate state: node="Node at 9eb65806-9efa-c938-bc7e-a31eed882685 [Candidate]" term=3 testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: entering candidate state: node="Node at 9eb65806-9efa-c938-bc7e-a31eed882685 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2025-02-09T06:52:36.902Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:36.902Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: election won: tally=4 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering follower state: follower="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Follower]" leader= testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: election won: tally=4 testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: entering leader state: leader="Node at 9eb65806-9efa-c938-bc7e-a31eed882685 [Leader]" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: entering leader state: leader="Node at 9eb65806-9efa-c938-bc7e-a31eed882685 [Leader]" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-31718b08-787f-39a7-143e-bd02cb427cd2 testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-31718b08-787f-39a7-143e-bd02cb427cd2 testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: entering follower state: follower="Node at 99c861f7-d592-33f2-7a9e-bccd0bd28952 [Follower]" leader= testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-99c861f7-d592-33f2-7a9e-bccd0bd28952 testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: added peer, starting replication: peer=server-99c861f7-d592-33f2-7a9e-bccd0bd28952 testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.903Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-99c861f7-d592-33f2-7a9e-bccd0bd28952 99c861f7-d592-33f2-7a9e-bccd0bd28952}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.903Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-99c861f7-d592-33f2-7a9e-bccd0bd28952 99c861f7-d592-33f2-7a9e-bccd0bd28952}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-9eb65806-9efa-c938-bc7e-a31eed882685 9eb65806-9efa-c938-bc7e-a31eed882685}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-9eb65806-9efa-c938-bc7e-a31eed882685 9eb65806-9efa-c938-bc7e-a31eed882685}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:36.902Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: pipelining replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testLoggerAdapter verbose: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" testing.go:149: server-99c861f7-d592-33f2-7a9e-bccd0bd28952: 2025-02-09T06:52:36.902Z [INFO] server-99c861f7-d592-33f2-7a9e-bccd0bd28952: aborting pipeline replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.007Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.902673379 +0000 UTC m=+17.881482158 (1 nodes), 82.949µs from start of poll, 383.094989ms from cluster start. Timeout at 2025-02-09 06:52:37.007835369 +0000 UTC m=+17.986644136, 105.161978ms after stability testing.go:149: cluster: 2025-02-09T06:52:37.007Z [INFO] stable state for Leader reached at 2025-02-09 06:52:36.902673379 +0000 UTC m=+17.881482158 (1 nodes), 82.949µs from start of poll, 383.094989ms from cluster start. Timeout at 2025-02-09 06:52:37.007835369 +0000 UTC m=+17.986644136, 105.161978ms after stability testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-758cf40e-3688-7e45-04ad-ef7e8d7fe5c8 758cf40e-3688-7e45-04ad-ef7e8d7fe5c8}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-31718b08-787f-39a7-143e-bd02cb427cd2 31718b08-787f-39a7-143e-bd02cb427cd2}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-7f28fa71-9f71-f154-6c0f-02ecacf2d840 7f28fa71-9f71-f154-6c0f-02ecacf2d840}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-8a069cab-9494-f9fa-96fb-2f80e9ef85ef 8a069cab-9494-f9fa-96fb-2f80e9ef85ef}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-99c861f7-d592-33f2-7a9e-bccd0bd28952 99c861f7-d592-33f2-7a9e-bccd0bd28952}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-99c861f7-d592-33f2-7a9e-bccd0bd28952 99c861f7-d592-33f2-7a9e-bccd0bd28952}" testLoggerAdapter verbose: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" testing.go:149: server-9eb65806-9efa-c938-bc7e-a31eed882685: 2025-02-09T06:52:37.007Z [INFO] server-9eb65806-9efa-c938-bc7e-a31eed882685: aborting pipeline replication: peer="{Voter server-b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd b73c3e1e-8f7f-2fb5-0eb8-199764cbf5bd}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-53cfd309-9471-b11e-a8d5-5e21553d627a: 2025-02-09T06:52:37.012Z [INFO] server-53cfd309-9471-b11e-a8d5-5e21553d627a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53cfd309-9471-b11e-a8d5-5e21553d627a Address:53cfd309-9471-b11e-a8d5-5e21553d627a} {Suffrage:Voter ID:server-06b4daae-7726-8ab8-97e3-c2066755bb01 Address:06b4daae-7726-8ab8-97e3-c2066755bb01} {Suffrage:Voter ID:server-731f7b0a-c194-21de-9926-3269aad60d1b Address:731f7b0a-c194-21de-9926-3269aad60d1b}]" testing.go:149: server-53cfd309-9471-b11e-a8d5-5e21553d627a: 2025-02-09T06:52:37.012Z [INFO] server-53cfd309-9471-b11e-a8d5-5e21553d627a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53cfd309-9471-b11e-a8d5-5e21553d627a Address:53cfd309-9471-b11e-a8d5-5e21553d627a} {Suffrage:Voter ID:server-06b4daae-7726-8ab8-97e3-c2066755bb01 Address:06b4daae-7726-8ab8-97e3-c2066755bb01} {Suffrage:Voter ID:server-731f7b0a-c194-21de-9926-3269aad60d1b Address:731f7b0a-c194-21de-9926-3269aad60d1b}]" testLoggerAdapter verbose: server-53cfd309-9471-b11e-a8d5-5e21553d627a: 2025-02-09T06:52:37.012Z [INFO] server-53cfd309-9471-b11e-a8d5-5e21553d627a: entering follower state: follower="Node at 53cfd309-9471-b11e-a8d5-5e21553d627a [Follower]" leader= testing.go:149: server-53cfd309-9471-b11e-a8d5-5e21553d627a: 2025-02-09T06:52:37.012Z [INFO] server-53cfd309-9471-b11e-a8d5-5e21553d627a: entering follower state: follower="Node at 53cfd309-9471-b11e-a8d5-5e21553d627a [Follower]" leader= testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.012Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53cfd309-9471-b11e-a8d5-5e21553d627a Address:53cfd309-9471-b11e-a8d5-5e21553d627a} {Suffrage:Voter ID:server-06b4daae-7726-8ab8-97e3-c2066755bb01 Address:06b4daae-7726-8ab8-97e3-c2066755bb01} {Suffrage:Voter ID:server-731f7b0a-c194-21de-9926-3269aad60d1b Address:731f7b0a-c194-21de-9926-3269aad60d1b}]" testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.012Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53cfd309-9471-b11e-a8d5-5e21553d627a Address:53cfd309-9471-b11e-a8d5-5e21553d627a} {Suffrage:Voter ID:server-06b4daae-7726-8ab8-97e3-c2066755bb01 Address:06b4daae-7726-8ab8-97e3-c2066755bb01} {Suffrage:Voter ID:server-731f7b0a-c194-21de-9926-3269aad60d1b Address:731f7b0a-c194-21de-9926-3269aad60d1b}]" testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.012Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: entering follower state: follower="Node at 06b4daae-7726-8ab8-97e3-c2066755bb01 [Follower]" leader= testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.012Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: entering follower state: follower="Node at 06b4daae-7726-8ab8-97e3-c2066755bb01 [Follower]" leader= testLoggerAdapter verbose: server-731f7b0a-c194-21de-9926-3269aad60d1b: 2025-02-09T06:52:37.012Z [INFO] server-731f7b0a-c194-21de-9926-3269aad60d1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53cfd309-9471-b11e-a8d5-5e21553d627a Address:53cfd309-9471-b11e-a8d5-5e21553d627a} {Suffrage:Voter ID:server-06b4daae-7726-8ab8-97e3-c2066755bb01 Address:06b4daae-7726-8ab8-97e3-c2066755bb01} {Suffrage:Voter ID:server-731f7b0a-c194-21de-9926-3269aad60d1b Address:731f7b0a-c194-21de-9926-3269aad60d1b}]" testing.go:149: server-731f7b0a-c194-21de-9926-3269aad60d1b: 2025-02-09T06:52:37.012Z [INFO] server-731f7b0a-c194-21de-9926-3269aad60d1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53cfd309-9471-b11e-a8d5-5e21553d627a Address:53cfd309-9471-b11e-a8d5-5e21553d627a} {Suffrage:Voter ID:server-06b4daae-7726-8ab8-97e3-c2066755bb01 Address:06b4daae-7726-8ab8-97e3-c2066755bb01} {Suffrage:Voter ID:server-731f7b0a-c194-21de-9926-3269aad60d1b Address:731f7b0a-c194-21de-9926-3269aad60d1b}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.012Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:37.012Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-731f7b0a-c194-21de-9926-3269aad60d1b: 2025-02-09T06:52:37.012Z [INFO] server-731f7b0a-c194-21de-9926-3269aad60d1b: entering follower state: follower="Node at 731f7b0a-c194-21de-9926-3269aad60d1b [Follower]" leader= testing.go:149: server-731f7b0a-c194-21de-9926-3269aad60d1b: 2025-02-09T06:52:37.012Z [INFO] server-731f7b0a-c194-21de-9926-3269aad60d1b: entering follower state: follower="Node at 731f7b0a-c194-21de-9926-3269aad60d1b [Follower]" leader= testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [WARN] server-06b4daae-7726-8ab8-97e3-c2066755bb01: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [WARN] server-06b4daae-7726-8ab8-97e3-c2066755bb01: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: entering candidate state: node="Node at 06b4daae-7726-8ab8-97e3-c2066755bb01 [Candidate]" term=2 testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: entering candidate state: node="Node at 06b4daae-7726-8ab8-97e3-c2066755bb01 [Candidate]" term=2 testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: election won: tally=2 testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: election won: tally=2 testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: entering leader state: leader="Node at 06b4daae-7726-8ab8-97e3-c2066755bb01 [Leader]" testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: entering leader state: leader="Node at 06b4daae-7726-8ab8-97e3-c2066755bb01 [Leader]" testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: added peer, starting replication: peer=server-53cfd309-9471-b11e-a8d5-5e21553d627a testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: added peer, starting replication: peer=server-53cfd309-9471-b11e-a8d5-5e21553d627a testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: added peer, starting replication: peer=server-731f7b0a-c194-21de-9926-3269aad60d1b testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: added peer, starting replication: peer=server-731f7b0a-c194-21de-9926-3269aad60d1b testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: pipelining replication: peer="{Voter server-53cfd309-9471-b11e-a8d5-5e21553d627a 53cfd309-9471-b11e-a8d5-5e21553d627a}" testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: pipelining replication: peer="{Voter server-53cfd309-9471-b11e-a8d5-5e21553d627a 53cfd309-9471-b11e-a8d5-5e21553d627a}" testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: pipelining replication: peer="{Voter server-731f7b0a-c194-21de-9926-3269aad60d1b 731f7b0a-c194-21de-9926-3269aad60d1b}" testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.070Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: pipelining replication: peer="{Voter server-731f7b0a-c194-21de-9926-3269aad60d1b 731f7b0a-c194-21de-9926-3269aad60d1b}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.175Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.070459316 +0000 UTC m=+18.049268091 (1 nodes), 58.160562ms from start of poll, 58.511395ms from cluster start. Timeout at 2025-02-09 06:52:37.175473581 +0000 UTC m=+18.154282341, 105.01425ms after stability testing.go:149: cluster: 2025-02-09T06:52:37.175Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.070459316 +0000 UTC m=+18.049268091 (1 nodes), 58.160562ms from start of poll, 58.511395ms from cluster start. Timeout at 2025-02-09 06:52:37.175473581 +0000 UTC m=+18.154282341, 105.01425ms after stability testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.175Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: aborting pipeline replication: peer="{Voter server-731f7b0a-c194-21de-9926-3269aad60d1b 731f7b0a-c194-21de-9926-3269aad60d1b}" testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.175Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: aborting pipeline replication: peer="{Voter server-731f7b0a-c194-21de-9926-3269aad60d1b 731f7b0a-c194-21de-9926-3269aad60d1b}" testLoggerAdapter verbose: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.175Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: aborting pipeline replication: peer="{Voter server-53cfd309-9471-b11e-a8d5-5e21553d627a 53cfd309-9471-b11e-a8d5-5e21553d627a}" testing.go:149: server-06b4daae-7726-8ab8-97e3-c2066755bb01: 2025-02-09T06:52:37.175Z [INFO] server-06b4daae-7726-8ab8-97e3-c2066755bb01: aborting pipeline replication: peer="{Voter server-53cfd309-9471-b11e-a8d5-5e21553d627a 53cfd309-9471-b11e-a8d5-5e21553d627a}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.176Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20e0bb57-6e37-3eb7-3c9e-dfc807551936 Address:20e0bb57-6e37-3eb7-3c9e-dfc807551936} {Suffrage:Voter ID:server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf Address:532ba30b-00ae-29b9-9acb-2a629f5e0ecf} {Suffrage:Voter ID:server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 Address:e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}]" testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.176Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20e0bb57-6e37-3eb7-3c9e-dfc807551936 Address:20e0bb57-6e37-3eb7-3c9e-dfc807551936} {Suffrage:Voter ID:server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf Address:532ba30b-00ae-29b9-9acb-2a629f5e0ecf} {Suffrage:Voter ID:server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 Address:e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}]" testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.176Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: entering follower state: follower="Node at 20e0bb57-6e37-3eb7-3c9e-dfc807551936 [Follower]" leader= testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.176Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: entering follower state: follower="Node at 20e0bb57-6e37-3eb7-3c9e-dfc807551936 [Follower]" leader= testLoggerAdapter verbose: server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: 2025-02-09T06:52:37.176Z [INFO] server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20e0bb57-6e37-3eb7-3c9e-dfc807551936 Address:20e0bb57-6e37-3eb7-3c9e-dfc807551936} {Suffrage:Voter ID:server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf Address:532ba30b-00ae-29b9-9acb-2a629f5e0ecf} {Suffrage:Voter ID:server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 Address:e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}]" testing.go:149: server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: 2025-02-09T06:52:37.176Z [INFO] server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20e0bb57-6e37-3eb7-3c9e-dfc807551936 Address:20e0bb57-6e37-3eb7-3c9e-dfc807551936} {Suffrage:Voter ID:server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf Address:532ba30b-00ae-29b9-9acb-2a629f5e0ecf} {Suffrage:Voter ID:server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 Address:e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}]" testLoggerAdapter verbose: server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: 2025-02-09T06:52:37.176Z [INFO] server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: entering follower state: follower="Node at 532ba30b-00ae-29b9-9acb-2a629f5e0ecf [Follower]" leader= testing.go:149: server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: 2025-02-09T06:52:37.176Z [INFO] server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: entering follower state: follower="Node at 532ba30b-00ae-29b9-9acb-2a629f5e0ecf [Follower]" leader= testLoggerAdapter verbose: server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: 2025-02-09T06:52:37.176Z [INFO] server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20e0bb57-6e37-3eb7-3c9e-dfc807551936 Address:20e0bb57-6e37-3eb7-3c9e-dfc807551936} {Suffrage:Voter ID:server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf Address:532ba30b-00ae-29b9-9acb-2a629f5e0ecf} {Suffrage:Voter ID:server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 Address:e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}]" testing.go:149: server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: 2025-02-09T06:52:37.176Z [INFO] server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20e0bb57-6e37-3eb7-3c9e-dfc807551936 Address:20e0bb57-6e37-3eb7-3c9e-dfc807551936} {Suffrage:Voter ID:server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf Address:532ba30b-00ae-29b9-9acb-2a629f5e0ecf} {Suffrage:Voter ID:server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 Address:e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.176Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:37.176Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: 2025-02-09T06:52:37.176Z [INFO] server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: entering follower state: follower="Node at e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 [Follower]" leader= testing.go:149: server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: 2025-02-09T06:52:37.176Z [INFO] server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6: entering follower state: follower="Node at e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 [Follower]" leader= testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [WARN] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [WARN] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: entering candidate state: node="Node at 20e0bb57-6e37-3eb7-3c9e-dfc807551936 [Candidate]" term=2 testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: entering candidate state: node="Node at 20e0bb57-6e37-3eb7-3c9e-dfc807551936 [Candidate]" term=2 testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: election won: tally=2 testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: election won: tally=2 testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: entering leader state: leader="Node at 20e0bb57-6e37-3eb7-3c9e-dfc807551936 [Leader]" testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: entering leader state: leader="Node at 20e0bb57-6e37-3eb7-3c9e-dfc807551936 [Leader]" testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: added peer, starting replication: peer=server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: added peer, starting replication: peer=server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: added peer, starting replication: peer=server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: added peer, starting replication: peer=server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: pipelining replication: peer="{Voter server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}" testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.248Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: pipelining replication: peer="{Voter server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}" testLoggerAdapter verbose: server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: 2025-02-09T06:52:37.248Z [WARN] server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: rejecting vote request since our last term is greater: candidate=20e0bb57-6e37-3eb7-3c9e-dfc807551936 last-term=2 last-candidate-term=1 testing.go:149: server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: 2025-02-09T06:52:37.248Z [WARN] server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf: rejecting vote request since our last term is greater: candidate=20e0bb57-6e37-3eb7-3c9e-dfc807551936 last-term=2 last-candidate-term=1 testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.249Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: pipelining replication: peer="{Voter server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf 532ba30b-00ae-29b9-9acb-2a629f5e0ecf}" testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.249Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: pipelining replication: peer="{Voter server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf 532ba30b-00ae-29b9-9acb-2a629f5e0ecf}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.354Z [INFO] stable state for Follower reached at 2025-02-09 06:52:37.248985261 +0000 UTC m=+18.227794038 (2 nodes), 72.293829ms from start of poll, 72.621465ms from cluster start. Timeout at 2025-02-09 06:52:37.354522846 +0000 UTC m=+18.333331604, 105.537566ms after stability testing.go:149: cluster: 2025-02-09T06:52:37.354Z [INFO] stable state for Follower reached at 2025-02-09 06:52:37.248985261 +0000 UTC m=+18.227794038 (2 nodes), 72.293829ms from start of poll, 72.621465ms from cluster start. Timeout at 2025-02-09 06:52:37.354522846 +0000 UTC m=+18.333331604, 105.537566ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.354Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:37.354Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.460Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.354605392 +0000 UTC m=+18.333414155 (1 nodes), 644ns from start of poll, 178.241582ms from cluster start. Timeout at 2025-02-09 06:52:37.460508157 +0000 UTC m=+18.439316915, 105.90276ms after stability testing.go:149: cluster: 2025-02-09T06:52:37.460Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.354605392 +0000 UTC m=+18.333414155 (1 nodes), 644ns from start of poll, 178.241582ms from cluster start. Timeout at 2025-02-09 06:52:37.460508157 +0000 UTC m=+18.439316915, 105.90276ms after stability testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.460Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: aborting pipeline replication: peer="{Voter server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}" testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.460Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: aborting pipeline replication: peer="{Voter server-e073dc33-ddd1-5e3f-af0a-1a511caf1ea6 e073dc33-ddd1-5e3f-af0a-1a511caf1ea6}" testLoggerAdapter verbose: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.460Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: aborting pipeline replication: peer="{Voter server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf 532ba30b-00ae-29b9-9acb-2a629f5e0ecf}" testing.go:149: server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: 2025-02-09T06:52:37.460Z [INFO] server-20e0bb57-6e37-3eb7-3c9e-dfc807551936: aborting pipeline replication: peer="{Voter server-532ba30b-00ae-29b9-9acb-2a629f5e0ecf 532ba30b-00ae-29b9-9acb-2a629f5e0ecf}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.461Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4bd74c1-05c6-e416-9825-d5aba531a5ad Address:a4bd74c1-05c6-e416-9825-d5aba531a5ad} {Suffrage:Voter ID:server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 Address:408d0d66-6ddc-aa04-77f7-ccd545d2b7a0} {Suffrage:Voter ID:server-286bbf55-db71-bc61-6347-72349c32f537 Address:286bbf55-db71-bc61-6347-72349c32f537}]" testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.461Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4bd74c1-05c6-e416-9825-d5aba531a5ad Address:a4bd74c1-05c6-e416-9825-d5aba531a5ad} {Suffrage:Voter ID:server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 Address:408d0d66-6ddc-aa04-77f7-ccd545d2b7a0} {Suffrage:Voter ID:server-286bbf55-db71-bc61-6347-72349c32f537 Address:286bbf55-db71-bc61-6347-72349c32f537}]" testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.461Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: entering follower state: follower="Node at a4bd74c1-05c6-e416-9825-d5aba531a5ad [Follower]" leader= testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.461Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: entering follower state: follower="Node at a4bd74c1-05c6-e416-9825-d5aba531a5ad [Follower]" leader= testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.461Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4bd74c1-05c6-e416-9825-d5aba531a5ad Address:a4bd74c1-05c6-e416-9825-d5aba531a5ad} {Suffrage:Voter ID:server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 Address:408d0d66-6ddc-aa04-77f7-ccd545d2b7a0} {Suffrage:Voter ID:server-286bbf55-db71-bc61-6347-72349c32f537 Address:286bbf55-db71-bc61-6347-72349c32f537}]" testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.461Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4bd74c1-05c6-e416-9825-d5aba531a5ad Address:a4bd74c1-05c6-e416-9825-d5aba531a5ad} {Suffrage:Voter ID:server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 Address:408d0d66-6ddc-aa04-77f7-ccd545d2b7a0} {Suffrage:Voter ID:server-286bbf55-db71-bc61-6347-72349c32f537 Address:286bbf55-db71-bc61-6347-72349c32f537}]" testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.461Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering follower state: follower="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Follower]" leader= testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.461Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering follower state: follower="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Follower]" leader= testLoggerAdapter verbose: server-286bbf55-db71-bc61-6347-72349c32f537: 2025-02-09T06:52:37.461Z [INFO] server-286bbf55-db71-bc61-6347-72349c32f537: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4bd74c1-05c6-e416-9825-d5aba531a5ad Address:a4bd74c1-05c6-e416-9825-d5aba531a5ad} {Suffrage:Voter ID:server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 Address:408d0d66-6ddc-aa04-77f7-ccd545d2b7a0} {Suffrage:Voter ID:server-286bbf55-db71-bc61-6347-72349c32f537 Address:286bbf55-db71-bc61-6347-72349c32f537}]" testing.go:149: server-286bbf55-db71-bc61-6347-72349c32f537: 2025-02-09T06:52:37.461Z [INFO] server-286bbf55-db71-bc61-6347-72349c32f537: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4bd74c1-05c6-e416-9825-d5aba531a5ad Address:a4bd74c1-05c6-e416-9825-d5aba531a5ad} {Suffrage:Voter ID:server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 Address:408d0d66-6ddc-aa04-77f7-ccd545d2b7a0} {Suffrage:Voter ID:server-286bbf55-db71-bc61-6347-72349c32f537 Address:286bbf55-db71-bc61-6347-72349c32f537}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.461Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:37.461Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-286bbf55-db71-bc61-6347-72349c32f537: 2025-02-09T06:52:37.461Z [INFO] server-286bbf55-db71-bc61-6347-72349c32f537: entering follower state: follower="Node at 286bbf55-db71-bc61-6347-72349c32f537 [Follower]" leader= testing.go:149: server-286bbf55-db71-bc61-6347-72349c32f537: 2025-02-09T06:52:37.461Z [INFO] server-286bbf55-db71-bc61-6347-72349c32f537: entering follower state: follower="Node at 286bbf55-db71-bc61-6347-72349c32f537 [Follower]" leader= testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [WARN] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [WARN] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering candidate state: node="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Candidate]" term=2 testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering candidate state: node="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Candidate]" term=2 testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: election won: tally=2 testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: election won: tally=2 testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering leader state: leader="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Leader]" testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering leader state: leader="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Leader]" testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: added peer, starting replication: peer=server-a4bd74c1-05c6-e416-9825-d5aba531a5ad testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: added peer, starting replication: peer=server-a4bd74c1-05c6-e416-9825-d5aba531a5ad testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: added peer, starting replication: peer=server-286bbf55-db71-bc61-6347-72349c32f537 testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: added peer, starting replication: peer=server-286bbf55-db71-bc61-6347-72349c32f537 testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: pipelining replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: pipelining replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: pipelining replication: peer="{Voter server-a4bd74c1-05c6-e416-9825-d5aba531a5ad a4bd74c1-05c6-e416-9825-d5aba531a5ad}" testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.514Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: pipelining replication: peer="{Voter server-a4bd74c1-05c6-e416-9825-d5aba531a5ad a4bd74c1-05c6-e416-9825-d5aba531a5ad}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.620Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.514836443 +0000 UTC m=+18.493645223 (1 nodes), 52.899763ms from start of poll, 53.283114ms from cluster start. Timeout at 2025-02-09 06:52:37.620173734 +0000 UTC m=+18.598982494, 105.337271ms after stability testing.go:149: cluster: 2025-02-09T06:52:37.620Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.514836443 +0000 UTC m=+18.493645223 (1 nodes), 52.899763ms from start of poll, 53.283114ms from cluster start. Timeout at 2025-02-09 06:52:37.620173734 +0000 UTC m=+18.598982494, 105.337271ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.620Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:37.620Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.725Z [INFO] stable state for Follower reached at 2025-02-09 06:52:37.620265578 +0000 UTC m=+18.599074341 (2 nodes), 825ns from start of poll, 158.712232ms from cluster start. Timeout at 2025-02-09 06:52:37.725445102 +0000 UTC m=+18.704253871, 105.17953ms after stability testing.go:149: cluster: 2025-02-09T06:52:37.725Z [INFO] stable state for Follower reached at 2025-02-09 06:52:37.620265578 +0000 UTC m=+18.599074341 (2 nodes), 825ns from start of poll, 158.712232ms from cluster start. Timeout at 2025-02-09 06:52:37.725445102 +0000 UTC m=+18.704253871, 105.17953ms after stability testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: entering candidate state: node="Node at a4bd74c1-05c6-e416-9825-d5aba531a5ad [Candidate]" term=3 testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: entering candidate state: node="Node at a4bd74c1-05c6-e416-9825-d5aba531a5ad [Candidate]" term=3 testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: election won: tally=2 testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: election won: tally=2 testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: entering leader state: leader="Node at a4bd74c1-05c6-e416-9825-d5aba531a5ad [Leader]" testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: entering leader state: leader="Node at a4bd74c1-05c6-e416-9825-d5aba531a5ad [Leader]" testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: added peer, starting replication: peer=server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.739Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: added peer, starting replication: peer=server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: added peer, starting replication: peer=server-286bbf55-db71-bc61-6347-72349c32f537 testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.739Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering follower state: follower="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Follower]" leader= testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: added peer, starting replication: peer=server-286bbf55-db71-bc61-6347-72349c32f537 testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: pipelining replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.739Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: entering follower state: follower="Node at 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 [Follower]" leader= testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.739Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: aborting pipeline replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testing.go:149: cluster: 2025-02-09T06:52:37.739Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: pipelining replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: pipelining replication: peer="{Voter server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0}" testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.739Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: aborting pipeline replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testLoggerAdapter verbose: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.739Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: aborting pipeline replication: peer="{Voter server-a4bd74c1-05c6-e416-9825-d5aba531a5ad a4bd74c1-05c6-e416-9825-d5aba531a5ad}" testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.739Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: pipelining replication: peer="{Voter server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0}" testing.go:149: server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: 2025-02-09T06:52:37.739Z [INFO] server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0: aborting pipeline replication: peer="{Voter server-a4bd74c1-05c6-e416-9825-d5aba531a5ad a4bd74c1-05c6-e416-9825-d5aba531a5ad}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.845Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.740020051 +0000 UTC m=+18.718828829 (1 nodes), 16.664µs from start of poll, 278.46672ms from cluster start. Timeout at 2025-02-09 06:52:37.845281148 +0000 UTC m=+18.824089938, 105.261109ms after stability testing.go:149: cluster: 2025-02-09T06:52:37.845Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.740020051 +0000 UTC m=+18.718828829 (1 nodes), 16.664µs from start of poll, 278.46672ms from cluster start. Timeout at 2025-02-09 06:52:37.845281148 +0000 UTC m=+18.824089938, 105.261109ms after stability testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.845Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: aborting pipeline replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.845Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: aborting pipeline replication: peer="{Voter server-286bbf55-db71-bc61-6347-72349c32f537 286bbf55-db71-bc61-6347-72349c32f537}" testLoggerAdapter verbose: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.845Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: aborting pipeline replication: peer="{Voter server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0}" testing.go:149: server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: 2025-02-09T06:52:37.845Z [INFO] server-a4bd74c1-05c6-e416-9825-d5aba531a5ad: aborting pipeline replication: peer="{Voter server-408d0d66-6ddc-aa04-77f7-ccd545d2b7a0 408d0d66-6ddc-aa04-77f7-ccd545d2b7a0}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.38s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-53d5b798-497b-8eca-effc-e3da80dadfb5: 2025-02-09T06:52:37.846Z [INFO] server-53d5b798-497b-8eca-effc-e3da80dadfb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53d5b798-497b-8eca-effc-e3da80dadfb5 Address:53d5b798-497b-8eca-effc-e3da80dadfb5} {Suffrage:Voter ID:server-9e5d54b2-92b2-1335-87b8-950f94e7c927 Address:9e5d54b2-92b2-1335-87b8-950f94e7c927} {Suffrage:Voter ID:server-f3695af8-c6ce-5ef1-5d0e-689a4594682b Address:f3695af8-c6ce-5ef1-5d0e-689a4594682b}]" testing.go:149: server-53d5b798-497b-8eca-effc-e3da80dadfb5: 2025-02-09T06:52:37.846Z [INFO] server-53d5b798-497b-8eca-effc-e3da80dadfb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53d5b798-497b-8eca-effc-e3da80dadfb5 Address:53d5b798-497b-8eca-effc-e3da80dadfb5} {Suffrage:Voter ID:server-9e5d54b2-92b2-1335-87b8-950f94e7c927 Address:9e5d54b2-92b2-1335-87b8-950f94e7c927} {Suffrage:Voter ID:server-f3695af8-c6ce-5ef1-5d0e-689a4594682b Address:f3695af8-c6ce-5ef1-5d0e-689a4594682b}]" testLoggerAdapter verbose: server-53d5b798-497b-8eca-effc-e3da80dadfb5: 2025-02-09T06:52:37.846Z [INFO] server-53d5b798-497b-8eca-effc-e3da80dadfb5: entering follower state: follower="Node at 53d5b798-497b-8eca-effc-e3da80dadfb5 [Follower]" leader= testing.go:149: server-53d5b798-497b-8eca-effc-e3da80dadfb5: 2025-02-09T06:52:37.846Z [INFO] server-53d5b798-497b-8eca-effc-e3da80dadfb5: entering follower state: follower="Node at 53d5b798-497b-8eca-effc-e3da80dadfb5 [Follower]" leader= testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.846Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53d5b798-497b-8eca-effc-e3da80dadfb5 Address:53d5b798-497b-8eca-effc-e3da80dadfb5} {Suffrage:Voter ID:server-9e5d54b2-92b2-1335-87b8-950f94e7c927 Address:9e5d54b2-92b2-1335-87b8-950f94e7c927} {Suffrage:Voter ID:server-f3695af8-c6ce-5ef1-5d0e-689a4594682b Address:f3695af8-c6ce-5ef1-5d0e-689a4594682b}]" testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.846Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53d5b798-497b-8eca-effc-e3da80dadfb5 Address:53d5b798-497b-8eca-effc-e3da80dadfb5} {Suffrage:Voter ID:server-9e5d54b2-92b2-1335-87b8-950f94e7c927 Address:9e5d54b2-92b2-1335-87b8-950f94e7c927} {Suffrage:Voter ID:server-f3695af8-c6ce-5ef1-5d0e-689a4594682b Address:f3695af8-c6ce-5ef1-5d0e-689a4594682b}]" testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.846Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: entering follower state: follower="Node at 9e5d54b2-92b2-1335-87b8-950f94e7c927 [Follower]" leader= testLoggerAdapter verbose: server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: 2025-02-09T06:52:37.846Z [INFO] server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53d5b798-497b-8eca-effc-e3da80dadfb5 Address:53d5b798-497b-8eca-effc-e3da80dadfb5} {Suffrage:Voter ID:server-9e5d54b2-92b2-1335-87b8-950f94e7c927 Address:9e5d54b2-92b2-1335-87b8-950f94e7c927} {Suffrage:Voter ID:server-f3695af8-c6ce-5ef1-5d0e-689a4594682b Address:f3695af8-c6ce-5ef1-5d0e-689a4594682b}]" testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.846Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: entering follower state: follower="Node at 9e5d54b2-92b2-1335-87b8-950f94e7c927 [Follower]" leader= testing.go:149: server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: 2025-02-09T06:52:37.846Z [INFO] server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53d5b798-497b-8eca-effc-e3da80dadfb5 Address:53d5b798-497b-8eca-effc-e3da80dadfb5} {Suffrage:Voter ID:server-9e5d54b2-92b2-1335-87b8-950f94e7c927 Address:9e5d54b2-92b2-1335-87b8-950f94e7c927} {Suffrage:Voter ID:server-f3695af8-c6ce-5ef1-5d0e-689a4594682b Address:f3695af8-c6ce-5ef1-5d0e-689a4594682b}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:37.846Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:37.846Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: 2025-02-09T06:52:37.846Z [INFO] server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: entering follower state: follower="Node at f3695af8-c6ce-5ef1-5d0e-689a4594682b [Follower]" leader= testing.go:149: server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: 2025-02-09T06:52:37.846Z [INFO] server-f3695af8-c6ce-5ef1-5d0e-689a4594682b: entering follower state: follower="Node at f3695af8-c6ce-5ef1-5d0e-689a4594682b [Follower]" leader= testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.905Z [WARN] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.905Z [WARN] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.905Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: entering candidate state: node="Node at 9e5d54b2-92b2-1335-87b8-950f94e7c927 [Candidate]" term=2 testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.905Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: entering candidate state: node="Node at 9e5d54b2-92b2-1335-87b8-950f94e7c927 [Candidate]" term=2 testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.905Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: election won: tally=2 testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.905Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: election won: tally=2 testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: entering leader state: leader="Node at 9e5d54b2-92b2-1335-87b8-950f94e7c927 [Leader]" testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: entering leader state: leader="Node at 9e5d54b2-92b2-1335-87b8-950f94e7c927 [Leader]" testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: added peer, starting replication: peer=server-53d5b798-497b-8eca-effc-e3da80dadfb5 testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: added peer, starting replication: peer=server-53d5b798-497b-8eca-effc-e3da80dadfb5 testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: added peer, starting replication: peer=server-f3695af8-c6ce-5ef1-5d0e-689a4594682b testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: added peer, starting replication: peer=server-f3695af8-c6ce-5ef1-5d0e-689a4594682b testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: pipelining replication: peer="{Voter server-53d5b798-497b-8eca-effc-e3da80dadfb5 53d5b798-497b-8eca-effc-e3da80dadfb5}" testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: pipelining replication: peer="{Voter server-53d5b798-497b-8eca-effc-e3da80dadfb5 53d5b798-497b-8eca-effc-e3da80dadfb5}" testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: pipelining replication: peer="{Voter server-f3695af8-c6ce-5ef1-5d0e-689a4594682b f3695af8-c6ce-5ef1-5d0e-689a4594682b}" testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:37.906Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: pipelining replication: peer="{Voter server-f3695af8-c6ce-5ef1-5d0e-689a4594682b f3695af8-c6ce-5ef1-5d0e-689a4594682b}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.011Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.906052933 +0000 UTC m=+18.884861711 (1 nodes), 59.27485ms from start of poll, 59.682457ms from cluster start. Timeout at 2025-02-09 06:52:38.011621996 +0000 UTC m=+18.990430764, 105.569053ms after stability testing.go:149: cluster: 2025-02-09T06:52:38.011Z [INFO] stable state for Leader reached at 2025-02-09 06:52:37.906052933 +0000 UTC m=+18.884861711 (1 nodes), 59.27485ms from start of poll, 59.682457ms from cluster start. Timeout at 2025-02-09 06:52:38.011621996 +0000 UTC m=+18.990430764, 105.569053ms after stability testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:38.011Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: cannot transfer leadership to itself testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:38.011Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: cannot transfer leadership to itself testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:38.011Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: aborting pipeline replication: peer="{Voter server-f3695af8-c6ce-5ef1-5d0e-689a4594682b f3695af8-c6ce-5ef1-5d0e-689a4594682b}" testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:38.011Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: aborting pipeline replication: peer="{Voter server-f3695af8-c6ce-5ef1-5d0e-689a4594682b f3695af8-c6ce-5ef1-5d0e-689a4594682b}" testLoggerAdapter verbose: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:38.011Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: aborting pipeline replication: peer="{Voter server-53d5b798-497b-8eca-effc-e3da80dadfb5 53d5b798-497b-8eca-effc-e3da80dadfb5}" testing.go:149: server-9e5d54b2-92b2-1335-87b8-950f94e7c927: 2025-02-09T06:52:38.011Z [INFO] server-9e5d54b2-92b2-1335-87b8-950f94e7c927: aborting pipeline replication: peer="{Voter server-53d5b798-497b-8eca-effc-e3da80dadfb5 53d5b798-497b-8eca-effc-e3da80dadfb5}" --- PASS: TestRaft_LeadershipTransferToItself (0.17s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-2942ab34-c24a-f504-08d5-837689be6a7d: 2025-02-09T06:52:38.012Z [INFO] server-2942ab34-c24a-f504-08d5-837689be6a7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2942ab34-c24a-f504-08d5-837689be6a7d Address:2942ab34-c24a-f504-08d5-837689be6a7d} {Suffrage:Voter ID:server-2d521933-f9a6-0b6f-d886-3626d358a11c Address:2d521933-f9a6-0b6f-d886-3626d358a11c} {Suffrage:Voter ID:server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e Address:c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}]" testing.go:149: server-2942ab34-c24a-f504-08d5-837689be6a7d: 2025-02-09T06:52:38.012Z [INFO] server-2942ab34-c24a-f504-08d5-837689be6a7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2942ab34-c24a-f504-08d5-837689be6a7d Address:2942ab34-c24a-f504-08d5-837689be6a7d} {Suffrage:Voter ID:server-2d521933-f9a6-0b6f-d886-3626d358a11c Address:2d521933-f9a6-0b6f-d886-3626d358a11c} {Suffrage:Voter ID:server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e Address:c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}]" testLoggerAdapter verbose: server-2942ab34-c24a-f504-08d5-837689be6a7d: 2025-02-09T06:52:38.012Z [INFO] server-2942ab34-c24a-f504-08d5-837689be6a7d: entering follower state: follower="Node at 2942ab34-c24a-f504-08d5-837689be6a7d [Follower]" leader= testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.012Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2942ab34-c24a-f504-08d5-837689be6a7d Address:2942ab34-c24a-f504-08d5-837689be6a7d} {Suffrage:Voter ID:server-2d521933-f9a6-0b6f-d886-3626d358a11c Address:2d521933-f9a6-0b6f-d886-3626d358a11c} {Suffrage:Voter ID:server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e Address:c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}]" testing.go:149: server-2942ab34-c24a-f504-08d5-837689be6a7d: 2025-02-09T06:52:38.012Z [INFO] server-2942ab34-c24a-f504-08d5-837689be6a7d: entering follower state: follower="Node at 2942ab34-c24a-f504-08d5-837689be6a7d [Follower]" leader= testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.012Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2942ab34-c24a-f504-08d5-837689be6a7d Address:2942ab34-c24a-f504-08d5-837689be6a7d} {Suffrage:Voter ID:server-2d521933-f9a6-0b6f-d886-3626d358a11c Address:2d521933-f9a6-0b6f-d886-3626d358a11c} {Suffrage:Voter ID:server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e Address:c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}]" testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.013Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: entering follower state: follower="Node at 2d521933-f9a6-0b6f-d886-3626d358a11c [Follower]" leader= testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.013Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: entering follower state: follower="Node at 2d521933-f9a6-0b6f-d886-3626d358a11c [Follower]" leader= testLoggerAdapter verbose: server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: 2025-02-09T06:52:38.013Z [INFO] server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2942ab34-c24a-f504-08d5-837689be6a7d Address:2942ab34-c24a-f504-08d5-837689be6a7d} {Suffrage:Voter ID:server-2d521933-f9a6-0b6f-d886-3626d358a11c Address:2d521933-f9a6-0b6f-d886-3626d358a11c} {Suffrage:Voter ID:server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e Address:c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}]" testing.go:149: server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: 2025-02-09T06:52:38.013Z [INFO] server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2942ab34-c24a-f504-08d5-837689be6a7d Address:2942ab34-c24a-f504-08d5-837689be6a7d} {Suffrage:Voter ID:server-2d521933-f9a6-0b6f-d886-3626d358a11c Address:2d521933-f9a6-0b6f-d886-3626d358a11c} {Suffrage:Voter ID:server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e Address:c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.013Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:38.013Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: 2025-02-09T06:52:38.013Z [INFO] server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: entering follower state: follower="Node at c1de82a1-a2ba-9cfc-47da-4efa687b0e8e [Follower]" leader= testing.go:149: server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: 2025-02-09T06:52:38.013Z [INFO] server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e: entering follower state: follower="Node at c1de82a1-a2ba-9cfc-47da-4efa687b0e8e [Follower]" leader= testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [WARN] server-2d521933-f9a6-0b6f-d886-3626d358a11c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [WARN] server-2d521933-f9a6-0b6f-d886-3626d358a11c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: entering candidate state: node="Node at 2d521933-f9a6-0b6f-d886-3626d358a11c [Candidate]" term=2 testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: entering candidate state: node="Node at 2d521933-f9a6-0b6f-d886-3626d358a11c [Candidate]" term=2 testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: election won: tally=2 testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: election won: tally=2 testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: entering leader state: leader="Node at 2d521933-f9a6-0b6f-d886-3626d358a11c [Leader]" testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: entering leader state: leader="Node at 2d521933-f9a6-0b6f-d886-3626d358a11c [Leader]" testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: added peer, starting replication: peer=server-2942ab34-c24a-f504-08d5-837689be6a7d testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: added peer, starting replication: peer=server-2942ab34-c24a-f504-08d5-837689be6a7d testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: added peer, starting replication: peer=server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: added peer, starting replication: peer=server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: pipelining replication: peer="{Voter server-2942ab34-c24a-f504-08d5-837689be6a7d 2942ab34-c24a-f504-08d5-837689be6a7d}" testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: pipelining replication: peer="{Voter server-2942ab34-c24a-f504-08d5-837689be6a7d 2942ab34-c24a-f504-08d5-837689be6a7d}" testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: pipelining replication: peer="{Voter server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}" testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.085Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: pipelining replication: peer="{Voter server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.191Z [INFO] stable state for Leader reached at 2025-02-09 06:52:38.085504524 +0000 UTC m=+19.064313302 (1 nodes), 72.267395ms from start of poll, 72.783123ms from cluster start. Timeout at 2025-02-09 06:52:38.191080707 +0000 UTC m=+19.169889467, 105.576165ms after stability testing.go:149: cluster: 2025-02-09T06:52:38.191Z [INFO] stable state for Leader reached at 2025-02-09 06:52:38.085504524 +0000 UTC m=+19.064313302 (1 nodes), 72.267395ms from start of poll, 72.783123ms from cluster start. Timeout at 2025-02-09 06:52:38.191080707 +0000 UTC m=+19.169889467, 105.576165ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.191Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2025-02-09T06:52:38.191Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.296Z [INFO] stable state for Follower reached at 2025-02-09 06:52:38.19117293 +0000 UTC m=+19.169981710 (2 nodes), 878ns from start of poll, 178.451531ms from cluster start. Timeout at 2025-02-09 06:52:38.296342086 +0000 UTC m=+19.275150869, 105.169159ms after stability testing.go:149: cluster: 2025-02-09T06:52:38.296Z [INFO] stable state for Follower reached at 2025-02-09 06:52:38.19117293 +0000 UTC m=+19.169981710 (2 nodes), 878ns from start of poll, 178.451531ms from cluster start. Timeout at 2025-02-09 06:52:38.296342086 +0000 UTC m=+19.275150869, 105.169159ms after stability testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.439Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: aborting pipeline replication: peer="{Voter server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}" testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.439Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: aborting pipeline replication: peer="{Voter server-c1de82a1-a2ba-9cfc-47da-4efa687b0e8e c1de82a1-a2ba-9cfc-47da-4efa687b0e8e}" testLoggerAdapter verbose: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.439Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: aborting pipeline replication: peer="{Voter server-2942ab34-c24a-f504-08d5-837689be6a7d 2942ab34-c24a-f504-08d5-837689be6a7d}" testing.go:149: server-2d521933-f9a6-0b6f-d886-3626d358a11c: 2025-02-09T06:52:38.439Z [INFO] server-2d521933-f9a6-0b6f-d886-3626d358a11c: aborting pipeline replication: peer="{Voter server-2942ab34-c24a-f504-08d5-837689be6a7d 2942ab34-c24a-f504-08d5-837689be6a7d}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.43s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.440Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c97ed00-795e-25e0-a0d3-6328b9991ac6 Address:0c97ed00-795e-25e0-a0d3-6328b9991ac6} {Suffrage:Voter ID:server-e64640f2-9154-c4ed-a882-cc366d5dd235 Address:e64640f2-9154-c4ed-a882-cc366d5dd235}]" testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.440Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c97ed00-795e-25e0-a0d3-6328b9991ac6 Address:0c97ed00-795e-25e0-a0d3-6328b9991ac6} {Suffrage:Voter ID:server-e64640f2-9154-c4ed-a882-cc366d5dd235 Address:e64640f2-9154-c4ed-a882-cc366d5dd235}]" testLoggerAdapter verbose: server-e64640f2-9154-c4ed-a882-cc366d5dd235: 2025-02-09T06:52:38.440Z [INFO] server-e64640f2-9154-c4ed-a882-cc366d5dd235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c97ed00-795e-25e0-a0d3-6328b9991ac6 Address:0c97ed00-795e-25e0-a0d3-6328b9991ac6} {Suffrage:Voter ID:server-e64640f2-9154-c4ed-a882-cc366d5dd235 Address:e64640f2-9154-c4ed-a882-cc366d5dd235}]" testing.go:149: server-e64640f2-9154-c4ed-a882-cc366d5dd235: 2025-02-09T06:52:38.440Z [INFO] server-e64640f2-9154-c4ed-a882-cc366d5dd235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c97ed00-795e-25e0-a0d3-6328b9991ac6 Address:0c97ed00-795e-25e0-a0d3-6328b9991ac6} {Suffrage:Voter ID:server-e64640f2-9154-c4ed-a882-cc366d5dd235 Address:e64640f2-9154-c4ed-a882-cc366d5dd235}]" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.440Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.440Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: entering follower state: follower="Node at 0c97ed00-795e-25e0-a0d3-6328b9991ac6 [Follower]" leader= testing.go:149: cluster: 2025-02-09T06:52:38.440Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-e64640f2-9154-c4ed-a882-cc366d5dd235: 2025-02-09T06:52:38.440Z [INFO] server-e64640f2-9154-c4ed-a882-cc366d5dd235: entering follower state: follower="Node at e64640f2-9154-c4ed-a882-cc366d5dd235 [Follower]" leader= testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.440Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: entering follower state: follower="Node at 0c97ed00-795e-25e0-a0d3-6328b9991ac6 [Follower]" leader= testing.go:149: server-e64640f2-9154-c4ed-a882-cc366d5dd235: 2025-02-09T06:52:38.440Z [INFO] server-e64640f2-9154-c4ed-a882-cc366d5dd235: entering follower state: follower="Node at e64640f2-9154-c4ed-a882-cc366d5dd235 [Follower]" leader= testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [WARN] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [WARN] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: entering candidate state: node="Node at 0c97ed00-795e-25e0-a0d3-6328b9991ac6 [Candidate]" term=2 testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: entering candidate state: node="Node at 0c97ed00-795e-25e0-a0d3-6328b9991ac6 [Candidate]" term=2 testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: election won: tally=2 testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: election won: tally=2 testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: entering leader state: leader="Node at 0c97ed00-795e-25e0-a0d3-6328b9991ac6 [Leader]" testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: entering leader state: leader="Node at 0c97ed00-795e-25e0-a0d3-6328b9991ac6 [Leader]" testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: added peer, starting replication: peer=server-e64640f2-9154-c4ed-a882-cc366d5dd235 testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.497Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: added peer, starting replication: peer=server-e64640f2-9154-c4ed-a882-cc366d5dd235 testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.498Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: pipelining replication: peer="{Voter server-e64640f2-9154-c4ed-a882-cc366d5dd235 e64640f2-9154-c4ed-a882-cc366d5dd235}" testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.498Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: pipelining replication: peer="{Voter server-e64640f2-9154-c4ed-a882-cc366d5dd235 e64640f2-9154-c4ed-a882-cc366d5dd235}" testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.603Z [INFO] stable state for Follower reached at 2025-02-09 06:52:38.497977027 +0000 UTC m=+19.476785805 (1 nodes), 57.66602ms from start of poll, 57.923086ms from cluster start. Timeout at 2025-02-09 06:52:38.603748504 +0000 UTC m=+19.582557261, 105.771456ms after stability testing.go:149: cluster: 2025-02-09T06:52:38.603Z [INFO] stable state for Follower reached at 2025-02-09 06:52:38.497977027 +0000 UTC m=+19.476785805 (1 nodes), 57.66602ms from start of poll, 57.923086ms from cluster start. Timeout at 2025-02-09 06:52:38.603748504 +0000 UTC m=+19.582557261, 105.771456ms after stability testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.603Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2025-02-09T06:52:38.603Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2025-02-09T06:52:38.709Z [INFO] stable state for Leader reached at 2025-02-09 06:52:38.603907039 +0000 UTC m=+19.582715815 (1 nodes), 474ns from start of poll, 163.853096ms from cluster start. Timeout at 2025-02-09 06:52:38.709503554 +0000 UTC m=+19.688312324, 105.596509ms after stability testing.go:149: cluster: 2025-02-09T06:52:38.709Z [INFO] stable state for Leader reached at 2025-02-09 06:52:38.603907039 +0000 UTC m=+19.582715815 (1 nodes), 474ns from start of poll, 163.853096ms from cluster start. Timeout at 2025-02-09 06:52:38.709503554 +0000 UTC m=+19.688312324, 105.596509ms after stability testLoggerAdapter verbose: server-e64640f2-9154-c4ed-a882-cc366d5dd235: 2025-02-09T06:52:38.720Z [INFO] server-e64640f2-9154-c4ed-a882-cc366d5dd235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c97ed00-795e-25e0-a0d3-6328b9991ac6 Address:0c97ed00-795e-25e0-a0d3-6328b9991ac6} {Suffrage:Voter ID:server-e64640f2-9154-c4ed-a882-cc366d5dd235 Address:e64640f2-9154-c4ed-a882-cc366d5dd235}]" testing.go:149: server-e64640f2-9154-c4ed-a882-cc366d5dd235: 2025-02-09T06:52:38.720Z [INFO] server-e64640f2-9154-c4ed-a882-cc366d5dd235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c97ed00-795e-25e0-a0d3-6328b9991ac6 Address:0c97ed00-795e-25e0-a0d3-6328b9991ac6} {Suffrage:Voter ID:server-e64640f2-9154-c4ed-a882-cc366d5dd235 Address:e64640f2-9154-c4ed-a882-cc366d5dd235}]" testLoggerAdapter verbose: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.720Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: aborting pipeline replication: peer="{Voter server-e64640f2-9154-c4ed-a882-cc366d5dd235 e64640f2-9154-c4ed-a882-cc366d5dd235}" testing.go:149: server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: 2025-02-09T06:52:38.720Z [INFO] server-0c97ed00-795e-25e0-a0d3-6328b9991ac6: aborting pipeline replication: peer="{Voter server-e64640f2-9154-c4ed-a882-cc366d5dd235 e64640f2-9154-c4ed-a882-cc366d5dd235}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 19.713s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_i386.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/849 and its subdirectories I: Current time: Sat Feb 8 18:52:43 -12 2025 I: pbuilder-time-stamp: 1739083963 Mon Jan 8 00:29:46 UTC 2024 I: 1st build successful. Starting 2nd build on remote node ionos12-i386.debian.net. Mon Jan 8 00:29:46 UTC 2024 I: Preparing to do remote build '2' on ionos12-i386.debian.net. Mon Jan 8 00:31:23 UTC 2024 I: Deleting $TMPDIR on ionos12-i386.debian.net. Mon Jan 8 00:31:24 UTC 2024 I: golang-github-hashicorp-raft_1.1.2-1_i386.changes: Format: 1.8 Date: Sun, 19 Jan 2020 18:15:42 +1100 Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Distribution: unstable Urgency: medium Maintainer: Debian Go Packaging Team Changed-By: Dmitry Smirnov Description: golang-github-hashicorp-raft-dev - implementation of the Raft consensus protocol Changes: golang-github-hashicorp-raft (1.1.2-1) unstable; urgency=medium . * New upstream release. Checksums-Sha1: 4fcae6db8ecb88479bbb266ab817605e714c9d9e 90036 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb a14860fbf34d4974719a9e743b5b321923d7ca9e 8274 golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo Checksums-Sha256: 579a17edb3722463f068b5ad1c77fca12707493b919392d4cb32cd828d42473e 90036 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb c09e15e19c7f6f9e729bdfac43b49cb3ce4d8dd3337f5e35c17b9846d1eb6d79 8274 golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo Files: d2b1b6458f45e5e0d928d454aeb4deeb 90036 devel optional golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 02649f889edfd8dad88d602b911d0c5d 8274 devel optional golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo Mon Jan 8 00:31:25 UTC 2024 I: diffoscope 240 will be used to compare the two builds: # Profiling output for: /usr/bin/diffoscope --timeout 7200 --html /srv/reproducible-results/rbuild-debian/r-b-build.F1G7eTIx/golang-github-hashicorp-raft_1.1.2-1.diffoscope.html --text /srv/reproducible-results/rbuild-debian/r-b-build.F1G7eTIx/golang-github-hashicorp-raft_1.1.2-1.diffoscope.txt --json /srv/reproducible-results/rbuild-debian/r-b-build.F1G7eTIx/golang-github-hashicorp-raft_1.1.2-1.diffoscope.json --profile=- /srv/reproducible-results/rbuild-debian/r-b-build.F1G7eTIx/b1/golang-github-hashicorp-raft_1.1.2-1_i386.changes /srv/reproducible-results/rbuild-debian/r-b-build.F1G7eTIx/b2/golang-github-hashicorp-raft_1.1.2-1_i386.changes ## command (total time: 0.000s) 0.000s 1 call cmp (internal) ## has_same_content_as (total time: 0.000s) 0.000s 1 call abc.DotChangesFile ## main (total time: 0.276s) 0.276s 2 calls outputs 0.000s 1 call cleanup ## recognizes (total time: 0.014s) 0.014s 12 calls diffoscope.comparators.binary.FilesystemFile 0.000s 10 calls abc.DotChangesFile ## specialize (total time: 0.000s) 0.000s 1 call specialize Mon Jan 8 00:31:26 UTC 2024 I: diffoscope 240 found no differences in the changes files, and a .buildinfo file also exists. Mon Jan 8 00:31:26 UTC 2024 I: golang-github-hashicorp-raft from bookworm built successfully and reproducibly on i386. Mon Jan 8 00:31:35 UTC 2024 I: Submitting .buildinfo files to external archives: Mon Jan 8 00:31:35 UTC 2024 I: Submitting 12K b1/golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo.asc Mon Jan 8 00:31:36 UTC 2024 I: Submitting 12K b2/golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo.asc Mon Jan 8 00:31:36 UTC 2024 I: Done submitting .buildinfo files to http://buildinfo.debian.net/api/submit. Mon Jan 8 00:31:36 UTC 2024 I: Done submitting .buildinfo files. Mon Jan 8 00:31:36 UTC 2024 I: Removing signed golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo.asc files: removed './b1/golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo.asc' removed './b2/golang-github-hashicorp-raft_1.1.2-1_i386.buildinfo.asc'