Mon May 15 14:01:49 UTC 2023 I: starting to build golang-github-hashicorp-raft/bookworm/arm64 on jenkins on '2023-05-15 14:01' Mon May 15 14:01:49 UTC 2023 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/arm64_1/1437/console.log Mon May 15 14:01:49 UTC 2023 I: Downloading source for bookworm/golang-github-hashicorp-raft=1.1.2-1 --2023-05-15 14:01:50-- http://cdn-fastly.deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.1.2-1.dsc Connecting to 78.137.99.97:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2408 (2.4K) [text/prs.lines.tag] Saving to: ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ 0K .. 100% 236M=0s 2023-05-15 14:01:50 (236 MB/s) - ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ saved [2408/2408] Mon May 15 14:01:50 UTC 2023 I: golang-github-hashicorp-raft_1.1.2-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.4.1 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) Package-List: golang-github-hashicorp-raft-dev deb devel optional arch=all Checksums-Sha1: 6eb88916d5b8568cc5e49dfd778e53f425f02612 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3efa08a30605d234327a7c1293b36ecde63ffbbe 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Checksums-Sha256: 6c3383ee430aa6b421cc4cb4cbc485b0e612bda2e94e88d416d13d65d375dc89 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3877bf052d5719266221189613502262d1be1af152c736d3dc8ed49734ad6dd8 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Files: 65844d5b08afa2d62ea4dedaf43e6047 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 036528358012213daa0588e86be2a3d4 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEULx8+TnSDCcqawZWUra72VOWjRsFAl4kBCwACgkQUra72VOW jRsPQBAAxzZChwaTUFMUIQz9GgdwyR5o2SjXw48JJ//lfygslStaiz8ju52lp33x a1d56obRwTy2QmY2s6+oV3Mq4sQWs7U/1PGnywKeA71Us10A4AAv5d9wMRq1eqzp PcyVdhfM0zFcmJf8Wr93Mi8wbCr9PT+oTxc59+zWDruS7xCoe3aDtaekVfb8QZyj OZPsaTtneSxKjqk+V5Jyx8X6Y6P2cwv89GDd4nxTxapw8H38e2cBk7BsYmXUNXye 6N7dOChDN+x6gqgBU0O97oex+QOpd4LNoFBl2gEi7kukw+jrjrdyKQCANPh0CKx0 K4MPq2BeI3qKJV21l2z7S30BSjduvyktC7efckw5CX4zgT9/kJ5e7ZAf/og+M2vc CoCV+aoAH1OOWbkPu0dsuJiA5z1ZkKGa8ZB5MgvUentk7LVd/HaXW9QDdRTNZT8z Jdh6/wnpSn8ZZEhAixTtqNt64s0sNWmh4krF0WgGNZ1djzYd6gVI1LqlJTaSRKFo nxGsEe+jnC6kv2u8XpLG+xevTKcu4UIFHq2ebpanFvxzqxAv1dWTBDDI13NApMZb Po5huhcdLapUW4NGW/3pkdJkpcY5u26zH6YvTxDICU/tHXDo5/BqI9E1jEj+5m5J XrNqpKLFjW2I9gRcmT5vy3O4edLCPBFs6MGQBK7u1V+nEkh8pJc= =IObb -----END PGP SIGNATURE----- Mon May 15 14:01:50 UTC 2023 I: Checking whether the package is not for us Mon May 15 14:01:50 UTC 2023 I: Starting 1st build on remote node codethink9-arm64.debian.net. Mon May 15 14:01:50 UTC 2023 I: Preparing to do remote build '1' on codethink9-arm64.debian.net. Mon May 15 14:03:55 UTC 2023 I: Deleting $TMPDIR on codethink9-arm64.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Sun Jun 16 08:24:52 -12 2024 I: pbuilder-time-stamp: 1718569492 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bookworm-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: Signature made Sat Jan 18 19:24:28 2020 -12 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.1.2-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/1861/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='arm64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=8' DISTRIBUTION='bookworm' HOME='/var/lib/jenkins' HOST_ARCH='arm64' IFS=' ' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='1861' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/pbuilderrc_VmCd --distribution bookworm --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID='117' SUDO_UID='110' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' USERNAME='root' _='/usr/bin/systemd-run' http_proxy='http://192.168.101.16:3128' I: uname -a Linux codethink9-arm64 4.15.0-210-generic #221-Ubuntu SMP Tue Apr 18 08:32:48 UTC 2023 aarch64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Jun 16 04:48 /bin -> usr/bin I: user script /srv/workspace/pbuilder/1861/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: arm64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19616 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.19-go{a} golang-1.19-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-3-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-diff-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 98 newly installed, 0 to remove and 0 not upgraded. Need to get 112 MB of archives. After unpacking 785 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bookworm/main arm64 sensible-utils all 0.0.17+nmu1 [19.0 kB] Get: 2 http://deb.debian.org/debian bookworm/main arm64 libmagic-mgc arm64 1:5.44-3 [305 kB] Get: 3 http://deb.debian.org/debian bookworm/main arm64 libmagic1 arm64 1:5.44-3 [98.5 kB] Get: 4 http://deb.debian.org/debian bookworm/main arm64 file arm64 1:5.44-3 [42.5 kB] Get: 5 http://deb.debian.org/debian bookworm/main arm64 gettext-base arm64 0.21-12 [159 kB] Get: 6 http://deb.debian.org/debian bookworm/main arm64 libuchardet0 arm64 0.0.7-1 [67.9 kB] Get: 7 http://deb.debian.org/debian bookworm/main arm64 groff-base arm64 1.22.4-10 [861 kB] Get: 8 http://deb.debian.org/debian bookworm/main arm64 bsdextrautils arm64 2.38.1-5+b1 [86.9 kB] Get: 9 http://deb.debian.org/debian bookworm/main arm64 libpipeline1 arm64 1.5.7-1 [36.4 kB] Get: 10 http://deb.debian.org/debian bookworm/main arm64 man-db arm64 2.11.2-2 [1369 kB] Get: 11 http://deb.debian.org/debian bookworm/main arm64 m4 arm64 1.4.19-3 [276 kB] Get: 12 http://deb.debian.org/debian bookworm/main arm64 autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian bookworm/main arm64 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian bookworm/main arm64 automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian bookworm/main arm64 autopoint all 0.21-12 [495 kB] Get: 16 http://deb.debian.org/debian bookworm/main arm64 libdebhelper-perl all 13.11.4 [81.2 kB] Get: 17 http://deb.debian.org/debian bookworm/main arm64 libtool all 2.4.7-5 [517 kB] Get: 18 http://deb.debian.org/debian bookworm/main arm64 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian bookworm/main arm64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian bookworm/main arm64 libsub-override-perl all 0.09-4 [9304 B] Get: 21 http://deb.debian.org/debian bookworm/main arm64 libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB] Get: 22 http://deb.debian.org/debian bookworm/main arm64 dh-strip-nondeterminism all 1.13.1-1 [8620 B] Get: 23 http://deb.debian.org/debian bookworm/main arm64 libelf1 arm64 0.188-2.1 [173 kB] Get: 24 http://deb.debian.org/debian bookworm/main arm64 dwz arm64 0.15-1 [101 kB] Get: 25 http://deb.debian.org/debian bookworm/main arm64 libicu72 arm64 72.1-3 [9204 kB] Get: 26 http://deb.debian.org/debian bookworm/main arm64 libxml2 arm64 2.9.14+dfsg-1.2 [620 kB] Get: 27 http://deb.debian.org/debian bookworm/main arm64 gettext arm64 0.21-12 [1248 kB] Get: 28 http://deb.debian.org/debian bookworm/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian bookworm/main arm64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian bookworm/main arm64 debhelper all 13.11.4 [942 kB] Get: 31 http://deb.debian.org/debian bookworm/main arm64 dh-golang all 1.59 [27.8 kB] Get: 32 http://deb.debian.org/debian bookworm/main arm64 golang-1.19-src all 1.19.8-2 [18.3 MB] Get: 33 http://deb.debian.org/debian bookworm/main arm64 golang-1.19-go arm64 1.19.8-2 [56.6 MB] Get: 34 http://deb.debian.org/debian bookworm/main arm64 golang-src all 2:1.19~1 [5756 B] Get: 35 http://deb.debian.org/debian bookworm/main arm64 golang-go arm64 2:1.19~1 [45.0 kB] Get: 36 http://deb.debian.org/debian bookworm/main arm64 golang-any arm64 2:1.19~1 [5876 B] Get: 37 http://deb.debian.org/debian bookworm/main arm64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 38 http://deb.debian.org/debian bookworm/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-sys-dev all 0.3.0-1 [393 kB] Get: 41 http://deb.debian.org/debian bookworm/main arm64 golang-github-mattn-go-isatty-dev all 0.0.17-1 [6016 B] Get: 42 http://deb.debian.org/debian bookworm/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian bookworm/main arm64 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 44 http://deb.debian.org/debian bookworm/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian bookworm/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian bookworm/main arm64 golang-github-stretchr-objx-dev all 0.5.0-1 [25.5 kB] Get: 47 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian bookworm/main arm64 golang-github-stretchr-testify-dev all 1.8.1-1 [67.9 kB] Get: 49 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian bookworm/main arm64 golang-github-pkg-errors-dev all 0.9.1-2 [13.2 kB] Get: 52 http://deb.debian.org/debian bookworm/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian bookworm/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian bookworm/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.2-2 [8480 B] Get: 56 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 57 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 58 http://deb.debian.org/debian bookworm/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 59 http://deb.debian.org/debian bookworm/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 60 http://deb.debian.org/debian bookworm/main arm64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 61 http://deb.debian.org/debian bookworm/main arm64 golang-github-golang-protobuf-1-3-dev all 1.3.5-4 [234 kB] Get: 62 http://deb.debian.org/debian bookworm/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 63 http://deb.debian.org/debian bookworm/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 64 http://deb.debian.org/debian bookworm/main arm64 golang-github-json-iterator-go-dev all 1.1.12-1 [62.2 kB] Get: 65 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-client-model-dev all 0.3.0-3 [10.5 kB] Get: 66 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 67 http://deb.debian.org/debian bookworm/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 68 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-text-dev all 0.7.0-1 [3863 kB] Get: 69 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-net-dev all 1:0.7.0+dfsg-1 [731 kB] Get: 70 http://deb.debian.org/debian bookworm/main arm64 golang-github-aws-aws-sdk-go-dev all 1.44.133-1 [10.5 MB] Get: 71 http://deb.debian.org/debian bookworm/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 72 http://deb.debian.org/debian bookworm/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 73 http://deb.debian.org/debian bookworm/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 74 http://deb.debian.org/debian bookworm/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 75 http://deb.debian.org/debian bookworm/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 76 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-oauth2-dev all 0.3.0-1 [35.0 kB] Get: 77 http://deb.debian.org/debian bookworm/main arm64 golang-google-protobuf-dev all 1.28.1-3 [542 kB] Get: 78 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 79 http://deb.debian.org/debian bookworm/main arm64 golang-protobuf-extensions-dev all 1.0.1-3 [29.5 kB] Get: 80 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-common-dev all 0.39.0-2 [110 kB] Get: 81 http://deb.debian.org/debian bookworm/main arm64 golang-golang-x-sync-dev all 0.1.0-1 [19.7 kB] Get: 82 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-procfs-dev all 0.8.0-3 [148 kB] Get: 83 http://deb.debian.org/debian bookworm/main arm64 golang-github-prometheus-client-golang-dev all 1.14.0-3 [174 kB] Get: 84 http://deb.debian.org/debian bookworm/main arm64 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 85 http://deb.debian.org/debian bookworm/main arm64 golang-github-creack-pty-dev all 1.1.18-1 [9796 B] Get: 86 http://deb.debian.org/debian bookworm/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 87 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 88 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 89 http://deb.debian.org/debian bookworm/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg-10 [20.8 kB] Get: 90 http://deb.debian.org/debian bookworm/main arm64 libsasl2-2 arm64 2.1.28+dfsg-10 [58.0 kB] Get: 91 http://deb.debian.org/debian bookworm/main arm64 libsasl2-dev arm64 2.1.28+dfsg-10 [230 kB] Get: 92 http://deb.debian.org/debian bookworm/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 93 http://deb.debian.org/debian bookworm/main arm64 golang-github-pkg-diff-dev all 0.0~git20210226.20ebb0f-1 [93.6 kB] Get: 94 http://deb.debian.org/debian bookworm/main arm64 golang-github-rogpeppe-go-internal-dev all 1.9.0-1 [117 kB] Get: 95 http://deb.debian.org/debian bookworm/main arm64 golang-github-kr-pretty-dev all 0.3.0-1 [14.9 kB] Get: 96 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 97 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 98 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 112 MB in 4s (26.0 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19616 files and directories currently installed.) Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ... Unpacking sensible-utils (0.0.17+nmu1) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../01-libmagic-mgc_1%3a5.44-3_arm64.deb ... Unpacking libmagic-mgc (1:5.44-3) ... Selecting previously unselected package libmagic1:arm64. Preparing to unpack .../02-libmagic1_1%3a5.44-3_arm64.deb ... Unpacking libmagic1:arm64 (1:5.44-3) ... Selecting previously unselected package file. Preparing to unpack .../03-file_1%3a5.44-3_arm64.deb ... Unpacking file (1:5.44-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../04-gettext-base_0.21-12_arm64.deb ... Unpacking gettext-base (0.21-12) ... Selecting previously unselected package libuchardet0:arm64. Preparing to unpack .../05-libuchardet0_0.0.7-1_arm64.deb ... Unpacking libuchardet0:arm64 (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../06-groff-base_1.22.4-10_arm64.deb ... Unpacking groff-base (1.22.4-10) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../07-bsdextrautils_2.38.1-5+b1_arm64.deb ... Unpacking bsdextrautils (2.38.1-5+b1) ... Selecting previously unselected package libpipeline1:arm64. Preparing to unpack .../08-libpipeline1_1.5.7-1_arm64.deb ... Unpacking libpipeline1:arm64 (1.5.7-1) ... Selecting previously unselected package man-db. Preparing to unpack .../09-man-db_2.11.2-2_arm64.deb ... Unpacking man-db (2.11.2-2) ... Selecting previously unselected package m4. Preparing to unpack .../10-m4_1.4.19-3_arm64.deb ... Unpacking m4 (1.4.19-3) ... Selecting previously unselected package autoconf. Preparing to unpack .../11-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../12-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../13-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../14-autopoint_0.21-12_all.deb ... Unpacking autopoint (0.21-12) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../15-libdebhelper-perl_13.11.4_all.deb ... Unpacking libdebhelper-perl (13.11.4) ... Selecting previously unselected package libtool. Preparing to unpack .../16-libtool_2.4.7-5_all.deb ... Unpacking libtool (2.4.7-5) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../17-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../18-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../19-libsub-override-perl_0.09-4_all.deb ... Unpacking libsub-override-perl (0.09-4) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../20-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../21-dh-strip-nondeterminism_1.13.1-1_all.deb ... Unpacking dh-strip-nondeterminism (1.13.1-1) ... Selecting previously unselected package libelf1:arm64. Preparing to unpack .../22-libelf1_0.188-2.1_arm64.deb ... Unpacking libelf1:arm64 (0.188-2.1) ... Selecting previously unselected package dwz. Preparing to unpack .../23-dwz_0.15-1_arm64.deb ... Unpacking dwz (0.15-1) ... Selecting previously unselected package libicu72:arm64. Preparing to unpack .../24-libicu72_72.1-3_arm64.deb ... Unpacking libicu72:arm64 (72.1-3) ... Selecting previously unselected package libxml2:arm64. Preparing to unpack .../25-libxml2_2.9.14+dfsg-1.2_arm64.deb ... Unpacking libxml2:arm64 (2.9.14+dfsg-1.2) ... Selecting previously unselected package gettext. Preparing to unpack .../26-gettext_0.21-12_arm64.deb ... Unpacking gettext (0.21-12) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../27-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../28-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../29-debhelper_13.11.4_all.deb ... Unpacking debhelper (13.11.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../30-dh-golang_1.59_all.deb ... Unpacking dh-golang (1.59) ... Selecting previously unselected package golang-1.19-src. Preparing to unpack .../31-golang-1.19-src_1.19.8-2_all.deb ... Unpacking golang-1.19-src (1.19.8-2) ... Selecting previously unselected package golang-1.19-go. Preparing to unpack .../32-golang-1.19-go_1.19.8-2_arm64.deb ... Unpacking golang-1.19-go (1.19.8-2) ... Selecting previously unselected package golang-src. Preparing to unpack .../33-golang-src_2%3a1.19~1_all.deb ... Unpacking golang-src (2:1.19~1) ... Selecting previously unselected package golang-go:arm64. Preparing to unpack .../34-golang-go_2%3a1.19~1_arm64.deb ... Unpacking golang-go:arm64 (2:1.19~1) ... Selecting previously unselected package golang-any:arm64. Preparing to unpack .../35-golang-any_2%3a1.19~1_arm64.deb ... Unpacking golang-any:arm64 (2:1.19~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../36-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../37-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../38-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../39-golang-golang-x-sys-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.3.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../40-golang-github-mattn-go-isatty-dev_0.0.17-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.17-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../41-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../42-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../43-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../44-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../45-golang-github-stretchr-objx-dev_0.5.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../46-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../47-golang-github-stretchr-testify-dev_1.8.1-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.8.1-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../48-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../49-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../50-golang-github-pkg-errors-dev_0.9.1-2_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-2) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../51-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../52-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../53-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../54-golang-github-hashicorp-go-uuid-dev_1.0.2-2_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../55-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../56-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../57-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../58-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../59-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-golang-protobuf-1-3-dev. Preparing to unpack .../60-golang-github-golang-protobuf-1-3-dev_1.3.5-4_all.deb ... Unpacking golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../61-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../62-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../63-golang-github-json-iterator-go-dev_1.1.12-1_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-1) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../64-golang-github-prometheus-client-model-dev_0.3.0-3_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.3.0-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../65-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../66-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../67-golang-golang-x-text-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.7.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../68-golang-golang-x-net-dev_1%3a0.7.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../69-golang-github-aws-aws-sdk-go-dev_1.44.133-1_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../70-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../71-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../72-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../73-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../74-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../75-golang-golang-x-oauth2-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.3.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../76-golang-google-protobuf-dev_1.28.1-3_all.deb ... Unpacking golang-google-protobuf-dev (1.28.1-3) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../77-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../78-golang-protobuf-extensions-dev_1.0.1-3_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-3) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../79-golang-github-prometheus-common-dev_0.39.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.39.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../80-golang-golang-x-sync-dev_0.1.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.1.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../81-golang-github-prometheus-procfs-dev_0.8.0-3_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.8.0-3) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../82-golang-github-prometheus-client-golang-dev_1.14.0-3_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.14.0-3) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../83-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../84-golang-github-creack-pty-dev_1.1.18-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.18-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../85-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../86-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../87-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:arm64. Preparing to unpack .../88-libsasl2-modules-db_2.1.28+dfsg-10_arm64.deb ... Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-2:arm64. Preparing to unpack .../89-libsasl2-2_2.1.28+dfsg-10_arm64.deb ... Unpacking libsasl2-2:arm64 (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../90-libsasl2-dev_2.1.28+dfsg-10_arm64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg-10) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../91-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-github-pkg-diff-dev. Preparing to unpack .../92-golang-github-pkg-diff-dev_0.0~git20210226.20ebb0f-1_all.deb ... Unpacking golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../93-golang-github-rogpeppe-go-internal-dev_1.9.0-1_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../94-golang-github-kr-pretty-dev_0.3.0-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../95-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../96-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../97-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:arm64 (1.5.7-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Setting up libicu72:arm64 (72.1-3) ... Setting up bsdextrautils (2.38.1-5+b1) ... Setting up golang-github-creack-pty-dev (1.1.18-1) ... Setting up golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Setting up golang-github-stretchr-objx-dev (0.5.0-1) ... Setting up libmagic-mgc (1:5.44-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-2) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.11.4) ... Setting up golang-golang-x-sys-dev (0.3.0-1) ... Setting up libmagic1:arm64 (1:5.44-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-12) ... Setting up golang-1.19-src (1.19.8-2) ... Setting up m4 (1.4.19-3) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Setting up file (1:5.44-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg-10) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.28.1-3) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.8.1-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-golang-x-sync-dev (0.1.0-1) ... Setting up autopoint (0.21-12) ... Setting up libsasl2-2:arm64 (2.1.28+dfsg-10) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up sensible-utils (0.0.17+nmu1) ... Setting up golang-golang-x-text-dev (0.7.0-1) ... Setting up libuchardet0:arm64 (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.8.0-3) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up libsub-override-perl (0.09-4) ... Setting up golang-github-prometheus-client-model-dev (0.3.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Setting up golang-src (2:1.19~1) ... Setting up libelf1:arm64 (0.188-2.1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:arm64 (2.9.14+dfsg-1.2) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up golang-1.19-go (1.19.8-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-github-kr-pretty-dev (0.3.0-1) ... Setting up libfile-stripnondeterminism-perl (1.13.1-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.17-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up gettext (0.21-12) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-5) ... Setting up golang-go:arm64 (2:1.19~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-protobuf-extensions-dev (1.0.1-3) ... Setting up golang-github-json-iterator-go-dev (1.1.12-1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:arm64 (2:1.19~1) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg-10) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.13.1-1) ... Setting up dwz (0.15-1) ... Setting up groff-base (1.22.4-10) ... Setting up golang-golang-x-oauth2-dev (0.3.0-1) ... Setting up golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up man-db (2.11.2-2) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.11.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.59) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.39.0-2) ... Setting up golang-github-prometheus-client-golang-dev (1.14.0-3) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Processing triggers for libc-bin (2.36-9) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.31-1.2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture arm64 fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader internal/goexperiment internal/goos internal/goarch internal/race runtime/internal/syscall sync/atomic internal/cpu unicode/utf8 unicode internal/abi runtime/internal/math runtime/internal/sys container/list internal/itoa math/bits crypto/internal/subtle crypto/subtle crypto/internal/boring/sig internal/bytealg runtime/internal/atomic encoding unicode/utf16 internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle math runtime internal/reflectlite sync internal/sysinfo internal/singleflight internal/testlog math/rand runtime/cgo errors sort internal/oserror internal/safefilepath path github.com/hashicorp/golang-lru/simplelru io strconv syscall vendor/golang.org/x/net/dns/dnsmessage crypto/internal/nistec/fiat strings hash bytes crypto/internal/randutil hash/crc32 crypto reflect crypto/rc4 vendor/golang.org/x/text/transform hash/crc64 bufio github.com/hashicorp/go-immutable-radix net/http/internal/ascii regexp/syntax html time internal/syscall/unix internal/syscall/execenv regexp context internal/poll io/fs embed internal/fmtsort encoding/binary os encoding/base64 crypto/md5 vendor/golang.org/x/crypto/curve25519/internal/field crypto/cipher vendor/golang.org/x/crypto/internal/poly1305 crypto/internal/edwards25519/field crypto/internal/nistec encoding/pem golang.org/x/sys/unix crypto/des crypto/internal/boring vendor/golang.org/x/crypto/chacha20 crypto/internal/edwards25519 internal/godebug fmt crypto/aes internal/intern crypto/sha512 net/netip crypto/hmac crypto/sha1 crypto/sha256 path/filepath vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf os/signal io/ioutil net log encoding/json encoding/hex compress/flate net/url math/big vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/http2/hpack compress/gzip mime mime/quotedprintable net/http/internal vendor/golang.org/x/text/secure/bidirule github.com/mattn/go-isatty encoding/gob go/token github.com/mattn/go-colorable text/template/parse github.com/fatih/color flag runtime/debug github.com/hashicorp/go-hclog crypto/rand crypto/elliptic crypto/internal/boring/bbig encoding/asn1 crypto/ed25519 crypto/rsa crypto/dsa vendor/golang.org/x/net/idna runtime/trace testing text/template crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa html/template vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-aarch64-linux-gnu && go test -vet=off -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2024-06-16T08:26:22.998-1200 [INFO] creating new snapshot: path=/tmp/raft2427954899/raft193094651/snapshots/3-10-1718569582998.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2024-06-16T08:26:22.999-1200 [INFO] creating new snapshot: path=/tmp/raft3117812912/snapshots/3-10-1718569582999.tmp testing.go:153: 2024-06-16T08:26:23.000-1200 [WARN] found temporary snapshot: name=3-10-1718569582999.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2024-06-16T08:26:23.001-1200 [INFO] creating new snapshot: path=/tmp/raft3896822289/snapshots/3-10-1718569583001.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention testing.go:153: 2024-06-16T08:26:23.001-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-10-1718569583001.tmp testing.go:153: 2024-06-16T08:26:23.002-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-11-1718569583002.tmp testing.go:153: 2024-06-16T08:26:23.002-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-12-1718569583002.tmp testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] reaping snapshot: path=/tmp/raft3216926845/snapshots/3-10-1718569583001 testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-13-1718569583003.tmp testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] reaping snapshot: path=/tmp/raft3216926845/snapshots/3-11-1718569583002 testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-14-1718569583003.tmp testing.go:153: 2024-06-16T08:26:23.004-1200 [INFO] reaping snapshot: path=/tmp/raft3216926845/snapshots/3-12-1718569583002 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2024-06-16T08:26:23.005-1200 [INFO] creating new snapshot: path=/tmp/raft179444173/snapshots/5-130350-1718569583005.tmp testing.go:153: 2024-06-16T08:26:23.006-1200 [INFO] creating new snapshot: path=/tmp/raft179444173/snapshots/36-204917-1718569583006.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.028-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.033-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.044-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.064-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.104-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.185-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.345-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.666-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:24.307-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:25.311-1200 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-16T08:26:26.315-1200 [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-4cf87f40-e7a2-7d13-6257-4047987d2979: 2024-06-16T08:26:27.032-1200 [INFO] server-4cf87f40-e7a2-7d13-6257-4047987d2979: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cf87f40-e7a2-7d13-6257-4047987d2979 Address:4cf87f40-e7a2-7d13-6257-4047987d2979}]" testing.go:149: server-4cf87f40-e7a2-7d13-6257-4047987d2979: 2024-06-16T08:26:27.032-1200 [INFO] server-4cf87f40-e7a2-7d13-6257-4047987d2979: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cf87f40-e7a2-7d13-6257-4047987d2979 Address:4cf87f40-e7a2-7d13-6257-4047987d2979}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-4ecc849f-4a26-e961-aa02-7169fdf8546c: 2024-06-16T08:26:27.033-1200 [INFO] server-4ecc849f-4a26-e961-aa02-7169fdf8546c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ecc849f-4a26-e961-aa02-7169fdf8546c Address:4ecc849f-4a26-e961-aa02-7169fdf8546c}]" testing.go:149: server-4ecc849f-4a26-e961-aa02-7169fdf8546c: 2024-06-16T08:26:27.033-1200 [INFO] server-4ecc849f-4a26-e961-aa02-7169fdf8546c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ecc849f-4a26-e961-aa02-7169fdf8546c Address:4ecc849f-4a26-e961-aa02-7169fdf8546c}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: initial configuration: index=0 servers=[] testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: initial configuration: index=0 servers=[] testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering follower state: follower="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Follower]" leader= testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: entering follower state: follower="Node at 225d14b1-8665-f8dd-15e9-0cf4e9946d71 [Follower]" leader= testLoggerAdapter verbose: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: initial configuration: index=0 servers=[] testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering follower state: follower="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Follower]" leader= testing.go:149: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: entering follower state: follower="Node at b38698c3-4377-2a9b-1d2d-68dfdf988e0a [Follower]" leader= testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: entering follower state: follower="Node at 225d14b1-8665-f8dd-15e9-0cf4e9946d71 [Follower]" leader= testing.go:149: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: entering follower state: follower="Node at b38698c3-4377-2a9b-1d2d-68dfdf988e0a [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.034-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:27.034-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.090-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: no known peers, aborting election testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.090-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: no known peers, aborting election testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering candidate state: node="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Candidate]" term=2 testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering candidate state: node="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Candidate]" term=2 testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: election won: tally=2 testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: election won: tally=2 testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering leader state: leader="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Leader]" testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering leader state: leader="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Leader]" testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.093-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.093-1200 [WARN] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.093-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.093-1200 [WARN] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" next=1 testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" next=1 testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" next=1 testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" next=1 testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.199-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.09363038 -1200 -12 m=+4.100946861 (2 nodes), 58.8971ms from start of poll, 59.5852ms from cluster start. Timeout at 2024-06-16 08:26:27.19915354 -1200 -12 m=+4.206470021, 105.52316ms after stability testing.go:149: cluster: 2024-06-16T08:26:27.199-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.09363038 -1200 -12 m=+4.100946861 (2 nodes), 58.8971ms from start of poll, 59.5852ms from cluster start. Timeout at 2024-06-16 08:26:27.19915354 -1200 -12 m=+4.206470021, 105.52316ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.199-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:27.199-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.305-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.19941434 -1200 -12 m=+4.206730801 (1 nodes), 1.56µs from start of poll, 165.36914ms from cluster start. Timeout at 2024-06-16 08:26:27.30525452 -1200 -12 m=+4.312571001, 105.8402ms after stability testing.go:149: cluster: 2024-06-16T08:26:27.305-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.19941434 -1200 -12 m=+4.206730801 (1 nodes), 1.56µs from start of poll, 165.36914ms from cluster start. Timeout at 2024-06-16 08:26:27.30525452 -1200 -12 m=+4.312571001, 105.8402ms after stability testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" --- PASS: TestRaft_LiveBootstrap (0.28s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: initial configuration: index=0 servers=[] testing.go:149: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: entering follower state: follower="Node at 49d40753-db4d-ac6a-8b5a-e434493f3691 [Follower]" leader= testing.go:149: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: entering follower state: follower="Node at 49d40753-db4d-ac6a-8b5a-e434493f3691 [Follower]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering follower state: follower="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Follower]" leader= testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering follower state: follower="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Follower]" leader= testLoggerAdapter verbose: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" testing.go:149: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" testLoggerAdapter verbose: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: entering follower state: follower="Node at 39203453-104e-4506-2072-0b11fde5e926 [Follower]" leader= testing.go:149: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: entering follower state: follower="Node at 39203453-104e-4506-2072-0b11fde5e926 [Follower]" leader= testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.319-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: initial configuration: index=0 servers=[] testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.319-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.320-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: entering follower state: follower="Node at 3681658f-1671-150f-ad54-42c5b997b434 [Follower]" leader= testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.320-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: entering follower state: follower="Node at 3681658f-1671-150f-ad54-42c5b997b434 [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.320-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:27.320-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.378-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: no known peers, aborting election testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.378-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: no known peers, aborting election testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering candidate state: node="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Candidate]" term=2 testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering candidate state: node="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Candidate]" term=2 testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: election won: tally=2 testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: election won: tally=2 testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering leader state: leader="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Leader]" testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering leader state: leader="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Leader]" testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-39203453-104e-4506-2072-0b11fde5e926 testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-39203453-104e-4506-2072-0b11fde5e926 testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.492-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.38632488 -1200 -12 m=+4.393641361 (1 nodes), 65.99142ms from start of poll, 68.17602ms from cluster start. Timeout at 2024-06-16 08:26:27.49223536 -1200 -12 m=+4.499551841, 105.91048ms after stability testing.go:149: cluster: 2024-06-16T08:26:27.492-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.38632488 -1200 -12 m=+4.393641361 (1 nodes), 65.99142ms from start of poll, 68.17602ms from cluster start. Timeout at 2024-06-16 08:26:27.49223536 -1200 -12 m=+4.499551841, 105.91048ms after stability testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: updating configuration: command=AddStaging server-id=server-3681658f-1671-150f-ad54-42c5b997b434 server-addr=3681658f-1671-150f-ad54-42c5b997b434 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926} {Suffrage:Voter ID:server-3681658f-1671-150f-ad54-42c5b997b434 Address:3681658f-1671-150f-ad54-42c5b997b434}]" testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: updating configuration: command=AddStaging server-id=server-3681658f-1671-150f-ad54-42c5b997b434 server-addr=3681658f-1671-150f-ad54-42c5b997b434 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926} {Suffrage:Voter ID:server-3681658f-1671-150f-ad54-42c5b997b434 Address:3681658f-1671-150f-ad54-42c5b997b434}]" testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-3681658f-1671-150f-ad54-42c5b997b434 testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-3681658f-1671-150f-ad54-42c5b997b434 testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.492-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.492-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: appendEntries rejected, sending older logs: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" next=1 testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: appendEntries rejected, sending older logs: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" next=1 testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.493-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.493-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.498-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:27.498-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.604-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.49873366 -1200 -12 m=+4.506050141 (1 nodes), 1.36µs from start of poll, 180.5848ms from cluster start. Timeout at 2024-06-16 08:26:27.6047072 -1200 -12 m=+4.612023681, 105.97354ms after stability testing.go:149: cluster: 2024-06-16T08:26:27.604-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.49873366 -1200 -12 m=+4.506050141 (1 nodes), 1.36µs from start of poll, 180.5848ms from cluster start. Timeout at 2024-06-16 08:26:27.6047072 -1200 -12 m=+4.612023681, 105.97354ms after stability testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" --- PASS: TestRaft_HasExistingState (0.29s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3420e22-7732-1e08-338b-a49595ae2e5f Address:e3420e22-7732-1e08-338b-a49595ae2e5f}]" testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3420e22-7732-1e08-338b-a49595ae2e5f Address:e3420e22-7732-1e08-338b-a49595ae2e5f}]" testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering follower state: follower="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Follower]" leader= testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering follower state: follower="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Follower]" leader= testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [WARN] server-e3420e22-7732-1e08-338b-a49595ae2e5f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [WARN] server-e3420e22-7732-1e08-338b-a49595ae2e5f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering candidate state: node="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Candidate]" term=2 testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering candidate state: node="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Candidate]" term=2 testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: election won: tally=1 testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: election won: tally=1 testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering leader state: leader="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Leader]" testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering leader state: leader="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Leader]" --- PASS: TestRaft_SingleNode (0.07s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" testing.go:149: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" testLoggerAdapter verbose: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: entering follower state: follower="Node at 3a9ba939-cc84-618a-6a50-b9f80f122c65 [Follower]" leader= testing.go:149: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: entering follower state: follower="Node at 3a9ba939-cc84-618a-6a50-b9f80f122c65 [Follower]" leader= testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering follower state: follower="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Follower]" leader= testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering follower state: follower="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Follower]" leader= testLoggerAdapter verbose: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" testing.go:149: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.675-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:27.675-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: entering follower state: follower="Node at 7b68fe81-8f04-e1e0-66fd-6430782d3d4d [Follower]" leader= testing.go:149: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: entering follower state: follower="Node at 7b68fe81-8f04-e1e0-66fd-6430782d3d4d [Follower]" leader= testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.745-1200 [WARN] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.745-1200 [WARN] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering candidate state: node="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Candidate]" term=2 testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering candidate state: node="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Candidate]" term=2 testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: election won: tally=2 testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: election won: tally=2 testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering leader state: leader="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Leader]" testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering leader state: leader="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Leader]" testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-3a9ba939-cc84-618a-6a50-b9f80f122c65 testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-3a9ba939-cc84-618a-6a50-b9f80f122c65 testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.851-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.74642138 -1200 -12 m=+4.753737981 (2 nodes), 70.5818ms from start of poll, 71.43196ms from cluster start. Timeout at 2024-06-16 08:26:27.85147534 -1200 -12 m=+4.858791841, 105.05386ms after stability testing.go:149: cluster: 2024-06-16T08:26:27.851-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.74642138 -1200 -12 m=+4.753737981 (2 nodes), 70.5818ms from start of poll, 71.43196ms from cluster start. Timeout at 2024-06-16 08:26:27.85147534 -1200 -12 m=+4.858791841, 105.05386ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.851-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:27.851-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.957-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.85172458 -1200 -12 m=+4.859041061 (1 nodes), 2µs from start of poll, 176.73504ms from cluster start. Timeout at 2024-06-16 08:26:27.95747648 -1200 -12 m=+4.964792941, 105.75188ms after stability testing.go:149: cluster: 2024-06-16T08:26:27.957-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.85172458 -1200 -12 m=+4.859041061 (1 nodes), 2µs from start of poll, 176.73504ms from cluster start. Timeout at 2024-06-16 08:26:27.95747648 -1200 -12 m=+4.964792941, 105.75188ms after stability testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" --- PASS: TestRaft_TripleNode (0.30s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering follower state: follower="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Follower]" leader= testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering follower state: follower="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Follower]" leader= testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.975-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:27.975-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=2 testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=2 testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: election won: tally=2 testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: election won: tally=2 testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering leader state: leader="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Leader]" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering leader state: leader="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Leader]" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.142-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.03693486 -1200 -12 m=+5.044251361 (2 nodes), 61.01946ms from start of poll, 61.93708ms from cluster start. Timeout at 2024-06-16 08:26:28.14278572 -1200 -12 m=+5.150102201, 105.85084ms after stability testing.go:149: cluster: 2024-06-16T08:26:28.142-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.03693486 -1200 -12 m=+5.044251361 (2 nodes), 61.01946ms from start of poll, 61.93708ms from cluster start. Timeout at 2024-06-16 08:26:28.14278572 -1200 -12 m=+5.150102201, 105.85084ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.142-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:28.142-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.248-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.14299162 -1200 -12 m=+5.150308081 (1 nodes), 2.24µs from start of poll, 167.9938ms from cluster start. Timeout at 2024-06-16 08:26:28.24832532 -1200 -12 m=+5.255641821, 105.33374ms after stability testing.go:149: cluster: 2024-06-16T08:26:28.248-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.14299162 -1200 -12 m=+5.150308081 (1 nodes), 2.24µs from start of poll, 167.9938ms from cluster start. Timeout at 2024-06-16 08:26:28.24832532 -1200 -12 m=+5.255641821, 105.33374ms after stability raft_test.go:336: [INFO] Disconnecting Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Leader] testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.264-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.264-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.265-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:28.265-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="append pipeline closed" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="append pipeline closed" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="append pipeline closed" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="append pipeline closed" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.281-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.281-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.286-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.286-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.296-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.296-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.297-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.297-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.301-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.301-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.307-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.307-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c time=50.12326ms testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c time=50.12326ms testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-ee94f1af-0538-dd75-0868-3267cffc729b time=50.14058ms testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-ee94f1af-0538-dd75-0868-3267cffc729b time=50.14058ms testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact quorum of nodes, stepping down testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [WARN] server-ee94f1af-0538-dd75-0868-3267cffc729b: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [WARN] server-ee94f1af-0538-dd75-0868-3267cffc729b: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering candidate state: node="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Candidate]" term=3 testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering candidate state: node="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Candidate]" term=3 testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.309-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: rejecting vote request since we have a leader: from=ee94f1af-0538-dd75-0868-3267cffc729b leader=6085a72a-2d6f-0e11-9b40-c701837dc407 testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.309-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: rejecting vote request since we have a leader: from=ee94f1af-0538-dd75-0868-3267cffc729b leader=6085a72a-2d6f-0e11-9b40-c701837dc407 testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [ERROR] server-ee94f1af-0538-dd75-0868-3267cffc729b: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [ERROR] server-ee94f1af-0538-dd75-0868-3267cffc729b: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=3 testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=3 testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.342-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: duplicate requestVote for same term: term=3 testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.342-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: Election timeout reached, restarting election testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: Election timeout reached, restarting election testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=4 testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=4 testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.396-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.396-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: election won: tally=2 testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: election won: tally=2 testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering leader state: leader="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Leader]" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering leader state: leader="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Leader]" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-6085a72a-2d6f-0e11-9b40-c701837dc407 testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-6085a72a-2d6f-0e11-9b40-c701837dc407 testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=3 testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=3 testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.406-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.406-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.417-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.417-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.423-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.423-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.436-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.436-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.443-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.443-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.447-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=50.84708ms testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.447-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=50.84708ms testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.466-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.466-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.473-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.473-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.477-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: Election timeout reached, restarting election testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.477-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: Election timeout reached, restarting election testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=4 testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=4 testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.497-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=100.7616ms testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.497-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=100.7616ms testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.508-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.40253032 -1200 -12 m=+5.409846781 (1 nodes), 137.0046ms from start of poll, 427.5325ms from cluster start. Timeout at 2024-06-16 08:26:28.50796646 -1200 -12 m=+5.515282981, 105.4362ms after stability testing.go:149: cluster: 2024-06-16T08:26:28.508-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.40253032 -1200 -12 m=+5.409846781 (1 nodes), 137.0046ms from start of poll, 427.5325ms from cluster start. Timeout at 2024-06-16 08:26:28.50796646 -1200 -12 m=+5.515282981, 105.4362ms after stability raft_test.go:371: [INFO] Reconnecting Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate] testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.515-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader=d7fde817-1e05-8df2-4277-e2c3b1ce543c testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.515-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader=d7fde817-1e05-8df2-4277-e2c3b1ce543c testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.515-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.515-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.520-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.520-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.521-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.521-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" --- PASS: TestRaft_LeaderFail (0.55s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.522-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" testing.go:149: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.522-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.522-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.522-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.523-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:28.523-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.523-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: entering follower state: follower="Node at d934f546-fac8-ac51-ea1e-ac58446887da [Follower]" leader= testing.go:149: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.523-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: entering follower state: follower="Node at d934f546-fac8-ac51-ea1e-ac58446887da [Follower]" leader= testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.523-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader= testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.523-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader= testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering follower state: follower="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Follower]" leader= testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering follower state: follower="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Follower]" leader= testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering candidate state: node="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Candidate]" term=2 testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering candidate state: node="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Candidate]" term=2 testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [WARN] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [WARN] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering candidate state: node="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Candidate]" term=2 testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering candidate state: node="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Candidate]" term=2 testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: duplicate requestVote for same term: term=2 testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: election won: tally=2 testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: election won: tally=2 testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering leader state: leader="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Leader]" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering leader state: leader="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Leader]" testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-d934f546-fac8-ac51-ea1e-ac58446887da testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-d934f546-fac8-ac51-ea1e-ac58446887da testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-03150ad5-4bb9-cb55-53ca-a6546c897d87 testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-03150ad5-4bb9-cb55-53ca-a6546c897d87 testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader=64bb9548-637e-d65a-be35-d88cd71db743 testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader=64bb9548-637e-d65a-be35-d88cd71db743 testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" testLoggerAdapter verbose: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.577-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: duplicate requestVote for same term: term=2 testing.go:149: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.577-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: rejecting vote request since we have a leader: from=03150ad5-4bb9-cb55-53ca-a6546c897d87 leader=64bb9548-637e-d65a-be35-d88cd71db743 testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: rejecting vote request since we have a leader: from=03150ad5-4bb9-cb55-53ca-a6546c897d87 leader=64bb9548-637e-d65a-be35-d88cd71db743 testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.682-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.57750126 -1200 -12 m=+5.584817721 (1 nodes), 54.2845ms from start of poll, 55.22396ms from cluster start. Timeout at 2024-06-16 08:26:28.68289086 -1200 -12 m=+5.690207321, 105.3896ms after stability testing.go:149: cluster: 2024-06-16T08:26:28.682-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.57750126 -1200 -12 m=+5.584817721 (1 nodes), 54.2845ms from start of poll, 55.22396ms from cluster start. Timeout at 2024-06-16 08:26:28.68289086 -1200 -12 m=+5.690207321, 105.3896ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.683-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:28.683-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.788-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.68313366 -1200 -12 m=+5.690450121 (2 nodes), 2.94µs from start of poll, 160.85636ms from cluster start. Timeout at 2024-06-16 08:26:28.78845418 -1200 -12 m=+5.795770661, 105.32054ms after stability testing.go:149: cluster: 2024-06-16T08:26:28.788-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.68313366 -1200 -12 m=+5.690450121 (2 nodes), 2.94µs from start of poll, 160.85636ms from cluster start. Timeout at 2024-06-16 08:26:28.78845418 -1200 -12 m=+5.795770661, 105.32054ms after stability testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to pipeline appendEntries: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="append pipeline closed" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to pipeline appendEntries: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="append pipeline closed" testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to appendEntries to: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="failed to connect to peer: d934f546-fac8-ac51-ea1e-ac58446887da" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to appendEntries to: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="failed to connect to peer: d934f546-fac8-ac51-ea1e-ac58446887da" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.799-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.799-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.813-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:28.813-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.919-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.81329822 -1200 -12 m=+5.820614681 (1 nodes), 1.58µs from start of poll, 291.02092ms from cluster start. Timeout at 2024-06-16 08:26:28.91920956 -1200 -12 m=+5.926526061, 105.91138ms after stability testing.go:149: cluster: 2024-06-16T08:26:28.919-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.81329822 -1200 -12 m=+5.820614681 (1 nodes), 1.58µs from start of poll, 291.02092ms from cluster start. Timeout at 2024-06-16 08:26:28.91920956 -1200 -12 m=+5.926526061, 105.91138ms after stability testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.920-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" testing.go:149: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.920-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" testLoggerAdapter verbose: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" testLoggerAdapter verbose: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.921-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: entering follower state: follower="Node at 11d515dd-acc8-771d-addc-4dfd5981f34a [Follower]" leader= testing.go:149: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.921-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering follower state: follower="Node at f0710021-f920-40c9-ab9c-55569a00223c [Follower]" leader= testing.go:149: cluster: 2024-06-16T08:26:28.921-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.921-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: entering follower state: follower="Node at 11d515dd-acc8-771d-addc-4dfd5981f34a [Follower]" leader= testLoggerAdapter verbose: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: entering follower state: follower="Node at 6fdbcaa1-dffd-eddf-6dad-578847450335 [Follower]" leader= testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering follower state: follower="Node at f0710021-f920-40c9-ab9c-55569a00223c [Follower]" leader= testing.go:149: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: entering follower state: follower="Node at 6fdbcaa1-dffd-eddf-6dad-578847450335 [Follower]" leader= testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [WARN] server-f0710021-f920-40c9-ab9c-55569a00223c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [WARN] server-f0710021-f920-40c9-ab9c-55569a00223c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering candidate state: node="Node at f0710021-f920-40c9-ab9c-55569a00223c [Candidate]" term=2 testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering candidate state: node="Node at f0710021-f920-40c9-ab9c-55569a00223c [Candidate]" term=2 testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: election won: tally=2 testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: election won: tally=2 testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering leader state: leader="Node at f0710021-f920-40c9-ab9c-55569a00223c [Leader]" testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering leader state: leader="Node at f0710021-f920-40c9-ab9c-55569a00223c [Leader]" testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-11d515dd-acc8-771d-addc-4dfd5981f34a testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-11d515dd-acc8-771d-addc-4dfd5981f34a testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-6fdbcaa1-dffd-eddf-6dad-578847450335 testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-6fdbcaa1-dffd-eddf-6dad-578847450335 testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.085-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.98045696 -1200 -12 m=+5.987773441 (1 nodes), 58.93466ms from start of poll, 59.83936ms from cluster start. Timeout at 2024-06-16 08:26:29.08585896 -1200 -12 m=+6.093175461, 105.40202ms after stability testing.go:149: cluster: 2024-06-16T08:26:29.085-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.98045696 -1200 -12 m=+5.987773441 (1 nodes), 58.93466ms from start of poll, 59.83936ms from cluster start. Timeout at 2024-06-16 08:26:29.08585896 -1200 -12 m=+6.093175461, 105.40202ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.086-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:29.086-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.192-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:29.0860566 -1200 -12 m=+6.093373081 (2 nodes), 2.5µs from start of poll, 165.439ms from cluster start. Timeout at 2024-06-16 08:26:29.19207548 -1200 -12 m=+6.199391961, 106.01888ms after stability testing.go:149: cluster: 2024-06-16T08:26:29.192-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:29.0860566 -1200 -12 m=+6.093373081 (2 nodes), 2.5µs from start of poll, 165.439ms from cluster start. Timeout at 2024-06-16 08:26:29.19207548 -1200 -12 m=+6.199391961, 106.01888ms after stability testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" --- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering follower state: follower="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Follower]" leader= testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering follower state: follower="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Follower]" leader= testLoggerAdapter verbose: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.193-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" testing.go:149: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.193-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" testLoggerAdapter verbose: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.194-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: entering follower state: follower="Node at c2bc3a84-64d4-1ea9-6937-06935dd1a27e [Follower]" leader= testing.go:149: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.194-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: entering follower state: follower="Node at c2bc3a84-64d4-1ea9-6937-06935dd1a27e [Follower]" leader= testLoggerAdapter verbose: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" testing.go:149: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.194-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:29.194-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: entering follower state: follower="Node at 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 [Follower]" leader= testing.go:149: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: entering follower state: follower="Node at 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 [Follower]" leader= testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.318-1200 [WARN] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.318-1200 [WARN] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering candidate state: node="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Candidate]" term=2 testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering candidate state: node="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Candidate]" term=2 testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: election won: tally=2 testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: election won: tally=2 testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering leader state: leader="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Leader]" testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering leader state: leader="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Leader]" testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.525-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.31940152 -1200 -12 m=+6.326718001 (1 nodes), 125.04392ms from start of poll, 125.90122ms from cluster start. Timeout at 2024-06-16 08:26:29.52535004 -1200 -12 m=+6.532666521, 205.94852ms after stability testing.go:149: cluster: 2024-06-16T08:26:29.525-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.31940152 -1200 -12 m=+6.326718001 (1 nodes), 125.04392ms from start of poll, 125.90122ms from cluster start. Timeout at 2024-06-16 08:26:29.52535004 -1200 -12 m=+6.532666521, 205.94852ms after stability testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" --- PASS: TestRaft_ApplyConcurrent (0.35s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.538-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae5a569-8074-3e76-b6c8-c068bf13107a Address:eae5a569-8074-3e76-b6c8-c068bf13107a}]" testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.538-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae5a569-8074-3e76-b6c8-c068bf13107a Address:eae5a569-8074-3e76-b6c8-c068bf13107a}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.538-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:29.538-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.539-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering follower state: follower="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Follower]" leader= testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.539-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering follower state: follower="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Follower]" leader= testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [WARN] server-eae5a569-8074-3e76-b6c8-c068bf13107a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [WARN] server-eae5a569-8074-3e76-b6c8-c068bf13107a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering candidate state: node="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Candidate]" term=2 testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering candidate state: node="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Candidate]" term=2 testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: election won: tally=1 testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: election won: tally=1 testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering leader state: leader="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Leader]" testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering leader state: leader="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Leader]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.918-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.7176431 -1200 -12 m=+6.724959581 (1 nodes), 178.60046ms from start of poll, 178.9662ms from cluster start. Timeout at 2024-06-16 08:26:29.91878398 -1200 -12 m=+6.926100461, 201.14088ms after stability testing.go:149: cluster: 2024-06-16T08:26:29.918-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.7176431 -1200 -12 m=+6.724959581 (1 nodes), 178.60046ms from start of poll, 178.9662ms from cluster start. Timeout at 2024-06-16 08:26:29.91878398 -1200 -12 m=+6.926100461, 201.14088ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.38s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" testLoggerAdapter verbose: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" testing.go:149: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering follower state: follower="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Follower]" leader= testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering follower state: follower="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Follower]" leader= testLoggerAdapter verbose: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: entering follower state: follower="Node at 6a24f9ca-593d-7129-aa83-48db88a724e4 [Follower]" leader= testing.go:149: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: entering follower state: follower="Node at 6a24f9ca-593d-7129-aa83-48db88a724e4 [Follower]" leader= testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: initial configuration: index=0 servers=[] testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.920-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:29.920-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: entering follower state: follower="Node at e25e0d45-e527-3038-4ff3-da4f002f0c86 [Follower]" leader= testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: entering follower state: follower="Node at e25e0d45-e527-3038-4ff3-da4f002f0c86 [Follower]" leader= testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.983-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: no known peers, aborting election testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.983-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: no known peers, aborting election testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering candidate state: node="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Candidate]" term=2 testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering candidate state: node="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Candidate]" term=2 testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: election won: tally=2 testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: election won: tally=2 testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering leader state: leader="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Leader]" testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering leader state: leader="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Leader]" testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-6a24f9ca-593d-7129-aa83-48db88a724e4 testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-6a24f9ca-593d-7129-aa83-48db88a724e4 testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.089-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.98465606 -1200 -12 m=+6.991972541 (1 nodes), 63.98704ms from start of poll, 64.87644ms from cluster start. Timeout at 2024-06-16 08:26:30.08992704 -1200 -12 m=+7.097243501, 105.27096ms after stability testing.go:149: cluster: 2024-06-16T08:26:30.089-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.98465606 -1200 -12 m=+6.991972541 (1 nodes), 63.98704ms from start of poll, 64.87644ms from cluster start. Timeout at 2024-06-16 08:26:30.08992704 -1200 -12 m=+7.097243501, 105.27096ms after stability testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: updating configuration: command=AddStaging server-id=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 server-addr=e25e0d45-e527-3038-4ff3-da4f002f0c86 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4} {Suffrage:Voter ID:server-e25e0d45-e527-3038-4ff3-da4f002f0c86 Address:e25e0d45-e527-3038-4ff3-da4f002f0c86}]" testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: updating configuration: command=AddStaging server-id=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 server-addr=e25e0d45-e527-3038-4ff3-da4f002f0c86 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4} {Suffrage:Voter ID:server-e25e0d45-e527-3038-4ff3-da4f002f0c86 Address:e25e0d45-e527-3038-4ff3-da4f002f0c86}]" testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:30.090-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:30.090-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.090-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: appendEntries rejected, sending older logs: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" next=1 testing.go:149: cluster: 2024-06-16T08:26:30.090-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: appendEntries rejected, sending older logs: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" next=1 testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.196-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.09066014 -1200 -12 m=+7.097976601 (1 nodes), 1.88µs from start of poll, 170.8805ms from cluster start. Timeout at 2024-06-16 08:26:30.196184 -1200 -12 m=+7.203500481, 105.52388ms after stability testing.go:149: cluster: 2024-06-16T08:26:30.196-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.09066014 -1200 -12 m=+7.097976601 (1 nodes), 1.88µs from start of poll, 170.8805ms from cluster start. Timeout at 2024-06-16 08:26:30.196184 -1200 -12 m=+7.203500481, 105.52388ms after stability testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" --- PASS: TestRaft_JoinNode (0.28s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a}]" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a}]" testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering follower state: follower="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Follower]" leader= testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering follower state: follower="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Follower]" leader= testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: initial configuration: index=0 servers=[] testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: entering follower state: follower="Node at 07f472e8-ca3b-7e2a-414f-b15085e0d778 [Follower]" leader= testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: entering follower state: follower="Node at 07f472e8-ca3b-7e2a-414f-b15085e0d778 [Follower]" leader= testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: initial configuration: index=0 servers=[] testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.198-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: entering follower state: follower="Node at 898769a0-7c22-9455-0fbd-cc7e431e89a4 [Follower]" leader= testing.go:149: cluster: 2024-06-16T08:26:30.198-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: entering follower state: follower="Node at 898769a0-7c22-9455-0fbd-cc7e431e89a4 [Follower]" leader= testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.260-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: no known peers, aborting election testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.260-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: no known peers, aborting election testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering candidate state: node="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Candidate]" term=2 testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering candidate state: node="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Candidate]" term=2 testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: election won: tally=1 testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: election won: tally=1 testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.275-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering leader state: leader="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Leader]" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.275-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering leader state: leader="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Leader]" testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.294-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: no known peers, aborting election testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.294-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.381-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.27511884 -1200 -12 m=+7.282435321 (1 nodes), 76.86966ms from start of poll, 77.80958ms from cluster start. Timeout at 2024-06-16 08:26:30.38116972 -1200 -12 m=+7.388486221, 106.0509ms after stability testing.go:149: cluster: 2024-06-16T08:26:30.381-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.27511884 -1200 -12 m=+7.282435321 (1 nodes), 76.86966ms from start of poll, 77.80958ms from cluster start. Timeout at 2024-06-16 08:26:30.38116972 -1200 -12 m=+7.388486221, 106.0509ms after stability testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 server-addr=07f472e8-ca3b-7e2a-414f-b15085e0d778 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778}]" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 server-addr=07f472e8-ca3b-7e2a-414f-b15085e0d778 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778}]" testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.381-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.381-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: cluster: 2024-06-16T08:26:30.381-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.381-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" next=1 testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" next=1 testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.487-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.3817044 -1200 -12 m=+7.389020881 (1 nodes), 960ns from start of poll, 184.39514ms from cluster start. Timeout at 2024-06-16 08:26:30.48769232 -1200 -12 m=+7.495008781, 105.9879ms after stability testing.go:149: cluster: 2024-06-16T08:26:30.487-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.3817044 -1200 -12 m=+7.389020881 (1 nodes), 960ns from start of poll, 184.39514ms from cluster start. Timeout at 2024-06-16 08:26:30.48769232 -1200 -12 m=+7.495008781, 105.9879ms after stability testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.487-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 server-addr=898769a0-7c22-9455-0fbd-cc7e431e89a4 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778} {Suffrage:Voter ID:server-898769a0-7c22-9455-0fbd-cc7e431e89a4 Address:898769a0-7c22-9455-0fbd-cc7e431e89a4}]" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.487-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 server-addr=898769a0-7c22-9455-0fbd-cc7e431e89a4 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778} {Suffrage:Voter ID:server-898769a0-7c22-9455-0fbd-cc7e431e89a4 Address:898769a0-7c22-9455-0fbd-cc7e431e89a4}]" testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.488-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.488-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.488-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" next=1 testing.go:149: cluster: 2024-06-16T08:26:30.488-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" next=1 testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.593-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.48833266 -1200 -12 m=+7.495649141 (1 nodes), 1.1µs from start of poll, 291.0234ms from cluster start. Timeout at 2024-06-16 08:26:30.59335244 -1200 -12 m=+7.600668901, 105.01976ms after stability testing.go:149: cluster: 2024-06-16T08:26:30.593-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.48833266 -1200 -12 m=+7.495649141 (1 nodes), 1.1µs from start of poll, 291.0234ms from cluster start. Timeout at 2024-06-16 08:26:30.59335244 -1200 -12 m=+7.600668901, 105.01976ms after stability testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" --- PASS: TestRaft_JoinNode_ConfigStore (0.40s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.594-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.594-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.595-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering follower state: follower="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Follower]" leader= testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.595-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering follower state: follower="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Follower]" leader= testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering follower state: follower="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Follower]" leader= testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering follower state: follower="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Follower]" leader= testLoggerAdapter verbose: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testing.go:149: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.595-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:30.595-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: entering follower state: follower="Node at a9662707-ebcd-22e7-1816-adfd93dc412c [Follower]" leader= testing.go:149: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: entering follower state: follower="Node at a9662707-ebcd-22e7-1816-adfd93dc412c [Follower]" leader= testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering candidate state: node="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Candidate]" term=2 testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering candidate state: node="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Candidate]" term=2 testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: election won: tally=2 testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: election won: tally=2 testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering leader state: leader="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Leader]" testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering leader state: leader="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Leader]" testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-a9662707-ebcd-22e7-1816-adfd93dc412c testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-a9662707-ebcd-22e7-1816-adfd93dc412c testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.768-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.66295126 -1200 -12 m=+7.670267741 (1 nodes), 67.37054ms from start of poll, 68.21782ms from cluster start. Timeout at 2024-06-16 08:26:30.7684797 -1200 -12 m=+7.775796181, 105.52844ms after stability testing.go:149: cluster: 2024-06-16T08:26:30.768-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.66295126 -1200 -12 m=+7.670267741 (1 nodes), 67.37054ms from start of poll, 68.21782ms from cluster start. Timeout at 2024-06-16 08:26:30.7684797 -1200 -12 m=+7.775796181, 105.52844ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.773-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:30.773-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.879-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:30.77393992 -1200 -12 m=+7.781256401 (2 nodes), 1.9µs from start of poll, 179.20648ms from cluster start. Timeout at 2024-06-16 08:26:30.87988976 -1200 -12 m=+7.887206221, 105.94982ms after stability testing.go:149: cluster: 2024-06-16T08:26:30.879-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:30.77393992 -1200 -12 m=+7.781256401 (2 nodes), 1.9µs from start of poll, 179.20648ms from cluster start. Timeout at 2024-06-16 08:26:30.87988976 -1200 -12 m=+7.887206221, 105.94982ms after stability testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: updating configuration: command=RemoveServer server-id=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 server-addr= servers="[{Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: updating configuration: command=RemoveServer server-id=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 server-addr= servers="[{Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: removed peer, stopping replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 last-index=3 testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: removed peer, stopping replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 last-index=3 testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [WARN] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: heartbeat timeout reached, starting election: last-leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [WARN] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: heartbeat timeout reached, starting election: last-leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering candidate state: node="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Candidate]" term=3 testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering candidate state: node="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Candidate]" term=3 testLoggerAdapter verbose: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.961-1200 [WARN] server-a9662707-ebcd-22e7-1816-adfd93dc412c: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 testing.go:149: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.961-1200 [WARN] server-a9662707-ebcd-22e7-1816-adfd93dc412c: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.961-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.961-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.989-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.989-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" --- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering follower state: follower="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Follower]" leader= testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering follower state: follower="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Follower]" leader= testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.991-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:30.991-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering follower state: follower="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Follower]" leader= testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering follower state: follower="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Follower]" leader= testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [WARN] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [WARN] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering candidate state: node="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Candidate]" term=2 testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering candidate state: node="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Candidate]" term=2 testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: election won: tally=2 testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: election won: tally=2 testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering leader state: leader="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Leader]" testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering leader state: leader="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Leader]" testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-cd4d8f62-3e79-73a2-1349-639a3e2f621b testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-cd4d8f62-3e79-73a2-1349-639a3e2f621b testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.05346856 -1200 -12 m=+8.060785041 (1 nodes), 61.80372ms from start of poll, 62.69052ms from cluster start. Timeout at 2024-06-16 08:26:31.15918994 -1200 -12 m=+8.166506421, 105.72138ms after stability testing.go:149: cluster: 2024-06-16T08:26:31.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.05346856 -1200 -12 m=+8.060785041 (1 nodes), 61.80372ms from start of poll, 62.69052ms from cluster start. Timeout at 2024-06-16 08:26:31.15918994 -1200 -12 m=+8.166506421, 105.72138ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.164-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:31.164-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.270-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.16466338 -1200 -12 m=+8.171979861 (2 nodes), 3.38µs from start of poll, 173.88534ms from cluster start. Timeout at 2024-06-16 08:26:31.27035898 -1200 -12 m=+8.277675541, 105.69568ms after stability testing.go:149: cluster: 2024-06-16T08:26:31.270-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.16466338 -1200 -12 m=+8.171979861 (2 nodes), 3.38µs from start of poll, 173.88534ms from cluster start. Timeout at 2024-06-16 08:26:31.27035898 -1200 -12 m=+8.277675541, 105.69568ms after stability testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.270-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: updating configuration: command=RemoveServer server-id=server-0062dc58-ef6b-0e34-7796-f85c50beed5d server-addr= servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b}]" testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.270-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: updating configuration: command=RemoveServer server-id=server-0062dc58-ef6b-0e34-7796-f85c50beed5d server-addr= servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b}]" testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: removed ourself, shutting down testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: removed ourself, shutting down testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [WARN] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [WARN] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering candidate state: node="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Candidate]" term=3 testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering candidate state: node="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Candidate]" term=3 testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: rejecting vote request since we have a leader: from=0e3a4095-7fbf-c2ef-587d-49551ac414fa leader=0062dc58-ef6b-0e34-7796-f85c50beed5d testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: rejecting vote request since we have a leader: from=0e3a4095-7fbf-c2ef-587d-49551ac414fa leader=0062dc58-ef6b-0e34-7796-f85c50beed5d testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=3 testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=3 testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: duplicate requestVote for same term: term=3 testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: Election timeout reached, restarting election testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: Election timeout reached, restarting election testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=4 testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=4 testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.442-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.442-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: election won: tally=2 testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: election won: tally=2 testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering leader state: leader="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Leader]" testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering leader state: leader="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Leader]" testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.482-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:31.482-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.588-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.48306386 -1200 -12 m=+8.490380321 (1 nodes), 54.82µs from start of poll, 492.2858ms from cluster start. Timeout at 2024-06-16 08:26:31.58823992 -1200 -12 m=+8.595556401, 105.17608ms after stability testing.go:149: cluster: 2024-06-16T08:26:31.588-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.48306386 -1200 -12 m=+8.490380321 (1 nodes), 54.82µs from start of poll, 492.2858ms from cluster start. Timeout at 2024-06-16 08:26:31.58823992 -1200 -12 m=+8.595556401, 105.17608ms after stability testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.588-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.588-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.638-1200 [ERROR] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: failed to heartbeat to: peer=0e3a4095-7fbf-c2ef-587d-49551ac414fa error="command timed out" testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.638-1200 [ERROR] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: failed to heartbeat to: peer=0e3a4095-7fbf-c2ef-587d-49551ac414fa error="command timed out" --- PASS: TestRaft_RemoveLeader (0.65s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.640-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.640-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.640-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.640-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.640-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.640-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.640-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:31.640-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.641-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.641-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.641-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering follower state: follower="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Follower]" leader= testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.641-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering follower state: follower="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Follower]" leader= testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.641-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.641-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.709-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.709-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering candidate state: node="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Candidate]" term=2 testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering candidate state: node="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Candidate]" term=2 testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: election won: tally=2 testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: election won: tally=2 testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering leader state: leader="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Leader]" testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering leader state: leader="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Leader]" testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.816-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.71111416 -1200 -12 m=+8.718430641 (2 nodes), 70.13074ms from start of poll, 70.98378ms from cluster start. Timeout at 2024-06-16 08:26:31.81663246 -1200 -12 m=+8.823948921, 105.51828ms after stability testing.go:149: cluster: 2024-06-16T08:26:31.816-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.71111416 -1200 -12 m=+8.718430641 (2 nodes), 70.13074ms from start of poll, 70.98378ms from cluster start. Timeout at 2024-06-16 08:26:31.81663246 -1200 -12 m=+8.823948921, 105.51828ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.816-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:31.816-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.922-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.81686032 -1200 -12 m=+8.824176781 (1 nodes), 1.8µs from start of poll, 176.72992ms from cluster start. Timeout at 2024-06-16 08:26:31.92223826 -1200 -12 m=+8.929554741, 105.37796ms after stability testing.go:149: cluster: 2024-06-16T08:26:31.922-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.81686032 -1200 -12 m=+8.824176781 (1 nodes), 1.8µs from start of poll, 176.72992ms from cluster start. Timeout at 2024-06-16 08:26:31.92223826 -1200 -12 m=+8.929554741, 105.37796ms after stability testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: updating configuration: command=RemoveServer server-id=server-b89d8880-c899-eb95-6af3-fbceba7c6165 server-addr= servers="[{Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: updating configuration: command=RemoveServer server-id=server-b89d8880-c899-eb95-6af3-fbceba7c6165 server-addr= servers="[{Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: removed ourself, transitioning to follower testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: removed ourself, transitioning to follower testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:32.000-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: not part of stable configuration, aborting election testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:32.000-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: not part of stable configuration, aborting election testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=3 testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=3 testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.017-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: rejecting vote request since we have a leader: from=fbb6f22a-8dde-c5a7-a827-f24b878c8ebc leader=b89d8880-c899-eb95-6af3-fbceba7c6165 testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.017-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: rejecting vote request since we have a leader: from=fbb6f22a-8dde-c5a7-a827-f24b878c8ebc leader=b89d8880-c899-eb95-6af3-fbceba7c6165 testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.031-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:32.031-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering candidate state: node="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Candidate]" term=3 testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering candidate state: node="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Candidate]" term=3 testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.033-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: duplicate requestVote for same term: term=3 testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.033-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: Election timeout reached, restarting election testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: Election timeout reached, restarting election testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=4 testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=4 testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.072-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.072-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: election won: tally=2 testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: election won: tally=2 testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering leader state: leader="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Leader]" testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering leader state: leader="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Leader]" testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.177-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.07232734 -1200 -12 m=+9.079643861 (1 nodes), 40.98038ms from start of poll, 432.197ms from cluster start. Timeout at 2024-06-16 08:26:32.17767624 -1200 -12 m=+9.184992741, 105.34888ms after stability testing.go:149: cluster: 2024-06-16T08:26:32.177-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.07232734 -1200 -12 m=+9.079643861 (1 nodes), 40.98038ms from start of poll, 432.197ms from cluster start. Timeout at 2024-06-16 08:26:32.17767624 -1200 -12 m=+9.184992741, 105.34888ms after stability testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.283-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.283-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.64s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.284-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.284-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testLoggerAdapter verbose: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.284-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testing.go:149: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.284-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testLoggerAdapter verbose: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testing.go:149: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testLoggerAdapter verbose: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testing.go:149: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.285-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:32.285-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.285-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.285-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= testLoggerAdapter verbose: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.285-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: entering follower state: follower="Node at fddfb24d-a02d-e4da-534f-901802878db2 [Follower]" leader= testing.go:149: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.285-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: entering follower state: follower="Node at fddfb24d-a02d-e4da-534f-901802878db2 [Follower]" leader= testLoggerAdapter verbose: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: entering follower state: follower="Node at bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d [Follower]" leader= testing.go:149: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: entering follower state: follower="Node at bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d [Follower]" leader= testLoggerAdapter verbose: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: entering follower state: follower="Node at fc14acdc-48c9-ff27-6746-526d724f67ff [Follower]" leader= testing.go:149: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: entering follower state: follower="Node at fc14acdc-48c9-ff27-6746-526d724f67ff [Follower]" leader= testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering candidate state: node="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Candidate]" term=2 testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering candidate state: node="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Candidate]" term=2 testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: election won: tally=3 testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: election won: tally=3 testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering leader state: leader="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Leader]" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering leader state: leader="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Leader]" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fddfb24d-a02d-e4da-534f-901802878db2 testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fddfb24d-a02d-e4da-534f-901802878db2 testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.444-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.33934194 -1200 -12 m=+9.346658401 (1 nodes), 53.80772ms from start of poll, 54.95956ms from cluster start. Timeout at 2024-06-16 08:26:32.44483336 -1200 -12 m=+9.452149841, 105.49144ms after stability testing.go:149: cluster: 2024-06-16T08:26:32.444-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.33934194 -1200 -12 m=+9.346658401 (1 nodes), 53.80772ms from start of poll, 54.95956ms from cluster start. Timeout at 2024-06-16 08:26:32.44483336 -1200 -12 m=+9.452149841, 105.49144ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.550-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:32.550-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.656-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.55073952 -1200 -12 m=+9.558056521 (3 nodes), 3.66µs from start of poll, 266.35768ms from cluster start. Timeout at 2024-06-16 08:26:32.65630494 -1200 -12 m=+9.663621421, 105.5649ms after stability testing.go:149: cluster: 2024-06-16T08:26:32.656-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.55073952 -1200 -12 m=+9.558056521 (3 nodes), 3.66µs from start of poll, 266.35768ms from cluster start. Timeout at 2024-06-16 08:26:32.65630494 -1200 -12 m=+9.663621421, 105.5649ms after stability testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: updating configuration: command=RemoveServer server-id=server-fc14acdc-48c9-ff27-6746-526d724f67ff server-addr= servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}]" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: updating configuration: command=RemoveServer server-id=server-fc14acdc-48c9-ff27-6746-526d724f67ff server-addr= servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}]" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="append pipeline closed" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="append pipeline closed" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: removed peer, stopping replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff last-index=3 testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: removed peer, stopping replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff last-index=3 testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="append pipeline closed" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="append pipeline closed" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.667-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.667-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.676-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.676-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.681-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.681-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.683-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.683-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.686-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.686-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.695-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.695-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.699-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.699-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.701-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.701-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.704-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.704-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.705-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=50.5593ms testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.705-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=50.5593ms testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=61.05256ms testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=61.05256ms testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d time=59.97694ms testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d time=59.97694ms testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact quorum of nodes, stepping down testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testing.go:149: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testLoggerAdapter verbose: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: entering follower state: follower="Node at af538914-1821-42ed-618e-9b1ec3c36966 [Follower]" leader= testing.go:149: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: entering follower state: follower="Node at af538914-1821-42ed-618e-9b1ec3c36966 [Follower]" leader= testLoggerAdapter verbose: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testing.go:149: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testLoggerAdapter verbose: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: entering follower state: follower="Node at 6dd18996-5082-c331-dddb-6c455a171679 [Follower]" leader= testing.go:149: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: entering follower state: follower="Node at 6dd18996-5082-c331-dddb-6c455a171679 [Follower]" leader= testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.718-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:32.718-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering follower state: follower="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Follower]" leader= testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering follower state: follower="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Follower]" leader= testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [WARN] server-4270c846-57e6-1c86-bab8-bd630057d2be: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [WARN] server-4270c846-57e6-1c86-bab8-bd630057d2be: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering candidate state: node="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Candidate]" term=2 testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering candidate state: node="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Candidate]" term=2 testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: election won: tally=2 testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: election won: tally=2 testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering leader state: leader="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Leader]" testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering leader state: leader="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Leader]" testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-af538914-1821-42ed-618e-9b1ec3c36966 testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-af538914-1821-42ed-618e-9b1ec3c36966 testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-6dd18996-5082-c331-dddb-6c455a171679 testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-6dd18996-5082-c331-dddb-6c455a171679 testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.882-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.77633578 -1200 -12 m=+9.783652261 (1 nodes), 57.44554ms from start of poll, 58.36796ms from cluster start. Timeout at 2024-06-16 08:26:32.8820105 -1200 -12 m=+9.889327121, 105.67486ms after stability testing.go:149: cluster: 2024-06-16T08:26:32.882-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.77633578 -1200 -12 m=+9.783652261 (1 nodes), 57.44554ms from start of poll, 58.36796ms from cluster start. Timeout at 2024-06-16 08:26:32.8820105 -1200 -12 m=+9.889327121, 105.67486ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.882-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:32.882-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.987-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.88223498 -1200 -12 m=+9.889551461 (2 nodes), 4.34µs from start of poll, 164.26716ms from cluster start. Timeout at 2024-06-16 08:26:32.98762006 -1200 -12 m=+9.994936541, 105.38508ms after stability testing.go:149: cluster: 2024-06-16T08:26:32.987-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.88223498 -1200 -12 m=+9.889551461 (2 nodes), 4.34µs from start of poll, 164.26716ms from cluster start. Timeout at 2024-06-16 08:26:32.98762006 -1200 -12 m=+9.994936541, 105.38508ms after stability testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.987-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: updating configuration: command=AddStaging server-id=server-af538914-1821-42ed-618e-9b1ec3c36966 server-addr=af538914-1821-42ed-618e-9b1ec3c36966 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.987-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: updating configuration: command=AddStaging server-id=server-af538914-1821-42ed-618e-9b1ec3c36966 server-addr=af538914-1821-42ed-618e-9b1ec3c36966 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testing.go:149: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testLoggerAdapter verbose: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: entering follower state: follower="Node at b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 [Follower]" leader= testing.go:149: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: entering follower state: follower="Node at b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 [Follower]" leader= testLoggerAdapter verbose: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testing.go:149: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testLoggerAdapter verbose: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: entering follower state: follower="Node at 6faaeb84-e589-5cf1-7604-6cb47ad8e92d [Follower]" leader= testing.go:149: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: entering follower state: follower="Node at 6faaeb84-e589-5cf1-7604-6cb47ad8e92d [Follower]" leader= testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.990-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:32.990-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering follower state: follower="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Follower]" leader= testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering follower state: follower="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Follower]" leader= testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [WARN] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [WARN] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering candidate state: node="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Candidate]" term=2 testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering candidate state: node="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Candidate]" term=2 testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: election won: tally=2 testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: election won: tally=2 testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering leader state: leader="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Leader]" testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering leader state: leader="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Leader]" testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.149-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.0437789 -1200 -12 m=+10.051095361 (1 nodes), 53.30942ms from start of poll, 54.24352ms from cluster start. Timeout at 2024-06-16 08:26:33.14894488 -1200 -12 m=+10.156261381, 105.16602ms after stability testing.go:149: cluster: 2024-06-16T08:26:33.149-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.0437789 -1200 -12 m=+10.051095361 (1 nodes), 53.30942ms from start of poll, 54.24352ms from cluster start. Timeout at 2024-06-16 08:26:33.14894488 -1200 -12 m=+10.156261381, 105.16602ms after stability testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: updating configuration: command=RemoveServer server-id=5705c359-0d59-4ebd-4fe8-01c185a91d9c server-addr= servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: updating configuration: command=RemoveServer server-id=5705c359-0d59-4ebd-4fe8-01c185a91d9c server-addr= servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.150-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:33.150-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering follower state: follower="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Follower]" leader= testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering follower state: follower="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Follower]" leader= testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [WARN] server-36ab0502-1ee0-9019-44be-303864e2599d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [WARN] server-36ab0502-1ee0-9019-44be-303864e2599d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering candidate state: node="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Candidate]" term=2 testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering candidate state: node="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Candidate]" term=2 testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: election won: tally=1 testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: election won: tally=1 testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering leader state: leader="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Leader]" testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering leader state: leader="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Leader]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.329-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.22349062 -1200 -12 m=+10.230807161 (1 nodes), 72.59678ms from start of poll, 72.94072ms from cluster start. Timeout at 2024-06-16 08:26:33.3292815 -1200 -12 m=+10.336597981, 105.79082ms after stability testing.go:149: cluster: 2024-06-16T08:26:33.329-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.22349062 -1200 -12 m=+10.230807161 (1 nodes), 72.59678ms from start of poll, 72.94072ms from cluster start. Timeout at 2024-06-16 08:26:33.3292815 -1200 -12 m=+10.336597981, 105.79082ms after stability testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.333-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: starting snapshot up to: index=102 testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.333-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: starting snapshot up to: index=102 testing.go:153: 2024-06-16T08:26:33.333-1200 [INFO] creating new snapshot: path=/tmp/raft696262971/snapshots/2-102-1718569593333.tmp testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: compacting logs: from=1 to=92 testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: snapshot complete up to: index=102 testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: snapshot complete up to: index=102 testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: restored from snapshot: id=2-102-1718569593333 testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: restored from snapshot: id=2-102-1718569593333 testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" --- PASS: TestRaft_SnapshotRestore (0.19s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.337-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:33.337-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering follower state: follower="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Follower]" leader= testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering follower state: follower="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Follower]" leader= testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.393-1200 [WARN] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.393-1200 [WARN] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering candidate state: node="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Candidate]" term=2 testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering candidate state: node="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Candidate]" term=2 testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: election won: tally=1 testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: election won: tally=1 testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering leader state: leader="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Leader]" testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering leader state: leader="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Leader]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.500-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.39432972 -1200 -12 m=+10.401646201 (1 nodes), 57.06134ms from start of poll, 57.4243ms from cluster start. Timeout at 2024-06-16 08:26:33.5000756 -1200 -12 m=+10.507392101, 105.7459ms after stability testing.go:149: cluster: 2024-06-16T08:26:33.500-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.39432972 -1200 -12 m=+10.401646201 (1 nodes), 57.06134ms from start of poll, 57.4243ms from cluster start. Timeout at 2024-06-16 08:26:33.5000756 -1200 -12 m=+10.507392101, 105.7459ms after stability testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.502-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: starting snapshot up to: index=102 testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.502-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: starting snapshot up to: index=102 testing.go:153: 2024-06-16T08:26:33.502-1200 [INFO] creating new snapshot: path=/tmp/raft704908476/snapshots/2-102-1718569593502.tmp testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: compacting logs: from=1 to=92 testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: snapshot complete up to: index=102 testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: snapshot complete up to: index=102 testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" --- PASS: TestRaft_NoRestoreOnStart (0.17s) === RUN TestRaft_SnapshotRestore_PeerChange testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering follower state: follower="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Follower]" leader= raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fc5db7f-287a-e808-6b09-c0c758b1d12b Address:0fc5db7f-287a-e808-6b09-c0c758b1d12b}]" testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fc5db7f-287a-e808-6b09-c0c758b1d12b Address:0fc5db7f-287a-e808-6b09-c0c758b1d12b}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.504-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:33.504-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering follower state: follower="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Follower]" leader= testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering follower state: follower="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Follower]" leader= testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [WARN] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering candidate state: node="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Candidate]" term=3 testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: election won: tally=1 testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering leader state: leader="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Leader]" testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.595-1200 [WARN] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.595-1200 [WARN] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering candidate state: node="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Candidate]" term=2 testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering candidate state: node="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Candidate]" term=2 testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: election won: tally=1 testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: election won: tally=1 testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering leader state: leader="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Leader]" testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering leader state: leader="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Leader]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.701-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.59629328 -1200 -12 m=+10.603609761 (1 nodes), 91.64214ms from start of poll, 91.9887ms from cluster start. Timeout at 2024-06-16 08:26:33.70157222 -1200 -12 m=+10.708888721, 105.27896ms after stability testing.go:149: cluster: 2024-06-16T08:26:33.701-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.59629328 -1200 -12 m=+10.603609761 (1 nodes), 91.64214ms from start of poll, 91.9887ms from cluster start. Timeout at 2024-06-16 08:26:33.70157222 -1200 -12 m=+10.708888721, 105.27896ms after stability testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.701-1200 [ERROR] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.701-1200 [ERROR] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: starting snapshot up to: index=12 testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: starting snapshot up to: index=12 testing.go:153: 2024-06-16T08:26:33.702-1200 [INFO] creating new snapshot: path=/tmp/raft617634784/snapshots/2-12-1718569593702.tmp testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: compacting logs: from=1 to=2 testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: snapshot complete up to: index=12 testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.20s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: entering follower state: follower="Node at b5071370-c7f5-11c6-8f28-fa99b8f7618e [Follower]" leader= testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: entering follower state: follower="Node at b5071370-c7f5-11c6-8f28-fa99b8f7618e [Follower]" leader= testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering follower state: follower="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Follower]" leader= testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering follower state: follower="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Follower]" leader= testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.704-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:33.704-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: entering follower state: follower="Node at 12bfc68f-2775-5023-31e0-b166f8e418b0 [Follower]" leader= testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: entering follower state: follower="Node at 12bfc68f-2775-5023-31e0-b166f8e418b0 [Follower]" leader= testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [WARN] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [WARN] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering candidate state: node="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Candidate]" term=2 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering candidate state: node="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Candidate]" term=2 testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: election won: tally=2 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: election won: tally=2 testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering leader state: leader="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Leader]" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering leader state: leader="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Leader]" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-b5071370-c7f5-11c6-8f28-fa99b8f7618e testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-b5071370-c7f5-11c6-8f28-fa99b8f7618e testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-12bfc68f-2775-5023-31e0-b166f8e418b0 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-12bfc68f-2775-5023-31e0-b166f8e418b0 testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.766-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.766-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.871-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.76584174 -1200 -12 m=+10.773158201 (1 nodes), 61.02984ms from start of poll, 61.8581ms from cluster start. Timeout at 2024-06-16 08:26:33.8714222 -1200 -12 m=+10.878738681, 105.58048ms after stability testing.go:149: cluster: 2024-06-16T08:26:33.871-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.76584174 -1200 -12 m=+10.773158201 (1 nodes), 61.02984ms from start of poll, 61.8581ms from cluster start. Timeout at 2024-06-16 08:26:33.8714222 -1200 -12 m=+10.878738681, 105.58048ms after stability testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: starting snapshot up to: index=12 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: starting snapshot up to: index=12 testing.go:153: 2024-06-16T08:26:33.872-1200 [INFO] creating new snapshot: path=/tmp/raft4118912225/snapshots/2-12-1718569593872.tmp testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: snapshot complete up to: index=12 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: snapshot complete up to: index=12 testing.go:153: 2024-06-16T08:26:33.873-1200 [INFO] creating new snapshot: path=/tmp/raft4118912225/snapshots/2-23-1718569593873.tmp testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: copied to local snapshot: bytes=71 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: restored user snapshot: index=1 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: restored user snapshot: index=1 testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.880-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.880-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:33.880-1200 [INFO] creating new snapshot: path=/tmp/raft389080924/snapshots/2-23-1718569593880.tmp testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.881-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: copied to local snapshot: bytes=71 testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.881-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: Installed remote snapshot testing.go:153: 2024-06-16T08:26:33.881-1200 [INFO] creating new snapshot: path=/tmp/raft584946946/snapshots/2-23-1718569593881.tmp testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: Installed remote snapshot testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.882-1200 [WARN] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: skipping application of old log: index=22 testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.882-1200 [WARN] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: skipping application of old log: index=22 testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.882-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.882-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: copied to local snapshot: bytes=71 testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: Installed remote snapshot testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: Installed remote snapshot testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.883-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.883-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering follower state: follower="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Follower]" leader= testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering follower state: follower="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Follower]" leader= testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: entering follower state: follower="Node at fac2cacf-c72b-be2e-7e42-eeea6ad8a487 [Follower]" leader= testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: entering follower state: follower="Node at fac2cacf-c72b-be2e-7e42-eeea6ad8a487 [Follower]" leader= testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.902-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:33.902-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: entering follower state: follower="Node at 5d0d7372-098f-5f1d-2c02-e4efe3e63b76 [Follower]" leader= testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: entering follower state: follower="Node at 5d0d7372-098f-5f1d-2c02-e4efe3e63b76 [Follower]" leader= testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [WARN] server-9c921841-4197-8da2-36d8-c32b894b1a29: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [WARN] server-9c921841-4197-8da2-36d8-c32b894b1a29: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering candidate state: node="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Candidate]" term=2 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering candidate state: node="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Candidate]" term=2 testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: election won: tally=2 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: election won: tally=2 testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering leader state: leader="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Leader]" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering leader state: leader="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Leader]" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.060-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.9548837 -1200 -12 m=+10.962200181 (1 nodes), 52.66668ms from start of poll, 53.58104ms from cluster start. Timeout at 2024-06-16 08:26:34.06018984 -1200 -12 m=+11.067506321, 105.30614ms after stability testing.go:149: cluster: 2024-06-16T08:26:34.060-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.9548837 -1200 -12 m=+10.962200181 (1 nodes), 52.66668ms from start of poll, 53.58104ms from cluster start. Timeout at 2024-06-16 08:26:34.06018984 -1200 -12 m=+11.067506321, 105.30614ms after stability testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.060-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: starting snapshot up to: index=12 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.060-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: starting snapshot up to: index=12 testing.go:153: 2024-06-16T08:26:34.060-1200 [INFO] creating new snapshot: path=/tmp/raft4232237734/snapshots/2-12-1718569594060.tmp testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.061-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: snapshot complete up to: index=12 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.061-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: snapshot complete up to: index=12 testing.go:153: 2024-06-16T08:26:34.062-1200 [INFO] creating new snapshot: path=/tmp/raft4232237734/snapshots/2-23-1718569594062.tmp testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.062-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: copied to local snapshot: bytes=71 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.062-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: restored user snapshot: index=1 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: restored user snapshot: index=1 testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.072-1200 [INFO] creating new snapshot: path=/tmp/raft3318448802/snapshots/2-23-1718569594072.tmp testing.go:153: 2024-06-16T08:26:34.072-1200 [INFO] creating new snapshot: path=/tmp/raft3024632065/snapshots/2-23-1718569594072.tmp testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: copied to local snapshot: bytes=71 testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: copied to local snapshot: bytes=71 testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: Installed remote snapshot testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: Installed remote snapshot testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: Installed remote snapshot testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: Installed remote snapshot testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [WARN] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: skipping application of old log: index=23 testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [WARN] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: skipping application of old log: index=23 testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [WARN] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: skipping application of old log: index=22 testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.073-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [WARN] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: skipping application of old log: index=22 testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.073-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.074-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.074-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering follower state: follower="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Follower]" leader= testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering follower state: follower="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Follower]" leader= testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.086-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.086-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.087-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader= testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.087-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader= testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.087-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:34.087-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: entering follower state: follower="Node at a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 [Follower]" leader= testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: entering follower state: follower="Node at a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 [Follower]" leader= testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering candidate state: node="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Candidate]" term=2 testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering candidate state: node="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Candidate]" term=2 testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering candidate state: node="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Candidate]" term=2 testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: election won: tally=2 testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering candidate state: node="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Candidate]" term=2 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: election won: tally=2 testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering leader state: leader="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Leader]" testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.161-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: duplicate requestVote for same term: term=2 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering leader state: leader="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Leader]" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: duplicate requestVote for same term: term=2 testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.161-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: duplicate requestVote for same term: term=2 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: rejecting vote request since we have a leader: from=748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 leader=75f47187-606a-287e-3ada-fd9fdf7652db testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: rejecting vote request since we have a leader: from=748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 leader=75f47187-606a-287e-3ada-fd9fdf7652db testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader=75f47187-606a-287e-3ada-fd9fdf7652db testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader=75f47187-606a-287e-3ada-fd9fdf7652db testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.267-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.16194074 -1200 -12 m=+11.169257221 (1 nodes), 74.55118ms from start of poll, 75.44748ms from cluster start. Timeout at 2024-06-16 08:26:34.26708708 -1200 -12 m=+11.274403581, 105.14636ms after stability testing.go:149: cluster: 2024-06-16T08:26:34.267-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.16194074 -1200 -12 m=+11.169257221 (1 nodes), 74.55118ms from start of poll, 75.44748ms from cluster start. Timeout at 2024-06-16 08:26:34.26708708 -1200 -12 m=+11.274403581, 105.14636ms after stability testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.267-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: starting snapshot up to: index=12 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.267-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: starting snapshot up to: index=12 testing.go:153: 2024-06-16T08:26:34.267-1200 [INFO] creating new snapshot: path=/tmp/raft3582674493/snapshots/2-12-1718569594267.tmp testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.268-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: snapshot complete up to: index=12 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.268-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: snapshot complete up to: index=12 testing.go:153: 2024-06-16T08:26:34.269-1200 [INFO] creating new snapshot: path=/tmp/raft3582674493/snapshots/2-23-1718569594269.tmp testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.269-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: copied to local snapshot: bytes=71 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.269-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: restored user snapshot: index=1 testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: restored user snapshot: index=1 testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.276-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.276-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.277-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.277-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.277-1200 [INFO] creating new snapshot: path=/tmp/raft53509326/snapshots/2-23-1718569594277.tmp testing.go:153: 2024-06-16T08:26:34.277-1200 [INFO] creating new snapshot: path=/tmp/raft1487043319/snapshots/2-23-1718569594277.tmp testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.277-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: copied to local snapshot: bytes=71 testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.277-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: Installed remote snapshot testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: copied to local snapshot: bytes=71 testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: Installed remote snapshot testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: skipping application of old log: index=22 testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: skipping application of old log: index=22 testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: Installed remote snapshot testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: Installed remote snapshot testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering follower state: follower="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Follower]" leader= testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering follower state: follower="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Follower]" leader= testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: entering follower state: follower="Node at 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 [Follower]" leader= testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: entering follower state: follower="Node at 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 [Follower]" leader= testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.291-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.291-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.292-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:34.292-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.292-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: entering follower state: follower="Node at f8a9cce6-167c-32ae-626d-909270ad700e [Follower]" leader= testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.292-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: entering follower state: follower="Node at f8a9cce6-167c-32ae-626d-909270ad700e [Follower]" leader= testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [WARN] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [WARN] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering candidate state: node="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Candidate]" term=2 testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering candidate state: node="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Candidate]" term=2 testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: election won: tally=2 testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: election won: tally=2 testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering leader state: leader="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Leader]" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering leader state: leader="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Leader]" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-f8a9cce6-167c-32ae-626d-909270ad700e testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-f8a9cce6-167c-32ae-626d-909270ad700e testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.449-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.34359024 -1200 -12 m=+11.350906721 (1 nodes), 51.51246ms from start of poll, 52.41596ms from cluster start. Timeout at 2024-06-16 08:26:34.44952492 -1200 -12 m=+11.456841401, 105.93468ms after stability testing.go:149: cluster: 2024-06-16T08:26:34.449-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.34359024 -1200 -12 m=+11.350906721 (1 nodes), 51.51246ms from start of poll, 52.41596ms from cluster start. Timeout at 2024-06-16 08:26:34.44952492 -1200 -12 m=+11.456841401, 105.93468ms after stability testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: starting snapshot up to: index=12 testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: starting snapshot up to: index=12 testing.go:153: 2024-06-16T08:26:34.450-1200 [INFO] creating new snapshot: path=/tmp/raft2487656942/snapshots/2-12-1718569594450.tmp testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: snapshot complete up to: index=12 testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: snapshot complete up to: index=12 testing.go:153: 2024-06-16T08:26:34.451-1200 [INFO] creating new snapshot: path=/tmp/raft2487656942/snapshots/2-113-1718569594451.tmp testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: copied to local snapshot: bytes=71 testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: restored user snapshot: index=1 testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: restored user snapshot: index=1 testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.459-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.459-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.459-1200 [INFO] creating new snapshot: path=/tmp/raft2616903032/snapshots/2-113-1718569594459.tmp testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: copied to local snapshot: bytes=71 testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: Installed remote snapshot testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: Installed remote snapshot testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [WARN] server-f8a9cce6-167c-32ae-626d-909270ad700e: skipping application of old log: index=22 testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [WARN] server-f8a9cce6-167c-32ae-626d-909270ad700e: skipping application of old log: index=22 testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.460-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.460-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.462-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.462-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.462-1200 [INFO] creating new snapshot: path=/tmp/raft783977926/snapshots/2-113-1718569594462.tmp testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: copied to local snapshot: bytes=71 testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: Installed remote snapshot testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: Installed remote snapshot testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.463-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.463-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: entering follower state: follower="Node at ccb1cd15-a369-56a9-bd89-ee3eca03a529 [Follower]" leader= testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: entering follower state: follower="Node at ccb1cd15-a369-56a9-bd89-ee3eca03a529 [Follower]" leader= testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering follower state: follower="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Follower]" leader= testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering follower state: follower="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Follower]" leader= testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.479-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:34.479-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: entering follower state: follower="Node at 20f55512-047a-c61f-7524-298ee940e3f5 [Follower]" leader= testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: entering follower state: follower="Node at 20f55512-047a-c61f-7524-298ee940e3f5 [Follower]" leader= testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [WARN] server-f921f1b9-52d8-92ec-d031-2021083efbc1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [WARN] server-f921f1b9-52d8-92ec-d031-2021083efbc1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering candidate state: node="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Candidate]" term=2 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering candidate state: node="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Candidate]" term=2 testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: election won: tally=2 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: election won: tally=2 testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering leader state: leader="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Leader]" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering leader state: leader="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Leader]" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-20f55512-047a-c61f-7524-298ee940e3f5 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-20f55512-047a-c61f-7524-298ee940e3f5 testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.656-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.55144138 -1200 -12 m=+11.558757861 (1 nodes), 71.7895ms from start of poll, 72.5549ms from cluster start. Timeout at 2024-06-16 08:26:34.65662068 -1200 -12 m=+11.663937161, 105.1793ms after stability testing.go:149: cluster: 2024-06-16T08:26:34.656-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.55144138 -1200 -12 m=+11.558757861 (1 nodes), 71.7895ms from start of poll, 72.5549ms from cluster start. Timeout at 2024-06-16 08:26:34.65662068 -1200 -12 m=+11.663937161, 105.1793ms after stability testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.657-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: starting snapshot up to: index=12 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.657-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: starting snapshot up to: index=12 testing.go:153: 2024-06-16T08:26:34.657-1200 [INFO] creating new snapshot: path=/tmp/raft3519587632/snapshots/2-12-1718569594657.tmp testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.658-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: snapshot complete up to: index=12 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.658-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: snapshot complete up to: index=12 testing.go:153: 2024-06-16T08:26:34.658-1200 [INFO] creating new snapshot: path=/tmp/raft3519587632/snapshots/2-1013-1718569594658.tmp testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: copied to local snapshot: bytes=71 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: restored user snapshot: index=1 testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: restored user snapshot: index=1 testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.665-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.665-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.665-1200 [INFO] creating new snapshot: path=/tmp/raft3103839251/snapshots/2-1013-1718569594665.tmp testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: copied to local snapshot: bytes=71 testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: Installed remote snapshot testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: Installed remote snapshot testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [WARN] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: skipping application of old log: index=22 testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [WARN] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: skipping application of old log: index=22 testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.666-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.666-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.667-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.667-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.668-1200 [INFO] creating new snapshot: path=/tmp/raft3458475960/snapshots/2-1013-1718569594668.tmp testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.668-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: copied to local snapshot: bytes=71 testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.668-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.669-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: Installed remote snapshot testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.669-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: Installed remote snapshot testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.669-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.669-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: entering follower state: follower="Node at a58ac0e2-2965-4558-48c1-70c75bd2e518 [Follower]" leader= testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: entering follower state: follower="Node at a58ac0e2-2965-4558-48c1-70c75bd2e518 [Follower]" leader= testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering follower state: follower="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Follower]" leader= testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering follower state: follower="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Follower]" leader= testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.685-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:34.685-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: entering follower state: follower="Node at 080b15d9-535a-359f-f790-4c3c8ad340ac [Follower]" leader= testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: entering follower state: follower="Node at 080b15d9-535a-359f-f790-4c3c8ad340ac [Follower]" leader= testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [WARN] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [WARN] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering candidate state: node="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Candidate]" term=2 testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering candidate state: node="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Candidate]" term=2 testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: election won: tally=2 testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: election won: tally=2 testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering leader state: leader="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Leader]" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering leader state: leader="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Leader]" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-a58ac0e2-2965-4558-48c1-70c75bd2e518 testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-a58ac0e2-2965-4558-48c1-70c75bd2e518 testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-080b15d9-535a-359f-f790-4c3c8ad340ac testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-080b15d9-535a-359f-f790-4c3c8ad340ac testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.758-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.758-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.863-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.75729106 -1200 -12 m=+11.764607521 (1 nodes), 71.5385ms from start of poll, 72.31258ms from cluster start. Timeout at 2024-06-16 08:26:34.8632906 -1200 -12 m=+11.870607081, 105.99956ms after stability testing.go:149: cluster: 2024-06-16T08:26:34.863-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.75729106 -1200 -12 m=+11.764607521 (1 nodes), 71.5385ms from start of poll, 72.31258ms from cluster start. Timeout at 2024-06-16 08:26:34.8632906 -1200 -12 m=+11.870607081, 105.99956ms after stability testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: starting snapshot up to: index=12 testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: starting snapshot up to: index=12 testing.go:153: 2024-06-16T08:26:34.864-1200 [INFO] creating new snapshot: path=/tmp/raft3326532870/snapshots/2-12-1718569594864.tmp testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: snapshot complete up to: index=12 testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: snapshot complete up to: index=12 testing.go:153: 2024-06-16T08:26:34.865-1200 [INFO] creating new snapshot: path=/tmp/raft3326532870/snapshots/2-10013-1718569594865.tmp testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: copied to local snapshot: bytes=71 testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: restored user snapshot: index=1 testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: restored user snapshot: index=1 testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.873-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.873-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.873-1200 [INFO] creating new snapshot: path=/tmp/raft788190039/snapshots/2-10013-1718569594873.tmp testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: copied to local snapshot: bytes=71 testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: Installed remote snapshot testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: Installed remote snapshot testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [WARN] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: skipping application of old log: index=22 testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [WARN] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: skipping application of old log: index=22 testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.874-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.874-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.876-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.876-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-16T08:26:34.876-1200 [INFO] creating new snapshot: path=/tmp/raft4216072965/snapshots/2-10013-1718569594876.tmp testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: copied to local snapshot: bytes=71 testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: Installed remote snapshot testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: Installed remote snapshot testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.877-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.877-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" --- PASS: TestRaft_UserRestore (1.19s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering follower state: follower="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Follower]" leader= testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering follower state: follower="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Follower]" leader= testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: entering follower state: follower="Node at 30b015ac-9bf9-5785-9244-abf831fd55ad [Follower]" leader= testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: entering follower state: follower="Node at 30b015ac-9bf9-5785-9244-abf831fd55ad [Follower]" leader= testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.893-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:34.893-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: entering follower state: follower="Node at b357c0c5-4d5a-09d9-0c03-78e414f2b6ee [Follower]" leader= testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: entering follower state: follower="Node at b357c0c5-4d5a-09d9-0c03-78e414f2b6ee [Follower]" leader= testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [WARN] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [WARN] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering candidate state: node="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Candidate]" term=2 testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering candidate state: node="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Candidate]" term=2 testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: election won: tally=2 testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: election won: tally=2 testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering leader state: leader="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Leader]" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering leader state: leader="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Leader]" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-30b015ac-9bf9-5785-9244-abf831fd55ad testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-30b015ac-9bf9-5785-9244-abf831fd55ad testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.054-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:34.94804662 -1200 -12 m=+11.955363081 (2 nodes), 54.0292ms from start of poll, 54.87626ms from cluster start. Timeout at 2024-06-16 08:26:35.05395696 -1200 -12 m=+12.061273421, 105.91034ms after stability testing.go:149: cluster: 2024-06-16T08:26:35.054-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:34.94804662 -1200 -12 m=+11.955363081 (2 nodes), 54.0292ms from start of poll, 54.87626ms from cluster start. Timeout at 2024-06-16 08:26:35.05395696 -1200 -12 m=+12.061273421, 105.91034ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.054-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:35.054-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.05419498 -1200 -12 m=+12.061511441 (1 nodes), 1.68µs from start of poll, 161.02462ms from cluster start. Timeout at 2024-06-16 08:26:35.1597508 -1200 -12 m=+12.167067261, 105.55582ms after stability testing.go:149: cluster: 2024-06-16T08:26:35.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.05419498 -1200 -12 m=+12.061511441 (1 nodes), 1.68µs from start of poll, 161.02462ms from cluster start. Timeout at 2024-06-16 08:26:35.1597508 -1200 -12 m=+12.167067261, 105.55582ms after stability testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to pipeline appendEntries: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="append pipeline closed" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to pipeline appendEntries: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="append pipeline closed" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to appendEntries to: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to appendEntries to: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to heartbeat to: peer=30b015ac-9bf9-5785-9244-abf831fd55ad error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to heartbeat to: peer=30b015ac-9bf9-5785-9244-abf831fd55ad error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: starting snapshot up to: index=102 testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: starting snapshot up to: index=102 testing.go:153: 2024-06-16T08:26:35.162-1200 [INFO] creating new snapshot: path=/tmp/raft2304783895/snapshots/2-102-1718569595162.tmp testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: compacting logs: from=1 to=92 testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.163-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: snapshot complete up to: index=102 testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.163-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: snapshot complete up to: index=102 testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.163-1200 [ERROR] server-30b015ac-9bf9-5785-9244-abf831fd55ad: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.163-1200 [ERROR] server-30b015ac-9bf9-5785-9244-abf831fd55ad: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: starting snapshot up to: index=99 testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: starting snapshot up to: index=99 testing.go:153: 2024-06-16T08:26:35.163-1200 [INFO] creating new snapshot: path=/tmp/raft2637793511/snapshots/2-99-1718569595163.tmp testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: compacting logs: from=1 to=92 testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: snapshot complete up to: index=99 testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: snapshot complete up to: index=99 testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.171-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to get log: index=2 error="log not found" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.171-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to get log: index=2 error="log not found" testing.go:153: 2024-06-16T08:26:35.171-1200 [INFO] creating new snapshot: path=/tmp/raft1561014670/snapshots/2-102-1718569595171.tmp testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: copied to local snapshot: bytes=693 testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: Installed remote snapshot testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: Installed remote snapshot testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [WARN] server-30b015ac-9bf9-5785-9244-abf831fd55ad: skipping application of old log: index=102 testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [WARN] server-30b015ac-9bf9-5785-9244-abf831fd55ad: skipping application of old log: index=102 testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.172-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.172-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" --- PASS: TestRaft_SendSnapshotFollower (0.28s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering follower state: follower="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Follower]" leader= testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering follower state: follower="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Follower]" leader= testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: entering follower state: follower="Node at 6b1b6425-c8ad-18af-bcf5-16ce97c318c5 [Follower]" leader= testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: entering follower state: follower="Node at 6b1b6425-c8ad-18af-bcf5-16ce97c318c5 [Follower]" leader= testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.177-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:35.177-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: entering follower state: follower="Node at 61080878-7c41-ecce-3952-326e5e3bf68b [Follower]" leader= testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: entering follower state: follower="Node at 61080878-7c41-ecce-3952-326e5e3bf68b [Follower]" leader= testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [WARN] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [WARN] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering candidate state: node="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Candidate]" term=2 testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering candidate state: node="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Candidate]" term=2 testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: election won: tally=2 testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: election won: tally=2 testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering leader state: leader="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Leader]" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering leader state: leader="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Leader]" testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-61080878-7c41-ecce-3952-326e5e3bf68b testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-61080878-7c41-ecce-3952-326e5e3bf68b testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.349-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.24366656 -1200 -12 m=+12.250983041 (2 nodes), 66.24922ms from start of poll, 67.05874ms from cluster start. Timeout at 2024-06-16 08:26:35.34949594 -1200 -12 m=+12.356812421, 105.82938ms after stability testing.go:149: cluster: 2024-06-16T08:26:35.349-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.24366656 -1200 -12 m=+12.250983041 (2 nodes), 66.24922ms from start of poll, 67.05874ms from cluster start. Timeout at 2024-06-16 08:26:35.34949594 -1200 -12 m=+12.356812421, 105.82938ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.349-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:35.349-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.455-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.34973602 -1200 -12 m=+12.357052541 (1 nodes), 1.54µs from start of poll, 173.12824ms from cluster start. Timeout at 2024-06-16 08:26:35.45494422 -1200 -12 m=+12.462260701, 105.20816ms after stability testing.go:149: cluster: 2024-06-16T08:26:35.455-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.34973602 -1200 -12 m=+12.357052541 (1 nodes), 1.54µs from start of poll, 173.12824ms from cluster start. Timeout at 2024-06-16 08:26:35.45494422 -1200 -12 m=+12.462260701, 105.20816ms after stability testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to pipeline appendEntries: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="append pipeline closed" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to pipeline appendEntries: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="append pipeline closed" testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to appendEntries to: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to appendEntries to: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: starting snapshot up to: index=102 testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: starting snapshot up to: index=102 testing.go:153: 2024-06-16T08:26:35.457-1200 [INFO] creating new snapshot: path=/tmp/raft2529079893/snapshots/2-102-1718569595457.tmp testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: compacting logs: from=1 to=92 testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.458-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: snapshot complete up to: index=102 testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.458-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: snapshot complete up to: index=102 testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.458-1200 [ERROR] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.458-1200 [ERROR] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: starting snapshot up to: index=99 testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: starting snapshot up to: index=99 testing.go:153: 2024-06-16T08:26:35.458-1200 [INFO] creating new snapshot: path=/tmp/raft3706296007/snapshots/2-99-1718569595458.tmp testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: compacting logs: from=1 to=92 testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: snapshot complete up to: index=99 testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: snapshot complete up to: index=99 testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.460-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to heartbeat to: peer=6b1b6425-c8ad-18af-bcf5-16ce97c318c5 error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.460-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to heartbeat to: peer=6b1b6425-c8ad-18af-bcf5-16ce97c318c5 error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.466-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to get log: index=2 error="log not found" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.466-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to get log: index=2 error="log not found" testing.go:153: 2024-06-16T08:26:35.466-1200 [INFO] creating new snapshot: path=/tmp/raft1240183984/snapshots/2-102-1718569595466.tmp testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: copied to local snapshot: bytes=693 testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: Installed remote snapshot testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: Installed remote snapshot testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.468-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.468-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.474-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:35.474-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: entering follower state: follower="Node at 59c5c5c4-b16d-c323-cd3f-1756057bd3ac [Follower]" leader= testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: entering follower state: follower="Node at 59c5c5c4-b16d-c323-cd3f-1756057bd3ac [Follower]" leader= testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=2 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=2 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.650-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.54573382 -1200 -12 m=+12.553050301 (1 nodes), 71.03842ms from start of poll, 71.89454ms from cluster start. Timeout at 2024-06-16 08:26:35.65074902 -1200 -12 m=+12.658065501, 105.0152ms after stability testing.go:149: cluster: 2024-06-16T08:26:35.650-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.54573382 -1200 -12 m=+12.553050301 (1 nodes), 71.03842ms from start of poll, 71.89454ms from cluster start. Timeout at 2024-06-16 08:26:35.65074902 -1200 -12 m=+12.658065501, 105.0152ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.656-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:35.656-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.761-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.65633362 -1200 -12 m=+12.663650101 (2 nodes), 3.12µs from start of poll, 182.49434ms from cluster start. Timeout at 2024-06-16 08:26:35.76134064 -1200 -12 m=+12.768657121, 105.00702ms after stability testing.go:149: cluster: 2024-06-16T08:26:35.761-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.65633362 -1200 -12 m=+12.663650101 (2 nodes), 3.12µs from start of poll, 182.49434ms from cluster start. Timeout at 2024-06-16 08:26:35.76134064 -1200 -12 m=+12.768657121, 105.00702ms after stability testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=RemoveServer server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr= servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=RemoveServer server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr= servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: removed peer, stopping replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=3 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: removed peer, stopping replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=3 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: heartbeat timeout reached, starting election: last-leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: heartbeat timeout reached, starting election: last-leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=3 testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=3 testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.823-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.823-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.823-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.823-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=AddStaging server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac} {Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1}]" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=AddStaging server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac} {Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1}]" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.867-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:35.867-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [ERROR] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: peer has newer term, stopping replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [ERROR] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: peer has newer term, stopping replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.868-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.868-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.910-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: Election timeout reached, restarting election testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.910-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: Election timeout reached, restarting election testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.911-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=4 testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.911-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=4 testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.911-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.911-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.911-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since our last index is greater: candidate=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=4 last-candidate-index=3 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.911-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since our last index is greater: candidate=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=4 last-candidate-index=3 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=5 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=5 testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: appendEntries rejected, sending older logs: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" next=4 testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: appendEntries rejected, sending older logs: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" next=4 testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.029-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.92372826 -1200 -12 m=+12.931044741 (1 nodes), 55.83098ms from start of poll, 449.88898ms from cluster start. Timeout at 2024-06-16 08:26:36.02949294 -1200 -12 m=+13.036809441, 105.7647ms after stability testing.go:149: cluster: 2024-06-16T08:26:36.029-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.92372826 -1200 -12 m=+12.931044741 (1 nodes), 55.83098ms from start of poll, 449.88898ms from cluster start. Timeout at 2024-06-16 08:26:36.02949294 -1200 -12 m=+13.036809441, 105.7647ms after stability testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" --- PASS: TestRaft_ReJoinFollower (0.56s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.031-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:36.031-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering follower state: follower="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Follower]" leader= testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering follower state: follower="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Follower]" leader= testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=2 testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=2 testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: election won: tally=2 testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: election won: tally=2 testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering leader state: leader="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Leader]" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering leader state: leader="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Leader]" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: added peer, starting replication: peer=server-7073535a-f1bb-b845-4065-b4fdc653862c testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: added peer, starting replication: peer=server-7073535a-f1bb-b845-4065-b4fdc653862c testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.116-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: pipelining replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.116-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: pipelining replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.221-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.1158521 -1200 -12 m=+13.123168581 (1 nodes), 84.2956ms from start of poll, 84.93222ms from cluster start. Timeout at 2024-06-16 08:26:36.22114212 -1200 -12 m=+13.228458581, 105.29ms after stability testing.go:149: cluster: 2024-06-16T08:26:36.221-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.1158521 -1200 -12 m=+13.123168581 (1 nodes), 84.2956ms from start of poll, 84.93222ms from cluster start. Timeout at 2024-06-16 08:26:36.22114212 -1200 -12 m=+13.228458581, 105.29ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.226-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:36.226-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.331-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:36.2266148 -1200 -12 m=+13.233931261 (1 nodes), 880ns from start of poll, 195.6949ms from cluster start. Timeout at 2024-06-16 08:26:36.33179496 -1200 -12 m=+13.339111441, 105.18018ms after stability testing.go:149: cluster: 2024-06-16T08:26:36.331-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:36.2266148 -1200 -12 m=+13.233931261 (1 nodes), 880ns from start of poll, 195.6949ms from cluster start. Timeout at 2024-06-16 08:26:36.33179496 -1200 -12 m=+13.339111441, 105.18018ms after stability raft_test.go:1492: [INFO] Disconnecting Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Follower] testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.332-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.332-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to pipeline appendEntries: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="append pipeline closed" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to pipeline appendEntries: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="append pipeline closed" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: aborting pipeline replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: aborting pipeline replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.343-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.343-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.354-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.354-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.360-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.360-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.375-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.375-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact: server-id=server-7073535a-f1bb-b845-4065-b4fdc653862c time=50.81704ms testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact: server-id=server-7073535a-f1bb-b845-4065-b4fdc653862c time=50.81704ms testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact quorum of nodes, stepping down testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.380-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:36.380-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.381-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.381-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: heartbeat timeout reached, starting election: last-leader=b79eafcb-5990-d1d8-594f-e3e6a8304541 testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: heartbeat timeout reached, starting election: last-leader=b79eafcb-5990-d1d8-594f-e3e6a8304541 testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=3 testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=3 testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=3 testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=3 testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.457-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.457-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=4 testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=4 testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.530-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.42558108 -1200 -12 m=+13.432897561 (0 nodes), 44.74156ms from start of poll, 394.6612ms from cluster start. Timeout at 2024-06-16 08:26:36.53077266 -1200 -12 m=+13.538089141, 105.19158ms after stability testing.go:149: cluster: 2024-06-16T08:26:36.530-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.42558108 -1200 -12 m=+13.432897561 (0 nodes), 44.74156ms from start of poll, 394.6612ms from cluster start. Timeout at 2024-06-16 08:26:36.53077266 -1200 -12 m=+13.538089141, 105.19158ms after stability testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=4 testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=4 testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=5 testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=5 testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=5 testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=5 testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" --- PASS: TestRaft_LeaderLeaseExpire (0.61s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.638-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.638-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.639-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering follower state: follower="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Follower]" leader= testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.639-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering follower state: follower="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Follower]" leader= testLoggerAdapter verbose: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" testing.go:149: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" testLoggerAdapter verbose: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: entering follower state: follower="Node at 50e866b8-aa0a-b005-cc90-24821d827da2 [Follower]" leader= testing.go:149: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: entering follower state: follower="Node at 50e866b8-aa0a-b005-cc90-24821d827da2 [Follower]" leader= testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.639-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:36.639-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader= testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader= testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [WARN] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [WARN] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [WARN] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [WARN] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering candidate state: node="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Candidate]" term=2 testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering candidate state: node="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Candidate]" term=2 testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering candidate state: node="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Candidate]" term=2 testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering candidate state: node="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Candidate]" term=2 testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: duplicate requestVote for same term: term=2 testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.695-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: duplicate requestVote for same term: term=2 testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: election won: tally=2 testing.go:149: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.695-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: duplicate requestVote for same term: term=2 testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: election won: tally=2 testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering leader state: leader="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Leader]" testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering leader state: leader="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Leader]" testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-50e866b8-aa0a-b005-cc90-24821d827da2 testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-50e866b8-aa0a-b005-cc90-24821d827da2 testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader=4fd32242-2c8f-4569-6ffc-b136a20ce6e5 testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader=4fd32242-2c8f-4569-6ffc-b136a20ce6e5 testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.800-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.6956387 -1200 -12 m=+13.702955161 (1 nodes), 55.79202ms from start of poll, 57.86948ms from cluster start. Timeout at 2024-06-16 08:26:36.80080624 -1200 -12 m=+13.808122741, 105.16758ms after stability testing.go:149: cluster: 2024-06-16T08:26:36.800-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.6956387 -1200 -12 m=+13.702955161 (1 nodes), 55.79202ms from start of poll, 57.86948ms from cluster start. Timeout at 2024-06-16 08:26:36.80080624 -1200 -12 m=+13.808122741, 105.16758ms after stability testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" testing.go:149: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" testLoggerAdapter verbose: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: entering follower state: follower="Node at a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a [Follower]" leader= testing.go:149: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: entering follower state: follower="Node at a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a [Follower]" leader= testLoggerAdapter verbose: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.815-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" testing.go:149: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.815-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" testLoggerAdapter verbose: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.816-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: entering follower state: follower="Node at 483446d2-b505-f048-e20e-d1719c98309c [Follower]" leader= testing.go:149: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.816-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: entering follower state: follower="Node at 483446d2-b505-f048-e20e-d1719c98309c [Follower]" leader= testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.816-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:36.816-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering follower state: follower="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Follower]" leader= testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering follower state: follower="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Follower]" leader= testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [WARN] server-339546d9-6142-e78a-8480-6ad5f21e1382: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [WARN] server-339546d9-6142-e78a-8480-6ad5f21e1382: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering candidate state: node="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Candidate]" term=2 testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering candidate state: node="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Candidate]" term=2 testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: election won: tally=2 testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: election won: tally=2 testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering leader state: leader="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Leader]" testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering leader state: leader="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Leader]" testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-483446d2-b505-f048-e20e-d1719c98309c testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-483446d2-b505-f048-e20e-d1719c98309c testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.975-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.87025556 -1200 -12 m=+13.877572061 (1 nodes), 53.95122ms from start of poll, 54.80834ms from cluster start. Timeout at 2024-06-16 08:26:36.97582156 -1200 -12 m=+13.983138041, 105.56598ms after stability testing.go:149: cluster: 2024-06-16T08:26:36.975-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.87025556 -1200 -12 m=+13.877572061 (1 nodes), 53.95122ms from start of poll, 54.80834ms from cluster start. Timeout at 2024-06-16 08:26:36.97582156 -1200 -12 m=+13.983138041, 105.56598ms after stability testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:37.027-1200 [ERROR] server-339546d9-6142-e78a-8480-6ad5f21e1382: failed to heartbeat to: peer=a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a error="command timed out" testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:37.027-1200 [ERROR] server-339546d9-6142-e78a-8480-6ad5f21e1382: failed to heartbeat to: peer=a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a error="command timed out" --- PASS: TestRaft_VerifyLeader (0.21s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d22b69d4-d137-d51b-7a09-477cfcb9f193 Address:d22b69d4-d137-d51b-7a09-477cfcb9f193}]" testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d22b69d4-d137-d51b-7a09-477cfcb9f193 Address:d22b69d4-d137-d51b-7a09-477cfcb9f193}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.028-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:37.028-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering follower state: follower="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Follower]" leader= testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering follower state: follower="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Follower]" leader= testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.092-1200 [WARN] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.092-1200 [WARN] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering candidate state: node="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Candidate]" term=2 testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering candidate state: node="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Candidate]" term=2 testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: election won: tally=1 testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: election won: tally=1 testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering leader state: leader="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Leader]" testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering leader state: leader="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Leader]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.199-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.09335442 -1200 -12 m=+14.100670921 (1 nodes), 64.97168ms from start of poll, 65.33102ms from cluster start. Timeout at 2024-06-16 08:26:37.19919004 -1200 -12 m=+14.206506521, 105.8356ms after stability testing.go:149: cluster: 2024-06-16T08:26:37.199-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.09335442 -1200 -12 m=+14.100670921 (1 nodes), 64.97168ms from start of poll, 65.33102ms from cluster start. Timeout at 2024-06-16 08:26:37.19919004 -1200 -12 m=+14.206506521, 105.8356ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.17s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" testing.go:149: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" testLoggerAdapter verbose: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: entering follower state: follower="Node at 7bc6efaf-433f-16bd-6dd9-b487baff2f46 [Follower]" leader= testing.go:149: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: entering follower state: follower="Node at 7bc6efaf-433f-16bd-6dd9-b487baff2f46 [Follower]" leader= testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.200-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:37.200-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering candidate state: node="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Candidate]" term=2 testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering candidate state: node="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Candidate]" term=2 testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: election won: tally=2 testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: election won: tally=2 testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering leader state: leader="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Leader]" testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering leader state: leader="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Leader]" testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: added peer, starting replication: peer=server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: added peer, starting replication: peer=server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: pipelining replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: pipelining replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.372-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.2671909 -1200 -12 m=+14.274507421 (1 nodes), 66.4374ms from start of poll, 67.04816ms from cluster start. Timeout at 2024-06-16 08:26:37.37239802 -1200 -12 m=+14.379714481, 105.20706ms after stability testing.go:149: cluster: 2024-06-16T08:26:37.372-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.2671909 -1200 -12 m=+14.274507421 (1 nodes), 66.4374ms from start of poll, 67.04816ms from cluster start. Timeout at 2024-06-16 08:26:37.37239802 -1200 -12 m=+14.379714481, 105.20706ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.372-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:37.372-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.478-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.3726227 -1200 -12 m=+14.379939181 (1 nodes), 1.3µs from start of poll, 172.47992ms from cluster start. Timeout at 2024-06-16 08:26:37.47856566 -1200 -12 m=+14.485882121, 105.94294ms after stability testing.go:149: cluster: 2024-06-16T08:26:37.478-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.3726227 -1200 -12 m=+14.379939181 (1 nodes), 1.3µs from start of poll, 172.47992ms from cluster start. Timeout at 2024-06-16 08:26:37.47856566 -1200 -12 m=+14.485882121, 105.94294ms after stability testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: new leader elected, stepping down testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: new leader elected, stepping down testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: aborting pipeline replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: aborting pipeline replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" testing.go:149: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" testLoggerAdapter verbose: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: entering follower state: follower="Node at 9302f777-e9f9-27e6-cce1-8b1b5c384063 [Follower]" leader= testing.go:149: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: entering follower state: follower="Node at 9302f777-e9f9-27e6-cce1-8b1b5c384063 [Follower]" leader= testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering follower state: follower="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Follower]" leader= testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering follower state: follower="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Follower]" leader= testLoggerAdapter verbose: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.480-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" testing.go:149: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.480-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.481-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:37.481-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.481-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: entering follower state: follower="Node at d4637939-e57b-e172-71e0-4c15fcf51275 [Follower]" leader= testing.go:149: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.481-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: entering follower state: follower="Node at d4637939-e57b-e172-71e0-4c15fcf51275 [Follower]" leader= testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [WARN] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [WARN] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering candidate state: node="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Candidate]" term=2 testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering candidate state: node="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Candidate]" term=2 testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: election won: tally=2 testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: election won: tally=2 testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering leader state: leader="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Leader]" testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering leader state: leader="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Leader]" testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-9302f777-e9f9-27e6-cce1-8b1b5c384063 testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-9302f777-e9f9-27e6-cce1-8b1b5c384063 testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-d4637939-e57b-e172-71e0-4c15fcf51275 testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-d4637939-e57b-e172-71e0-4c15fcf51275 testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.665-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.55915026 -1200 -12 m=+14.566466721 (1 nodes), 78.08304ms from start of poll, 79.02318ms from cluster start. Timeout at 2024-06-16 08:26:37.66515364 -1200 -12 m=+14.672470121, 106.0034ms after stability testing.go:149: cluster: 2024-06-16T08:26:37.665-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.55915026 -1200 -12 m=+14.566466721 (1 nodes), 78.08304ms from start of poll, 79.02318ms from cluster start. Timeout at 2024-06-16 08:26:37.66515364 -1200 -12 m=+14.672470121, 106.0034ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.670-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:37.670-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.776-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.67066182 -1200 -12 m=+14.677978301 (2 nodes), 2.5µs from start of poll, 190.53476ms from cluster start. Timeout at 2024-06-16 08:26:37.77631462 -1200 -12 m=+14.783631081, 105.65278ms after stability testing.go:149: cluster: 2024-06-16T08:26:37.776-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.67066182 -1200 -12 m=+14.677978301 (2 nodes), 2.5µs from start of poll, 190.53476ms from cluster start. Timeout at 2024-06-16 08:26:37.77631462 -1200 -12 m=+14.783631081, 105.65278ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 9302f777-e9f9-27e6-cce1-8b1b5c384063 [Follower] testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [ERROR] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: failed to heartbeat to: peer=9302f777-e9f9-27e6-cce1-8b1b5c384063 error="failed to connect to peer: 9302f777-e9f9-27e6-cce1-8b1b5c384063" testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [ERROR] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: failed to heartbeat to: peer=9302f777-e9f9-27e6-cce1-8b1b5c384063 error="failed to connect to peer: 9302f777-e9f9-27e6-cce1-8b1b5c384063" testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765 Address:8716a8e2-6f8f-d72c-c28a-bbfc84c20765}]" testing.go:149: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765 Address:8716a8e2-6f8f-d72c-c28a-bbfc84c20765}]" testLoggerAdapter verbose: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: entering leader state: leader="Node at 8716a8e2-6f8f-d72c-c28a-bbfc84c20765 [Leader]" testing.go:149: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: entering leader state: leader="Node at 8716a8e2-6f8f-d72c-c28a-bbfc84c20765 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.778-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e23deb9-1301-c7f9-7649-5ffa948073cf Address:3e23deb9-1301-c7f9-7649-5ffa948073cf}]" testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.778-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e23deb9-1301-c7f9-7649-5ffa948073cf Address:3e23deb9-1301-c7f9-7649-5ffa948073cf}]" testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.779-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering follower state: follower="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Follower]" leader= testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.779-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering follower state: follower="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Follower]" leader= testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [WARN] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [WARN] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering candidate state: node="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Candidate]" term=2 testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering candidate state: node="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Candidate]" term=2 testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: election won: tally=1 testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: election won: tally=1 testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering leader state: leader="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Leader]" testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering leader state: leader="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Leader]" --- PASS: TestRaft_NotifyCh (0.05s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering follower state: follower="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Follower]" leader= testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering follower state: follower="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Follower]" leader= testLoggerAdapter verbose: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" testing.go:149: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" testLoggerAdapter verbose: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: entering follower state: follower="Node at c0bab25e-7e19-06a0-2904-472e4a26c59e [Follower]" leader= testing.go:149: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: entering follower state: follower="Node at c0bab25e-7e19-06a0-2904-472e4a26c59e [Follower]" leader= testLoggerAdapter verbose: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" testing.go:149: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.832-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:37.832-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: entering follower state: follower="Node at d19d1332-f7ab-7cbb-7c64-ddf9777ae649 [Follower]" leader= testing.go:149: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: entering follower state: follower="Node at d19d1332-f7ab-7cbb-7c64-ddf9777ae649 [Follower]" leader= testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [WARN] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [WARN] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering candidate state: node="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Candidate]" term=2 testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering candidate state: node="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Candidate]" term=2 testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: election won: tally=2 testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: election won: tally=2 testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering leader state: leader="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Leader]" testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering leader state: leader="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Leader]" testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-c0bab25e-7e19-06a0-2904-472e4a26c59e testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-c0bab25e-7e19-06a0-2904-472e4a26c59e testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.019-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.91352908 -1200 -12 m=+14.920845561 (2 nodes), 80.748ms from start of poll, 81.56062ms from cluster start. Timeout at 2024-06-16 08:26:38.01954832 -1200 -12 m=+15.026864781, 106.01922ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.019-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.91352908 -1200 -12 m=+14.920845561 (2 nodes), 80.748ms from start of poll, 81.56062ms from cluster start. Timeout at 2024-06-16 08:26:38.01954832 -1200 -12 m=+15.026864781, 106.01922ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.019-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:38.019-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.124-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.01981172 -1200 -12 m=+15.027128181 (1 nodes), 1.8µs from start of poll, 187.84324ms from cluster start. Timeout at 2024-06-16 08:26:38.12486722 -1200 -12 m=+15.132183721, 105.05554ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.124-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.01981172 -1200 -12 m=+15.027128181 (1 nodes), 1.8µs from start of poll, 187.84324ms from cluster start. Timeout at 2024-06-16 08:26:38.12486722 -1200 -12 m=+15.132183721, 105.05554ms after stability testLoggerAdapter verbose: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:38.125-1200 [WARN] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: rejecting vote request since we have a leader: from=c0bab25e-7e19-06a0-2904-472e4a26c59e leader=fb9848aa-fe9c-019f-2f64-85cc632d6ac7 testing.go:149: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:38.125-1200 [WARN] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: rejecting vote request since we have a leader: from=c0bab25e-7e19-06a0-2904-472e4a26c59e leader=fb9848aa-fe9c-019f-2f64-85cc632d6ac7 testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" --- PASS: TestRaft_Voting (0.29s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.126-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" testing.go:149: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.126-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" testLoggerAdapter verbose: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.127-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: entering follower state: follower="Node at 142a0958-dc90-dcc8-0452-42518ebfa8cc [Follower]" leader= testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" testing.go:149: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.127-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: entering follower state: follower="Node at 142a0958-dc90-dcc8-0452-42518ebfa8cc [Follower]" leader= testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering follower state: follower="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Follower]" leader= testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering follower state: follower="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Follower]" leader= testLoggerAdapter verbose: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" testing.go:149: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.127-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:38.127-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: entering follower state: follower="Node at 4000368a-7fef-2bd9-a4f7-3cbc7723cec3 [Follower]" leader= testing.go:149: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: entering follower state: follower="Node at 4000368a-7fef-2bd9-a4f7-3cbc7723cec3 [Follower]" leader= testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.181-1200 [WARN] server-f80f0a21-23cf-101c-34bb-29cc543c36df: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.181-1200 [WARN] server-f80f0a21-23cf-101c-34bb-29cc543c36df: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering candidate state: node="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Candidate]" term=2 testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering candidate state: node="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Candidate]" term=2 testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: election won: tally=2 testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: election won: tally=2 testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering leader state: leader="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Leader]" testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering leader state: leader="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Leader]" testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-142a0958-dc90-dcc8-0452-42518ebfa8cc testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-142a0958-dc90-dcc8-0452-42518ebfa8cc testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.288-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:38.1823694 -1200 -12 m=+15.189685881 (2 nodes), 54.74634ms from start of poll, 55.74502ms from cluster start. Timeout at 2024-06-16 08:26:38.28809394 -1200 -12 m=+15.295410421, 105.72454ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.288-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:38.1823694 -1200 -12 m=+15.189685881 (2 nodes), 54.74634ms from start of poll, 55.74502ms from cluster start. Timeout at 2024-06-16 08:26:38.28809394 -1200 -12 m=+15.295410421, 105.72454ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.288-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:38.288-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.393-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.2882826 -1200 -12 m=+15.295599061 (1 nodes), 1.68µs from start of poll, 161.6582ms from cluster start. Timeout at 2024-06-16 08:26:38.39366024 -1200 -12 m=+15.400976721, 105.37766ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.393-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.2882826 -1200 -12 m=+15.295599061 (1 nodes), 1.68µs from start of poll, 161.6582ms from cluster start. Timeout at 2024-06-16 08:26:38.39366024 -1200 -12 m=+15.400976721, 105.37766ms after stability testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" testing.go:149: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" testLoggerAdapter verbose: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: entering follower state: follower="Node at d4477fa5-ec15-842a-b091-51c27397c59c [Follower]" leader= testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering follower state: follower="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Follower]" leader= testing.go:149: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: entering follower state: follower="Node at d4477fa5-ec15-842a-b091-51c27397c59c [Follower]" leader= testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering follower state: follower="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Follower]" leader= testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: initial configuration: index=0 servers=[] testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.395-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:38.395-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: entering follower state: follower="Node at b0359b58-f66c-2b3d-f4ac-c9646ef95e9d [Follower]" leader= testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: entering follower state: follower="Node at b0359b58-f66c-2b3d-f4ac-c9646ef95e9d [Follower]" leader= testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: heartbeat timeout reached, starting election: last-leader= testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering candidate state: node="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Candidate]" term=2 testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering candidate state: node="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Candidate]" term=2 testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: election won: tally=2 testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: election won: tally=2 testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering leader state: leader="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Leader]" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering leader state: leader="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Leader]" testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=d4477fa5-ec15-842a-b091-51c27397c59c testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=d4477fa5-ec15-842a-b091-51c27397c59c testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.450-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: no known peers, aborting election testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.450-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.552-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.44743852 -1200 -12 m=+15.454755001 (1 nodes), 51.62112ms from start of poll, 52.45526ms from cluster start. Timeout at 2024-06-16 08:26:38.552614 -1200 -12 m=+15.559930481, 105.17548ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.552-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.44743852 -1200 -12 m=+15.454755001 (1 nodes), 51.62112ms from start of poll, 52.45526ms from cluster start. Timeout at 2024-06-16 08:26:38.552614 -1200 -12 m=+15.559930481, 105.17548ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.552-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:38.552-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.658-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.55281652 -1200 -12 m=+15.560132981 (1 nodes), 1.72µs from start of poll, 157.83324ms from cluster start. Timeout at 2024-06-16 08:26:38.6579564 -1200 -12 m=+15.665272881, 105.1399ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.658-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.55281652 -1200 -12 m=+15.560132981 (1 nodes), 1.72µs from start of poll, 157.83324ms from cluster start. Timeout at 2024-06-16 08:26:38.6579564 -1200 -12 m=+15.665272881, 105.1399ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.658-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:38.658-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.763-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.6581582 -1200 -12 m=+15.665474681 (1 nodes), 2.1µs from start of poll, 263.17494ms from cluster start. Timeout at 2024-06-16 08:26:38.76317236 -1200 -12 m=+15.770488841, 105.01416ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.763-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.6581582 -1200 -12 m=+15.665474681 (1 nodes), 2.1µs from start of poll, 263.17494ms from cluster start. Timeout at 2024-06-16 08:26:38.76317236 -1200 -12 m=+15.770488841, 105.01416ms after stability testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=AddStaging server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1} {Suffrage:Voter ID:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d Address:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}]" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=AddStaging server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1} {Suffrage:Voter ID:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d Address:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}]" testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.763-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.763-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: appendEntries rejected, sending older logs: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" next=1 testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: appendEntries rejected, sending older logs: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" next=1 testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.769-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:38.769-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.874-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.76967666 -1200 -12 m=+15.776993121 (1 nodes), 1.48µs from start of poll, 374.69338ms from cluster start. Timeout at 2024-06-16 08:26:38.87471556 -1200 -12 m=+15.882032061, 105.03894ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.874-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.76967666 -1200 -12 m=+15.776993121 (1 nodes), 1.48µs from start of poll, 374.69338ms from cluster start. Timeout at 2024-06-16 08:26:38.87471556 -1200 -12 m=+15.882032061, 105.03894ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.874-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:38.874-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.980-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.87489558 -1200 -12 m=+15.882212061 (1 nodes), 1.76µs from start of poll, 479.91232ms from cluster start. Timeout at 2024-06-16 08:26:38.9805888 -1200 -12 m=+15.987905261, 105.6932ms after stability testing.go:149: cluster: 2024-06-16T08:26:38.980-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.87489558 -1200 -12 m=+15.882212061 (1 nodes), 1.76µs from start of poll, 479.91232ms from cluster start. Timeout at 2024-06-16 08:26:38.9805888 -1200 -12 m=+15.987905261, 105.6932ms after stability testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=RemoveServer server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr= servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=RemoveServer server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr= servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: removed peer, stopping replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d last-index=4 testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: removed peer, stopping replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d last-index=4 testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.59s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering follower state: follower="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Follower]" leader= testLoggerAdapter verbose: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering follower state: follower="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Follower]" leader= testing.go:149: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.982-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: entering follower state: follower="Node at 6e1ca5ec-74f1-904c-7558-ddcf468882f0 [Follower]" leader= testing.go:149: cluster: 2024-06-16T08:26:38.982-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: entering follower state: follower="Node at 6e1ca5ec-74f1-904c-7558-ddcf468882f0 [Follower]" leader= testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: heartbeat timeout reached, starting election: last-leader= testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering candidate state: node="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Candidate]" term=2 testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering candidate state: node="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Candidate]" term=2 testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: election won: tally=2 testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: election won: tally=2 testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering leader state: leader="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Leader]" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering leader state: leader="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Leader]" testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.147-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:39.04115154 -1200 -12 m=+16.048468021 (1 nodes), 58.53756ms from start of poll, 59.1523ms from cluster start. Timeout at 2024-06-16 08:26:39.14717118 -1200 -12 m=+16.154487661, 106.01964ms after stability testing.go:149: cluster: 2024-06-16T08:26:39.147-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:39.04115154 -1200 -12 m=+16.048468021 (1 nodes), 58.53756ms from start of poll, 59.1523ms from cluster start. Timeout at 2024-06-16 08:26:39.14717118 -1200 -12 m=+16.154487661, 106.01964ms after stability testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: initial configuration: index=0 servers=[] testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.147-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:39.147-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: entering follower state: follower="Node at 2d998396-b25c-5a51-4324-f0d7aebaad6b [Follower]" leader= testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: entering follower state: follower="Node at 2d998396-b25c-5a51-4324-f0d7aebaad6b [Follower]" leader= testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.201-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: no known peers, aborting election testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.201-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.252-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.1477657 -1200 -12 m=+16.155082181 (1 nodes), 1.12µs from start of poll, 165.76646ms from cluster start. Timeout at 2024-06-16 08:26:39.25285324 -1200 -12 m=+16.260169701, 105.08752ms after stability testing.go:149: cluster: 2024-06-16T08:26:39.252-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.1477657 -1200 -12 m=+16.155082181 (1 nodes), 1.12µs from start of poll, 165.76646ms from cluster start. Timeout at 2024-06-16 08:26:39.25285324 -1200 -12 m=+16.260169701, 105.08752ms after stability testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=AddStaging server-id=server-2d998396-b25c-5a51-4324-f0d7aebaad6b server-addr=2d998396-b25c-5a51-4324-f0d7aebaad6b servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=AddStaging server-id=server-2d998396-b25c-5a51-4324-f0d7aebaad6b server-addr=2d998396-b25c-5a51-4324-f0d7aebaad6b servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=server-2d998396-b25c-5a51-4324-f0d7aebaad6b testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=server-2d998396-b25c-5a51-4324-f0d7aebaad6b testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.253-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.253-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: appendEntries rejected, sending older logs: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" next=1 testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: appendEntries rejected, sending older logs: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" next=1 testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.259-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:39.259-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.365-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.2593279 -1200 -12 m=+16.266644361 (1 nodes), 1.6µs from start of poll, 277.32864ms from cluster start. Timeout at 2024-06-16 08:26:39.36508228 -1200 -12 m=+16.372398761, 105.7544ms after stability testing.go:149: cluster: 2024-06-16T08:26:39.365-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.2593279 -1200 -12 m=+16.266644361 (1 nodes), 1.6µs from start of poll, 277.32864ms from cluster start. Timeout at 2024-06-16 08:26:39.36508228 -1200 -12 m=+16.372398761, 105.7544ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.365-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:39.365-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.471-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.36526892 -1200 -12 m=+16.372585381 (1 nodes), 1.24µs from start of poll, 383.26966ms from cluster start. Timeout at 2024-06-16 08:26:39.47114692 -1200 -12 m=+16.478463401, 105.87802ms after stability testing.go:149: cluster: 2024-06-16T08:26:39.471-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.36526892 -1200 -12 m=+16.372585381 (1 nodes), 1.24µs from start of poll, 383.26966ms from cluster start. Timeout at 2024-06-16 08:26:39.47114692 -1200 -12 m=+16.478463401, 105.87802ms after stability testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=RemoveServer server-id=6e1ca5ec-74f1-904c-7558-ddcf468882f0 server-addr= servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=RemoveServer server-id=6e1ca5ec-74f1-904c-7558-ddcf468882f0 server-addr= servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: removed peer, stopping replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 last-index=4 testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: removed peer, stopping replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 last-index=4 testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.49s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering follower state: follower="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Follower]" leader= testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering follower state: follower="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Follower]" leader= testLoggerAdapter verbose: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" testing.go:149: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.473-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:39.473-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: entering follower state: follower="Node at 54ba0f95-76fd-ffb5-6f29-5753657abb1a [Follower]" leader= testing.go:149: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: entering follower state: follower="Node at 54ba0f95-76fd-ffb5-6f29-5753657abb1a [Follower]" leader= testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [WARN] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [WARN] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering candidate state: node="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Candidate]" term=2 testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering candidate state: node="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Candidate]" term=2 testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: election won: tally=2 testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: election won: tally=2 testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering leader state: leader="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Leader]" testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering leader state: leader="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Leader]" testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-9738814d-4416-3c04-4fee-62a746a4a56d testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-9738814d-4416-3c04-4fee-62a746a4a56d testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.638-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.53328876 -1200 -12 m=+16.540605261 (1 nodes), 59.61988ms from start of poll, 60.40556ms from cluster start. Timeout at 2024-06-16 08:26:39.63851342 -1200 -12 m=+16.645829881, 105.22462ms after stability testing.go:149: cluster: 2024-06-16T08:26:39.638-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.53328876 -1200 -12 m=+16.540605261 (1 nodes), 59.61988ms from start of poll, 60.40556ms from cluster start. Timeout at 2024-06-16 08:26:39.63851342 -1200 -12 m=+16.645829881, 105.22462ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.638-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:39.638-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.744-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.63871358 -1200 -12 m=+16.646030041 (1 nodes), 1.38µs from start of poll, 165.83034ms from cluster start. Timeout at 2024-06-16 08:26:39.74405586 -1200 -12 m=+16.751372361, 105.34232ms after stability testing.go:149: cluster: 2024-06-16T08:26:39.744-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.63871358 -1200 -12 m=+16.646030041 (1 nodes), 1.38µs from start of poll, 165.83034ms from cluster start. Timeout at 2024-06-16 08:26:39.74405586 -1200 -12 m=+16.751372361, 105.34232ms after stability testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering candidate state: node="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Candidate]" term=3 testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering candidate state: node="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.744-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:39.744-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: election won: tally=2 testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: election won: tally=2 testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering leader state: leader="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Leader]" testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering leader state: leader="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Leader]" testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.850-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.74465376 -1200 -12 m=+16.751970241 (1 nodes), 170.8µs from start of poll, 271.77054ms from cluster start. Timeout at 2024-06-16 08:26:39.85048896 -1200 -12 m=+16.857805441, 105.8352ms after stability testing.go:149: cluster: 2024-06-16T08:26:39.850-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.74465376 -1200 -12 m=+16.751970241 (1 nodes), 170.8µs from start of poll, 271.77054ms from cluster start. Timeout at 2024-06-16 08:26:39.85048896 -1200 -12 m=+16.857805441, 105.8352ms after stability testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.901-1200 [ERROR] server-9738814d-4416-3c04-4fee-62a746a4a56d: failed to heartbeat to: peer=2dc81aac-7f82-a03d-abb5-f7d064fdf622 error="command timed out" testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.901-1200 [ERROR] server-9738814d-4416-3c04-4fee-62a746a4a56d: failed to heartbeat to: peer=2dc81aac-7f82-a03d-abb5-f7d064fdf622 error="command timed out" --- PASS: TestRaft_LeadershipTransfer (0.43s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9db2393c-0ada-1268-71ef-bf3e6fa8b556 Address:9db2393c-0ada-1268-71ef-bf3e6fa8b556}]" testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9db2393c-0ada-1268-71ef-bf3e6fa8b556 Address:9db2393c-0ada-1268-71ef-bf3e6fa8b556}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.902-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:39.902-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering follower state: follower="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Follower]" leader= testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering follower state: follower="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Follower]" leader= testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [WARN] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [WARN] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering candidate state: node="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Candidate]" term=2 testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering candidate state: node="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Candidate]" term=2 testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: election won: tally=1 testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: election won: tally=1 testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering leader state: leader="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Leader]" testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering leader state: leader="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Leader]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.103-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.99765244 -1200 -12 m=+17.004968941 (1 nodes), 95.44664ms from start of poll, 95.81222ms from cluster start. Timeout at 2024-06-16 08:26:40.10362474 -1200 -12 m=+17.110941221, 105.97228ms after stability testing.go:149: cluster: 2024-06-16T08:26:40.103-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.99765244 -1200 -12 m=+17.004968941 (1 nodes), 95.44664ms from start of poll, 95.81222ms from cluster start. Timeout at 2024-06-16 08:26:40.10362474 -1200 -12 m=+17.110941221, 105.97228ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.20s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering follower state: follower="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Follower]" leader= testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering follower state: follower="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Follower]" leader= testLoggerAdapter verbose: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testing.go:149: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testLoggerAdapter verbose: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: entering follower state: follower="Node at 1b19f367-68b0-83c5-a5af-93a15af4b137 [Follower]" leader= testing.go:149: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: entering follower state: follower="Node at 1b19f367-68b0-83c5-a5af-93a15af4b137 [Follower]" leader= testLoggerAdapter verbose: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testing.go:149: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testLoggerAdapter verbose: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: entering follower state: follower="Node at d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 [Follower]" leader= testing.go:149: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: entering follower state: follower="Node at d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 [Follower]" leader= testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= testLoggerAdapter verbose: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testing.go:149: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testLoggerAdapter verbose: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: entering follower state: follower="Node at adef2612-db49-5c2a-48bb-68bddcfa669c [Follower]" leader= testing.go:149: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: entering follower state: follower="Node at adef2612-db49-5c2a-48bb-68bddcfa669c [Follower]" leader= testLoggerAdapter verbose: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.107-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testing.go:149: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.107-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testLoggerAdapter verbose: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.108-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: entering follower state: follower="Node at 7d2164a7-6111-1ca9-8553-6b9b14836751 [Follower]" leader= testing.go:149: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.108-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: entering follower state: follower="Node at 7d2164a7-6111-1ca9-8553-6b9b14836751 [Follower]" leader= testLoggerAdapter verbose: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testing.go:149: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.108-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:40.108-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: entering follower state: follower="Node at 7cc58c3c-784b-80b8-d215-af553a5127ea [Follower]" leader= testing.go:149: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: entering follower state: follower="Node at 7cc58c3c-784b-80b8-d215-af553a5127ea [Follower]" leader= testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.158-1200 [WARN] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.158-1200 [WARN] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering candidate state: node="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Candidate]" term=2 testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering candidate state: node="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Candidate]" term=2 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: election won: tally=4 testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: election won: tally=4 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering leader state: leader="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Leader]" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering leader state: leader="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Leader]" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.160-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.160-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.264-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.15942808 -1200 -12 m=+17.166744561 (1 nodes), 50.9432ms from start of poll, 53.26924ms from cluster start. Timeout at 2024-06-16 08:26:40.26467756 -1200 -12 m=+17.271994021, 105.24946ms after stability testing.go:149: cluster: 2024-06-16T08:26:40.264-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.15942808 -1200 -12 m=+17.166744561 (1 nodes), 50.9432ms from start of poll, 53.26924ms from cluster start. Timeout at 2024-06-16 08:26:40.26467756 -1200 -12 m=+17.271994021, 105.24946ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.264-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:40.264-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.370-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.26489708 -1200 -12 m=+17.272213561 (6 nodes), 5.08µs from start of poll, 158.73824ms from cluster start. Timeout at 2024-06-16 08:26:40.37024978 -1200 -12 m=+17.377566241, 105.35268ms after stability testing.go:149: cluster: 2024-06-16T08:26:40.370-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.26489708 -1200 -12 m=+17.272213561 (6 nodes), 5.08µs from start of poll, 158.73824ms from cluster start. Timeout at 2024-06-16 08:26:40.37024978 -1200 -12 m=+17.377566241, 105.35268ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.370-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:40.370-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.476-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.3705117 -1200 -12 m=+17.377828181 (1 nodes), 2.08µs from start of poll, 264.35286ms from cluster start. Timeout at 2024-06-16 08:26:40.47643874 -1200 -12 m=+17.483755221, 105.92704ms after stability testing.go:149: cluster: 2024-06-16T08:26:40.476-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.3705117 -1200 -12 m=+17.377828181 (1 nodes), 2.08µs from start of poll, 264.35286ms from cluster start. Timeout at 2024-06-16 08:26:40.47643874 -1200 -12 m=+17.483755221, 105.92704ms after stability testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering candidate state: node="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Candidate]" term=3 testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering candidate state: node="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.476-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:40.476-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: election won: tally=4 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: election won: tally=4 testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering leader state: leader="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Leader]" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering leader state: leader="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Leader]" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-3b81aed2-de30-b2fd-178b-c0547516bc6f testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-3b81aed2-de30-b2fd-178b-c0547516bc6f testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.582-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.47700508 -1200 -12 m=+17.484321561 (1 nodes), 133.66µs from start of poll, 370.84624ms from cluster start. Timeout at 2024-06-16 08:26:40.58212122 -1200 -12 m=+17.589437721, 105.11616ms after stability testing.go:149: cluster: 2024-06-16T08:26:40.582-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.47700508 -1200 -12 m=+17.484321561 (1 nodes), 133.66µs from start of poll, 370.84624ms from cluster start. Timeout at 2024-06-16 08:26:40.58212122 -1200 -12 m=+17.589437721, 105.11616ms after stability testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" testing.go:149: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" testLoggerAdapter verbose: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: entering follower state: follower="Node at 24e88ed1-84f4-62d8-524a-b72536736910 [Follower]" leader= testing.go:149: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: entering follower state: follower="Node at 24e88ed1-84f4-62d8-524a-b72536736910 [Follower]" leader= testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering follower state: follower="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Follower]" leader= testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering follower state: follower="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Follower]" leader= testLoggerAdapter verbose: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" testing.go:149: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.584-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:40.584-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: entering follower state: follower="Node at 92a4c800-1d35-1df7-2f45-bb26af00e0d0 [Follower]" leader= testing.go:149: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: entering follower state: follower="Node at 92a4c800-1d35-1df7-2f45-bb26af00e0d0 [Follower]" leader= testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [WARN] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [WARN] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering candidate state: node="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Candidate]" term=2 testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering candidate state: node="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Candidate]" term=2 testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: election won: tally=2 testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: election won: tally=2 testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering leader state: leader="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Leader]" testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering leader state: leader="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Leader]" testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-24e88ed1-84f4-62d8-524a-b72536736910 testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-24e88ed1-84f4-62d8-524a-b72536736910 testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.743-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.63782152 -1200 -12 m=+17.645138001 (1 nodes), 52.85372ms from start of poll, 53.68132ms from cluster start. Timeout at 2024-06-16 08:26:40.74379874 -1200 -12 m=+17.751115221, 105.97722ms after stability testing.go:149: cluster: 2024-06-16T08:26:40.743-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.63782152 -1200 -12 m=+17.645138001 (1 nodes), 52.85372ms from start of poll, 53.68132ms from cluster start. Timeout at 2024-06-16 08:26:40.74379874 -1200 -12 m=+17.751115221, 105.97722ms after stability testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" testing.go:149: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" testLoggerAdapter verbose: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: entering follower state: follower="Node at cd678b2b-104a-f4db-7c80-817e7a97742e [Follower]" leader= testing.go:149: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: entering follower state: follower="Node at cd678b2b-104a-f4db-7c80-817e7a97742e [Follower]" leader= testLoggerAdapter verbose: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" testing.go:149: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" testLoggerAdapter verbose: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: entering follower state: follower="Node at be344f10-26a5-ee75-bc9d-aa2e88ff747d [Follower]" leader= testing.go:149: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: entering follower state: follower="Node at be344f10-26a5-ee75-bc9d-aa2e88ff747d [Follower]" leader= testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.745-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:40.745-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering follower state: follower="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Follower]" leader= testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering follower state: follower="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Follower]" leader= testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [WARN] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [WARN] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering candidate state: node="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Candidate]" term=2 testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering candidate state: node="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Candidate]" term=2 testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: election won: tally=2 testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: election won: tally=2 testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering leader state: leader="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Leader]" testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering leader state: leader="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Leader]" testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-cd678b2b-104a-f4db-7c80-817e7a97742e testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-cd678b2b-104a-f4db-7c80-817e7a97742e testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-be344f10-26a5-ee75-bc9d-aa2e88ff747d testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-be344f10-26a5-ee75-bc9d-aa2e88ff747d testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.935-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.83079336 -1200 -12 m=+17.838109981 (2 nodes), 84.88246ms from start of poll, 85.67246ms from cluster start. Timeout at 2024-06-16 08:26:40.935897 -1200 -12 m=+17.943213481, 105.1035ms after stability testing.go:149: cluster: 2024-06-16T08:26:40.935-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.83079336 -1200 -12 m=+17.838109981 (2 nodes), 84.88246ms from start of poll, 85.67246ms from cluster start. Timeout at 2024-06-16 08:26:40.935897 -1200 -12 m=+17.943213481, 105.1035ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.936-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:40.936-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.041-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.93641662 -1200 -12 m=+17.943733081 (1 nodes), 1.48µs from start of poll, 191.29556ms from cluster start. Timeout at 2024-06-16 08:26:41.04150304 -1200 -12 m=+18.048819521, 105.08644ms after stability testing.go:149: cluster: 2024-06-16T08:26:41.041-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.93641662 -1200 -12 m=+17.943733081 (1 nodes), 1.48µs from start of poll, 191.29556ms from cluster start. Timeout at 2024-06-16 08:26:41.04150304 -1200 -12 m=+18.048819521, 105.08644ms after stability testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.30s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering follower state: follower="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Follower]" leader= testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering follower state: follower="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Follower]" leader= testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= testLoggerAdapter verbose: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" testing.go:149: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.043-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:41.043-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: entering follower state: follower="Node at 56c05aba-b15b-4657-5714-48d65e7bf55d [Follower]" leader= testing.go:149: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: entering follower state: follower="Node at 56c05aba-b15b-4657-5714-48d65e7bf55d [Follower]" leader= testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [WARN] server-98584a7b-babb-a728-bde5-c6725db92476: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [WARN] server-98584a7b-babb-a728-bde5-c6725db92476: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering candidate state: node="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Candidate]" term=2 testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering candidate state: node="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Candidate]" term=2 testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: election won: tally=2 testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: election won: tally=2 testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering leader state: leader="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Leader]" testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering leader state: leader="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Leader]" testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-e2fc4e69-3c0e-96b6-b518-1f46772f472a testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-e2fc4e69-3c0e-96b6-b518-1f46772f472a testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.215-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.10971026 -1200 -12 m=+18.117026741 (1 nodes), 65.81598ms from start of poll, 66.67466ms from cluster start. Timeout at 2024-06-16 08:26:41.21543002 -1200 -12 m=+18.222746501, 105.71976ms after stability testing.go:149: cluster: 2024-06-16T08:26:41.215-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.10971026 -1200 -12 m=+18.117026741 (1 nodes), 65.81598ms from start of poll, 66.67466ms from cluster start. Timeout at 2024-06-16 08:26:41.21543002 -1200 -12 m=+18.222746501, 105.71976ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.215-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:41.215-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.321-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.21560404 -1200 -12 m=+18.222920501 (2 nodes), 2.26µs from start of poll, 172.56842ms from cluster start. Timeout at 2024-06-16 08:26:41.3211377 -1200 -12 m=+18.328454161, 105.53366ms after stability testing.go:149: cluster: 2024-06-16T08:26:41.321-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.21560404 -1200 -12 m=+18.222920501 (2 nodes), 2.26µs from start of poll, 172.56842ms from cluster start. Timeout at 2024-06-16 08:26:41.3211377 -1200 -12 m=+18.328454161, 105.53366ms after stability testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering candidate state: node="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Candidate]" term=3 testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering candidate state: node="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.354-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:41.354-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: election won: tally=2 testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: election won: tally=2 testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering leader state: leader="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Leader]" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering leader state: leader="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Leader]" testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-98584a7b-babb-a728-bde5-c6725db92476 testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-98584a7b-babb-a728-bde5-c6725db92476 testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.355-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.355-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.460-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.35497818 -1200 -12 m=+18.362294641 (1 nodes), 110µs from start of poll, 311.94256ms from cluster start. Timeout at 2024-06-16 08:26:41.460514 -1200 -12 m=+18.467830481, 105.53584ms after stability testing.go:149: cluster: 2024-06-16T08:26:41.460-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.35497818 -1200 -12 m=+18.362294641 (1 nodes), 110µs from start of poll, 311.94256ms from cluster start. Timeout at 2024-06-16 08:26:41.460514 -1200 -12 m=+18.467830481, 105.53584ms after stability testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering follower state: follower="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Follower]" leader= testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering follower state: follower="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Follower]" leader= testLoggerAdapter verbose: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" testing.go:149: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" testLoggerAdapter verbose: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: entering follower state: follower="Node at 9754a28d-e77b-5efb-331f-a7162c327381 [Follower]" leader= testing.go:149: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: entering follower state: follower="Node at 9754a28d-e77b-5efb-331f-a7162c327381 [Follower]" leader= testLoggerAdapter verbose: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" testing.go:149: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.462-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:41.462-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: entering follower state: follower="Node at d68c3943-0312-dc4c-b744-b153b7a95ede [Follower]" leader= testing.go:149: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: entering follower state: follower="Node at d68c3943-0312-dc4c-b744-b153b7a95ede [Follower]" leader= testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [WARN] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [WARN] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering candidate state: node="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Candidate]" term=2 testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering candidate state: node="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Candidate]" term=2 testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: election won: tally=2 testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: election won: tally=2 testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering leader state: leader="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Leader]" testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering leader state: leader="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Leader]" testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-9754a28d-e77b-5efb-331f-a7162c327381 testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-9754a28d-e77b-5efb-331f-a7162c327381 testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-d68c3943-0312-dc4c-b744-b153b7a95ede testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-d68c3943-0312-dc4c-b744-b153b7a95ede testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.637-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.53156266 -1200 -12 m=+18.538879141 (1 nodes), 68.77614ms from start of poll, 69.6601ms from cluster start. Timeout at 2024-06-16 08:26:41.63734374 -1200 -12 m=+18.644660221, 105.78108ms after stability testing.go:149: cluster: 2024-06-16T08:26:41.637-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.53156266 -1200 -12 m=+18.538879141 (1 nodes), 68.77614ms from start of poll, 69.6601ms from cluster start. Timeout at 2024-06-16 08:26:41.63734374 -1200 -12 m=+18.644660221, 105.78108ms after stability testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: cannot transfer leadership to itself testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: cannot transfer leadership to itself testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" --- PASS: TestRaft_LeadershipTransferToItself (0.18s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.639-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.639-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" testLoggerAdapter verbose: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.639-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" testing.go:149: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.639-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" testLoggerAdapter verbose: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" testing.go:149: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.640-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:41.640-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.640-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering follower state: follower="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Follower]" leader= testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.640-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering follower state: follower="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Follower]" leader= testLoggerAdapter verbose: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.640-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: entering follower state: follower="Node at 3738d183-b7d0-fdce-2491-5120c847ada3 [Follower]" leader= testing.go:149: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.640-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: entering follower state: follower="Node at 3738d183-b7d0-fdce-2491-5120c847ada3 [Follower]" leader= testLoggerAdapter verbose: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: entering follower state: follower="Node at e3ef3ece-af13-5659-8468-bb183482d52c [Follower]" leader= testing.go:149: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: entering follower state: follower="Node at e3ef3ece-af13-5659-8468-bb183482d52c [Follower]" leader= testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [WARN] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [WARN] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering candidate state: node="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Candidate]" term=2 testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering candidate state: node="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Candidate]" term=2 testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: election won: tally=2 testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: election won: tally=2 testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering leader state: leader="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Leader]" testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering leader state: leader="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Leader]" testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-3738d183-b7d0-fdce-2491-5120c847ada3 testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-3738d183-b7d0-fdce-2491-5120c847ada3 testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-e3ef3ece-af13-5659-8468-bb183482d52c testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-e3ef3ece-af13-5659-8468-bb183482d52c testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.802-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.69726464 -1200 -12 m=+18.704581101 (1 nodes), 56.9132ms from start of poll, 57.92894ms from cluster start. Timeout at 2024-06-16 08:26:41.80247928 -1200 -12 m=+18.809795761, 105.21466ms after stability testing.go:149: cluster: 2024-06-16T08:26:41.802-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.69726464 -1200 -12 m=+18.704581101 (1 nodes), 56.9132ms from start of poll, 57.92894ms from cluster start. Timeout at 2024-06-16 08:26:41.80247928 -1200 -12 m=+18.809795761, 105.21466ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.802-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:41.802-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.908-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.80277744 -1200 -12 m=+18.810093921 (2 nodes), 7.68µs from start of poll, 163.44176ms from cluster start. Timeout at 2024-06-16 08:26:41.90839514 -1200 -12 m=+18.915711641, 105.61772ms after stability testing.go:149: cluster: 2024-06-16T08:26:41.908-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.80277744 -1200 -12 m=+18.810093921 (2 nodes), 7.68µs from start of poll, 163.44176ms from cluster start. Timeout at 2024-06-16 08:26:41.90839514 -1200 -12 m=+18.915711641, 105.61772ms after stability testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.71s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering follower state: follower="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Follower]" leader= testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering follower state: follower="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Follower]" leader= testLoggerAdapter verbose: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" testing.go:149: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.346-1200 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-16T08:26:42.346-1200 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: entering follower state: follower="Node at 6f2f7c51-f298-28a0-e5d0-3a74a07d729a [Follower]" leader= testing.go:149: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: entering follower state: follower="Node at 6f2f7c51-f298-28a0-e5d0-3a74a07d729a [Follower]" leader= testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [WARN] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [WARN] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering candidate state: node="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Candidate]" term=2 testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering candidate state: node="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Candidate]" term=2 testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: election won: tally=2 testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: election won: tally=2 testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering leader state: leader="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Leader]" testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering leader state: leader="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Leader]" testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: added peer, starting replication: peer=server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: added peer, starting replication: peer=server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.404-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: pipelining replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.404-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: pipelining replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.510-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:42.40390224 -1200 -12 m=+19.411218721 (1 nodes), 56.91232ms from start of poll, 57.61294ms from cluster start. Timeout at 2024-06-16 08:26:42.50997246 -1200 -12 m=+19.517288941, 106.07022ms after stability testing.go:149: cluster: 2024-06-16T08:26:42.510-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:42.40390224 -1200 -12 m=+19.411218721 (1 nodes), 56.91232ms from start of poll, 57.61294ms from cluster start. Timeout at 2024-06-16 08:26:42.50997246 -1200 -12 m=+19.517288941, 106.07022ms after stability testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.510-1200 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-16T08:26:42.510-1200 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.615-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:42.51022702 -1200 -12 m=+19.517543501 (1 nodes), 2µs from start of poll, 163.93772ms from cluster start. Timeout at 2024-06-16 08:26:42.6156204 -1200 -12 m=+19.622936881, 105.39338ms after stability testing.go:149: cluster: 2024-06-16T08:26:42.615-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:42.51022702 -1200 -12 m=+19.517543501 (1 nodes), 2µs from start of poll, 163.93772ms from cluster start. Timeout at 2024-06-16 08:26:42.6156204 -1200 -12 m=+19.622936881, 105.39338ms after stability testLoggerAdapter verbose: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.627-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" testing.go:149: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.627-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.627-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: aborting pipeline replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.627-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: aborting pipeline replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 19.655s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.1.2-1_arm64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/1861 and its subdirectories I: Current time: Sun Jun 16 08:26:53 -12 2024 I: pbuilder-time-stamp: 1718569613 Mon May 15 14:03:55 UTC 2023 I: 1st build successful. Starting 2nd build on remote node codethink10-arm64.debian.net. Mon May 15 14:03:55 UTC 2023 I: Preparing to do remote build '2' on codethink10-arm64.debian.net. Mon May 15 14:06:38 UTC 2023 I: Deleting $TMPDIR on codethink10-arm64.debian.net. Mon May 15 14:06:39 UTC 2023 I: golang-github-hashicorp-raft_1.1.2-1_arm64.changes: Format: 1.8 Date: Sun, 19 Jan 2020 18:15:42 +1100 Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Distribution: unstable Urgency: medium Maintainer: Debian Go Packaging Team Changed-By: Dmitry Smirnov Description: golang-github-hashicorp-raft-dev - implementation of the Raft consensus protocol Changes: golang-github-hashicorp-raft (1.1.2-1) unstable; urgency=medium . * New upstream release. Checksums-Sha1: 4fcae6db8ecb88479bbb266ab817605e714c9d9e 90036 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 0a897f2cbb7f369ac942eebe4ff485d08969d08f 8240 golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo Checksums-Sha256: 579a17edb3722463f068b5ad1c77fca12707493b919392d4cb32cd828d42473e 90036 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 0d7f48894de0f3748b138fca03352dc3ba49660686be291e9cbac4d6aeeb5a60 8240 golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo Files: d2b1b6458f45e5e0d928d454aeb4deeb 90036 devel optional golang-github-hashicorp-raft-dev_1.1.2-1_all.deb df08adc72cabaf243665f87a1c0a149d 8240 devel optional golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo Mon May 15 14:06:46 UTC 2023 I: diffoscope 242 will be used to compare the two builds: # Profiling output for: /usr/bin/diffoscope --timeout 7200 --html /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/golang-github-hashicorp-raft_1.1.2-1.diffoscope.html --text /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/golang-github-hashicorp-raft_1.1.2-1.diffoscope.txt --json /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/golang-github-hashicorp-raft_1.1.2-1.diffoscope.json --profile=- /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/b1/golang-github-hashicorp-raft_1.1.2-1_arm64.changes /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/b2/golang-github-hashicorp-raft_1.1.2-1_arm64.changes ## command (total time: 0.000s) 0.000s 1 call cmp (internal) ## has_same_content_as (total time: 0.000s) 0.000s 1 call abc.DotChangesFile ## main (total time: 0.702s) 0.702s 2 calls outputs 0.000s 1 call cleanup ## recognizes (total time: 0.024s) 0.024s 12 calls diffoscope.comparators.binary.FilesystemFile 0.000s 10 calls abc.DotChangesFile ## specialize (total time: 0.001s) 0.001s 1 call specialize Mon May 15 14:06:48 UTC 2023 I: diffoscope 242 found no differences in the changes files, and a .buildinfo file also exists. Mon May 15 14:06:48 UTC 2023 I: golang-github-hashicorp-raft from bookworm built successfully and reproducibly on arm64. Mon May 15 14:06:50 UTC 2023 I: Submitting .buildinfo files to external archives: Mon May 15 14:06:50 UTC 2023 I: Submitting 12K b1/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc Mon May 15 14:06:51 UTC 2023 I: Submitting 12K b2/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc Mon May 15 14:06:52 UTC 2023 I: Done submitting .buildinfo files to http://buildinfo.debian.net/api/submit. Mon May 15 14:06:52 UTC 2023 I: Done submitting .buildinfo files. Mon May 15 14:06:52 UTC 2023 I: Removing signed golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc files: removed './b1/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc' removed './b2/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc'