Tue Jun 11 14:37:21 UTC 2024 I: starting to build golang-github-hashicorp-raft/unstable/armhf on jenkins on '2024-06-11 14:37' Tue Jun 11 14:37:21 UTC 2024 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/armhf_33/16086/console.log Tue Jun 11 14:37:21 UTC 2024 I: Downloading source for unstable/golang-github-hashicorp-raft=1.5.0-1 --2024-06-11 14:37:21-- http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc Connecting to 46.16.76.132:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2480 (2.4K) [text/prs.lines.tag] Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ 0K .. 100% 357M=0s 2024-06-11 14:37:21 (357 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480] Tue Jun 11 14:37:21 UTC 2024 I: golang-github-hashicorp-raft_1.5.0-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.5.0-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.6.2 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev Package-List: golang-github-hashicorp-raft-dev deb golang optional arch=all Checksums-Sha1: cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Checksums-Sha256: f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Files: 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg +DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z 14FQMVvbK31veGXJjA== =okTI -----END PGP SIGNATURE----- Tue Jun 11 14:37:21 UTC 2024 I: Checking whether the package is not for us Tue Jun 11 14:37:21 UTC 2024 I: Starting 1st build on remote node virt32z-armhf-rb.debian.net. Tue Jun 11 14:37:21 UTC 2024 I: Preparing to do remote build '1' on virt32z-armhf-rb.debian.net. Tue Jun 11 14:42:07 UTC 2024 I: Deleting $TMPDIR on virt32z-armhf-rb.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Tue Jun 11 02:37:27 -12 2024 I: pbuilder-time-stamp: 1718116647 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source gpgv: Signature made Thu Jul 27 15:15:46 2023 gpgv: using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29 gpgv: issuer "dswarbrick@debian.org" gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/21988/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='armhf' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=3 ' DISTRIBUTION='unstable' HOME='/root' HOST_ARCH='armhf' IFS=' ' INVOCATION_ID='39ce3d54e30e403a8bec9890bffaefda' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='21988' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.MQJGjjCW/pbuilderrc_1ABF --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.MQJGjjCW/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='110' SUDO_UID='103' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://10.0.0.15:3142/' I: uname -a Linux virt32z 6.1.0-21-armmp-lpae #1 SMP Debian 6.1.90-1 (2024-05-03) armv7l GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Jun 5 22:35 /bin -> usr/bin I: user script /srv/workspace/pbuilder/21988/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: armhf Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19460 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.22-go{a} golang-1.22-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 102 newly installed, 0 to remove and 0 not upgraded. Need to get 85.9 MB of archives. After unpacking 614 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian unstable/main armhf sensible-utils all 0.0.22 [22.4 kB] Get: 2 http://deb.debian.org/debian unstable/main armhf libmagic-mgc armhf 1:5.45-3 [314 kB] Get: 3 http://deb.debian.org/debian unstable/main armhf libmagic1t64 armhf 1:5.45-3 [98.1 kB] Get: 4 http://deb.debian.org/debian unstable/main armhf file armhf 1:5.45-3 [42.0 kB] Get: 5 http://deb.debian.org/debian unstable/main armhf gettext-base armhf 0.21-14+b1 [157 kB] Get: 6 http://deb.debian.org/debian unstable/main armhf libuchardet0 armhf 0.0.8-1+b1 [65.7 kB] Get: 7 http://deb.debian.org/debian unstable/main armhf groff-base armhf 1.23.0-4 [1090 kB] Get: 8 http://deb.debian.org/debian unstable/main armhf bsdextrautils armhf 2.40.1-8.1 [87.7 kB] Get: 9 http://deb.debian.org/debian unstable/main armhf libpipeline1 armhf 1.5.7-2 [33.3 kB] Get: 10 http://deb.debian.org/debian unstable/main armhf man-db armhf 2.12.1-2 [1376 kB] Get: 11 http://deb.debian.org/debian unstable/main armhf m4 armhf 1.4.19-4 [264 kB] Get: 12 http://deb.debian.org/debian unstable/main armhf autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian unstable/main armhf autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian unstable/main armhf automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian unstable/main armhf autopoint all 0.21-14 [496 kB] Get: 16 http://deb.debian.org/debian unstable/main armhf libdebhelper-perl all 13.15.3 [88.0 kB] Get: 17 http://deb.debian.org/debian unstable/main armhf libtool all 2.4.7-7 [517 kB] Get: 18 http://deb.debian.org/debian unstable/main armhf dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian unstable/main armhf libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian unstable/main armhf libfile-stripnondeterminism-perl all 1.14.0-1 [19.5 kB] Get: 21 http://deb.debian.org/debian unstable/main armhf dh-strip-nondeterminism all 1.14.0-1 [8448 B] Get: 22 http://deb.debian.org/debian unstable/main armhf libelf1t64 armhf 0.191-1+b1 [183 kB] Get: 23 http://deb.debian.org/debian unstable/main armhf dwz armhf 0.15-1+b2 [106 kB] Get: 24 http://deb.debian.org/debian unstable/main armhf libicu72 armhf 72.1-4+b1 [9070 kB] Get: 25 http://deb.debian.org/debian unstable/main armhf libxml2 armhf 2.12.7+dfsg-3 [583 kB] Get: 26 http://deb.debian.org/debian unstable/main armhf gettext armhf 0.21-14+b1 [1230 kB] Get: 27 http://deb.debian.org/debian unstable/main armhf intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 28 http://deb.debian.org/debian unstable/main armhf po-debconf all 1.0.21+nmu1 [248 kB] Get: 29 http://deb.debian.org/debian unstable/main armhf debhelper all 13.15.3 [901 kB] Get: 30 http://deb.debian.org/debian unstable/main armhf dh-golang all 1.62 [24.7 kB] Get: 31 http://deb.debian.org/debian unstable/main armhf golang-1.22-src all 1.22.4-1 [18.8 MB] Get: 32 http://deb.debian.org/debian unstable/main armhf golang-1.22-go armhf 1.22.4-1 [21.5 MB] Get: 33 http://deb.debian.org/debian unstable/main armhf golang-src all 2:1.22~3 [5020 B] Get: 34 http://deb.debian.org/debian unstable/main armhf golang-go armhf 2:1.22~3 [44.2 kB] Get: 35 http://deb.debian.org/debian unstable/main armhf golang-any armhf 2:1.22~3 [5104 B] Get: 36 http://deb.debian.org/debian unstable/main armhf golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 37 http://deb.debian.org/debian unstable/main armhf golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 38 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 39 http://deb.debian.org/debian unstable/main armhf golang-golang-x-sys-dev all 0.20.0-1 [430 kB] Get: 40 http://deb.debian.org/debian unstable/main armhf golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 41 http://deb.debian.org/debian unstable/main armhf golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 42 http://deb.debian.org/debian unstable/main armhf golang-github-fatih-color-dev all 1.16.0-1 [13.8 kB] Get: 43 http://deb.debian.org/debian unstable/main armhf golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 44 http://deb.debian.org/debian unstable/main armhf golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 45 http://deb.debian.org/debian unstable/main armhf golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 46 http://deb.debian.org/debian unstable/main armhf golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 47 http://deb.debian.org/debian unstable/main armhf golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB] Get: 48 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 49 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 50 http://deb.debian.org/debian unstable/main armhf golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 51 http://deb.debian.org/debian unstable/main armhf golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 52 http://deb.debian.org/debian unstable/main armhf golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 53 http://deb.debian.org/debian unstable/main armhf golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 54 http://deb.debian.org/debian unstable/main armhf golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 55 http://deb.debian.org/debian unstable/main armhf golang-google-protobuf-dev all 1.33.0-1 [625 kB] Get: 56 http://deb.debian.org/debian unstable/main armhf golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 57 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 58 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 60 http://deb.debian.org/debian unstable/main armhf golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 61 http://deb.debian.org/debian unstable/main armhf golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 62 http://deb.debian.org/debian unstable/main armhf golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 63 http://deb.debian.org/debian unstable/main armhf golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian unstable/main armhf golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 65 http://deb.debian.org/debian unstable/main armhf golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 66 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 67 http://deb.debian.org/debian unstable/main armhf golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 68 http://deb.debian.org/debian unstable/main armhf golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 69 http://deb.debian.org/debian unstable/main armhf golang-golang-x-term-dev all 0.20.0-1 [14.8 kB] Get: 70 http://deb.debian.org/debian unstable/main armhf golang-golang-x-text-dev all 0.15.0-1 [4006 kB] Get: 71 http://deb.debian.org/debian unstable/main armhf golang-golang-x-crypto-dev all 1:0.23.0-1 [1689 kB] Get: 72 http://deb.debian.org/debian unstable/main armhf golang-golang-x-net-dev all 1:0.25.0+dfsg-1 [895 kB] Get: 73 http://deb.debian.org/debian unstable/main armhf golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 74 http://deb.debian.org/debian unstable/main armhf golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 75 http://deb.debian.org/debian unstable/main armhf golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 76 http://deb.debian.org/debian unstable/main armhf golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 77 http://deb.debian.org/debian unstable/main armhf golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 78 http://deb.debian.org/debian unstable/main armhf golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 79 http://deb.debian.org/debian unstable/main armhf golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB] Get: 80 http://deb.debian.org/debian unstable/main armhf golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 81 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-common-dev all 0.53.0-1 [114 kB] Get: 82 http://deb.debian.org/debian unstable/main armhf golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 83 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB] Get: 84 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-client-golang-dev all 1.19.0-1 [179 kB] Get: 85 http://deb.debian.org/debian unstable/main armhf golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 86 http://deb.debian.org/debian unstable/main armhf golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 87 http://deb.debian.org/debian unstable/main armhf golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 88 http://deb.debian.org/debian unstable/main armhf golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 89 http://deb.debian.org/debian unstable/main armhf golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 90 http://deb.debian.org/debian unstable/main armhf libsasl2-modules-db armhf 2.1.28+dfsg1-6 [18.0 kB] Get: 91 http://deb.debian.org/debian unstable/main armhf libsasl2-2 armhf 2.1.28+dfsg1-6 [50.1 kB] Get: 92 http://deb.debian.org/debian unstable/main armhf libssl-dev armhf 3.2.2-1 [2351 kB] Get: 93 http://deb.debian.org/debian unstable/main armhf libsasl2-dev armhf 2.1.28+dfsg1-6 [215 kB] Get: 94 http://deb.debian.org/debian unstable/main armhf golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 95 http://deb.debian.org/debian unstable/main armhf golang-golang-x-mod-dev all 0.17.0-1 [103 kB] Get: 96 http://deb.debian.org/debian unstable/main armhf golang-github-yuin-goldmark-dev all 1.7.1-1 [113 kB] Get: 97 http://deb.debian.org/debian unstable/main armhf golang-golang-x-tools-dev all 1:0.20.0+ds-1 [1571 kB] Get: 98 http://deb.debian.org/debian unstable/main armhf golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 99 http://deb.debian.org/debian unstable/main armhf golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 100 http://deb.debian.org/debian unstable/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 101 http://deb.debian.org/debian unstable/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 102 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 85.9 MB in 3s (32.7 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19460 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.22_all.deb ... Unpacking sensible-utils (0.0.22) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3_armhf.deb ... Unpacking libmagic-mgc (1:5.45-3) ... Selecting previously unselected package libmagic1t64:armhf. Preparing to unpack .../002-libmagic1t64_1%3a5.45-3_armhf.deb ... Unpacking libmagic1t64:armhf (1:5.45-3) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.45-3_armhf.deb ... Unpacking file (1:5.45-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.21-14+b1_armhf.deb ... Unpacking gettext-base (0.21-14+b1) ... Selecting previously unselected package libuchardet0:armhf. Preparing to unpack .../005-libuchardet0_0.0.8-1+b1_armhf.deb ... Unpacking libuchardet0:armhf (0.0.8-1+b1) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-4_armhf.deb ... Unpacking groff-base (1.23.0-4) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.40.1-8.1_armhf.deb ... Unpacking bsdextrautils (2.40.1-8.1) ... Selecting previously unselected package libpipeline1:armhf. Preparing to unpack .../008-libpipeline1_1.5.7-2_armhf.deb ... Unpacking libpipeline1:armhf (1.5.7-2) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.12.1-2_armhf.deb ... Unpacking man-db (2.12.1-2) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-4_armhf.deb ... Unpacking m4 (1.4.19-4) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.21-14_all.deb ... Unpacking autopoint (0.21-14) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../015-libdebhelper-perl_13.15.3_all.deb ... Unpacking libdebhelper-perl (13.15.3) ... Selecting previously unselected package libtool. Preparing to unpack .../016-libtool_2.4.7-7_all.deb ... Unpacking libtool (2.4.7-7) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../017-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../020-dh-strip-nondeterminism_1.14.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.14.0-1) ... Selecting previously unselected package libelf1t64:armhf. Preparing to unpack .../021-libelf1t64_0.191-1+b1_armhf.deb ... Unpacking libelf1t64:armhf (0.191-1+b1) ... Selecting previously unselected package dwz. Preparing to unpack .../022-dwz_0.15-1+b2_armhf.deb ... Unpacking dwz (0.15-1+b2) ... Selecting previously unselected package libicu72:armhf. Preparing to unpack .../023-libicu72_72.1-4+b1_armhf.deb ... Unpacking libicu72:armhf (72.1-4+b1) ... Selecting previously unselected package libxml2:armhf. Preparing to unpack .../024-libxml2_2.12.7+dfsg-3_armhf.deb ... Unpacking libxml2:armhf (2.12.7+dfsg-3) ... Selecting previously unselected package gettext. Preparing to unpack .../025-gettext_0.21-14+b1_armhf.deb ... Unpacking gettext (0.21-14+b1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../028-debhelper_13.15.3_all.deb ... Unpacking debhelper (13.15.3) ... Selecting previously unselected package dh-golang. Preparing to unpack .../029-dh-golang_1.62_all.deb ... Unpacking dh-golang (1.62) ... Selecting previously unselected package golang-1.22-src. Preparing to unpack .../030-golang-1.22-src_1.22.4-1_all.deb ... Unpacking golang-1.22-src (1.22.4-1) ... Selecting previously unselected package golang-1.22-go. Preparing to unpack .../031-golang-1.22-go_1.22.4-1_armhf.deb ... Unpacking golang-1.22-go (1.22.4-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../032-golang-src_2%3a1.22~3_all.deb ... Unpacking golang-src (2:1.22~3) ... Selecting previously unselected package golang-go:armhf. Preparing to unpack .../033-golang-go_2%3a1.22~3_armhf.deb ... Unpacking golang-go:armhf (2:1.22~3) ... Selecting previously unselected package golang-any:armhf. Preparing to unpack .../034-golang-any_2%3a1.22~3_armhf.deb ... Unpacking golang-any:armhf (2:1.22~3) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../038-golang-golang-x-sys-dev_0.20.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.20.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../041-golang-github-fatih-color-dev_1.16.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.16.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../054-golang-google-protobuf-dev_1.33.0-1_all.deb ... Unpacking golang-google-protobuf-dev (1.33.0-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../065-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../066-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../067-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../068-golang-golang-x-term-dev_0.20.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.20.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../069-golang-golang-x-text-dev_0.15.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.15.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../070-golang-golang-x-crypto-dev_1%3a0.23.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.23.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../071-golang-golang-x-net-dev_1%3a0.25.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.25.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../072-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../073-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../074-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../075-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../076-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../078-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../079-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../080-golang-github-prometheus-common-dev_0.53.0-1_all.deb ... Unpacking golang-github-prometheus-common-dev (0.53.0-1) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../081-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../082-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../083-golang-github-prometheus-client-golang-dev_1.19.0-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.19.0-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../084-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../085-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../086-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../087-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../088-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:armhf. Preparing to unpack .../089-libsasl2-modules-db_2.1.28+dfsg1-6_armhf.deb ... Unpacking libsasl2-modules-db:armhf (2.1.28+dfsg1-6) ... Selecting previously unselected package libsasl2-2:armhf. Preparing to unpack .../090-libsasl2-2_2.1.28+dfsg1-6_armhf.deb ... Unpacking libsasl2-2:armhf (2.1.28+dfsg1-6) ... Selecting previously unselected package libssl-dev:armhf. Preparing to unpack .../091-libssl-dev_3.2.2-1_armhf.deb ... Unpacking libssl-dev:armhf (3.2.2-1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../092-libsasl2-dev_2.1.28+dfsg1-6_armhf.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-6) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../093-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../094-golang-golang-x-mod-dev_0.17.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.17.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../095-golang-github-yuin-goldmark-dev_1.7.1-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.1-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../096-golang-golang-x-tools-dev_1%3a0.20.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.20.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../097-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../098-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../099-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../100-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../101-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:armhf (1.5.7-2) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up libicu72:armhf (72.1-4+b1) ... Setting up bsdextrautils (2.40.1-8.1) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.45-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.15.3) ... Setting up golang-golang-x-sys-dev (0.20.0-1) ... Setting up libmagic1t64:armhf (1:5.45-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-14+b1) ... Setting up m4 (1.4.19-4) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-golang-x-term-dev (0.20.0-1) ... Setting up file (1:5.45-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up libelf1t64:armhf (0.191-1+b1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:armhf (2.1.28+dfsg1-6) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.33.0-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.9.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.1-1) ... Setting up golang-1.22-src (1.22.4-1) ... Setting up libssl-dev:armhf (3.2.2-1) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.17.0-1) ... Setting up autopoint (0.21-14) ... Setting up libsasl2-2:armhf (2.1.28+dfsg1-6) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b2) ... Setting up sensible-utils (0.0.22) ... Setting up golang-golang-x-text-dev (0.15.0-1) ... Setting up libuchardet0:armhf (0.0.8-1+b1) ... Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.23.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.15.0-1) ... Setting up golang-src (2:1.22~3) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:armhf (2.12.7+dfsg-3) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.0-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.21-14+b1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-7) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-1.22-go (1.22.4-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-6) ... Setting up golang-golang-x-net-dev (1:0.25.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.14.0-1) ... Setting up groff-base (1.23.0-4) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.16.0-1) ... Setting up golang-go:armhf (2:1.22~3) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.20.0+ds-1) ... Setting up man-db (2.12.1-2) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-any:armhf (2:1.22~3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.15.3) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.62) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.53.0-1) ... Setting up golang-github-prometheus-client-golang-dev (1.19.0-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.38-13) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture armhf debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader internal/goarch internal/cpu internal/coverage/rtcov internal/abi internal/chacha8rand internal/bytealg internal/godebugs internal/goexperiment internal/goos runtime/internal/atomic runtime/internal/math runtime/internal/sys runtime/internal/syscall internal/race sync/atomic unicode unicode/utf8 container/list runtime internal/itoa math/bits math crypto/internal/alias crypto/internal/boring/sig cmp slices encoding unicode/utf16 log/internal internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/alias internal/reflectlite crypto/subtle sync internal/testlog internal/bisect internal/singleflight internal/godebug runtime/cgo errors io sort bytes strings internal/oserror syscall bufio hash strconv crypto reflect time crypto/internal/randutil internal/syscall/unix internal/safefilepath internal/syscall/execenv path math/rand github.com/hashicorp/golang-lru/simplelru github.com/hashicorp/go-immutable-radix vendor/golang.org/x/net/dns/dnsmessage context internal/poll io/fs encoding/binary internal/fmtsort os internal/intern net/netip crypto/cipher crypto/internal/boring encoding/base64 hash/crc32 crypto/aes crypto/des fmt net crypto/internal/edwards25519/field crypto/internal/nistec/fiat math/big crypto/rand encoding/json log compress/flate compress/gzip crypto/internal/nistec crypto/internal/bigmod crypto/internal/boring/bbig crypto/sha512 encoding/asn1 crypto/ecdh crypto/elliptic vendor/golang.org/x/crypto/cryptobyte crypto/internal/edwards25519 crypto/ecdsa crypto/ed25519 crypto/hmac crypto/md5 crypto/rc4 crypto/rsa crypto/sha1 crypto/sha256 crypto/dsa encoding/hex encoding/pem crypto/x509/pkix net/url path/filepath vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/internal/poly1305 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/text/transform crypto/x509 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/text/secure/bidirule net/textproto vendor/golang.org/x/net/http2/hpack crypto/tls vendor/golang.org/x/net/idna mime mime/quotedprintable mime/multipart vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy net/http/internal net/http/internal/ascii os/signal golang.org/x/sys/unix io/ioutil regexp/syntax regexp internal/saferio encoding/gob net/http/httptrace net/http github.com/mattn/go-isatty github.com/mattn/go-colorable go/token github.com/fatih/color github.com/hashicorp/go-hclog html text/template/parse hash/crc64 flag internal/sysinfo runtime/debug text/template runtime/trace testing html/template net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench ? github.com/hashicorp/raft/bench [no test files] === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2024-06-11T14:39:53.424Z [INFO] creating new snapshot: path=/tmp/raft860568882/raft3995661227/snapshots/3-10-1718116793424.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot 2024-06-11T14:39:53.431Z [INFO] creating new snapshot: path=/tmp/raft253750245/snapshots/3-10-1718116793431.tmp 2024-06-11T14:39:53.435Z [WARN] found temporary snapshot: name=3-10-1718116793431.tmp --- PASS: TestFileSS_CreateSnapshot (0.01s) === RUN TestFileSS_CancelSnapshot 2024-06-11T14:39:53.446Z [INFO] creating new snapshot: path=/tmp/raft1705601080/snapshots/3-10-1718116793446.tmp --- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention 2024-06-11T14:39:53.453Z [INFO] creating new snapshot: path=/tmp/raft2780533364/snapshots/3-10-1718116793453.tmp 2024-06-11T14:39:53.465Z [INFO] creating new snapshot: path=/tmp/raft2780533364/snapshots/3-11-1718116793465.tmp 2024-06-11T14:39:53.477Z [INFO] creating new snapshot: path=/tmp/raft2780533364/snapshots/3-12-1718116793477.tmp 2024-06-11T14:39:53.490Z [INFO] reaping snapshot: path=/tmp/raft2780533364/snapshots/3-10-1718116793453 2024-06-11T14:39:53.490Z [INFO] creating new snapshot: path=/tmp/raft2780533364/snapshots/3-13-1718116793490.tmp 2024-06-11T14:39:53.502Z [INFO] reaping snapshot: path=/tmp/raft2780533364/snapshots/3-11-1718116793465 2024-06-11T14:39:53.503Z [INFO] creating new snapshot: path=/tmp/raft2780533364/snapshots/3-14-1718116793503.tmp 2024-06-11T14:39:53.515Z [INFO] reaping snapshot: path=/tmp/raft2780533364/snapshots/3-12-1718116793477 --- PASS: TestFileSS_Retention (0.06s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2024-06-11T14:39:53.517Z [INFO] creating new snapshot: path=/tmp/raft2708908881/snapshots/5-130350-1718116793517.tmp 2024-06-11T14:39:53.530Z [INFO] creating new snapshot: path=/tmp/raft2708908881/snapshots/36-204917-1718116793530.tmp --- PASS: TestFileSS_Ordering (0.03s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.04s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 2024-06-11T14:39:53.697Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 2024-06-11T14:39:53.750Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 2024-06-11T14:39:53.803Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2024-06-11T14:39:53.817Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:53.823Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:53.833Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:53.854Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:53.894Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:53.975Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:54.136Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:54.457Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:55.098Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:56.098Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-11T14:39:57.098Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2024-06-11T14:39:57.820Z [INFO] server-c9298034-8e5e-7c0d-78db-f316b5c89c2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9298034-8e5e-7c0d-78db-f316b5c89c2a Address:c9298034-8e5e-7c0d-78db-f316b5c89c2a}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2024-06-11T14:39:57.821Z [INFO] server-a230985c-4b72-3350-ea41-84896bc498e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a230985c-4b72-3350-ea41-84896bc498e6 Address:a230985c-4b72-3350-ea41-84896bc498e6}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2024-06-11T14:39:57.822Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: initial configuration: index=0 servers=[] 2024-06-11T14:39:57.823Z [INFO] server-142eec7f-123d-2176-2921-cfed9c4f5d7c: initial configuration: index=0 servers=[] 2024-06-11T14:39:57.823Z [INFO] server-0812b930-113b-029e-7c5a-63c280746706: initial configuration: index=0 servers=[] 2024-06-11T14:39:57.823Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: entering follower state: follower="Node at 33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4 [Follower]" leader-address= leader-id= 2024-06-11T14:39:57.823Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:39:57.823Z [INFO] server-142eec7f-123d-2176-2921-cfed9c4f5d7c: entering follower state: follower="Node at 142eec7f-123d-2176-2921-cfed9c4f5d7c [Follower]" leader-address= leader-id= 2024-06-11T14:39:57.823Z [INFO] server-0812b930-113b-029e-7c5a-63c280746706: entering follower state: follower="Node at 0812b930-113b-029e-7c5a-63c280746706 [Follower]" leader-address= leader-id= 2024-06-11T14:39:57.883Z [WARN] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:57.883Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: entering candidate state: node="Node at 33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4 [Candidate]" term=2 2024-06-11T14:39:57.883Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: election won: term=2 tally=2 2024-06-11T14:39:57.883Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: entering leader state: leader="Node at 33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4 [Leader]" 2024-06-11T14:39:57.883Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: added peer, starting replication: peer=server-142eec7f-123d-2176-2921-cfed9c4f5d7c 2024-06-11T14:39:57.883Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: added peer, starting replication: peer=server-0812b930-113b-029e-7c5a-63c280746706 2024-06-11T14:39:57.883Z [WARN] server-0812b930-113b-029e-7c5a-63c280746706: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2024-06-11T14:39:57.883Z [WARN] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: appendEntries rejected, sending older logs: peer="{Voter server-0812b930-113b-029e-7c5a-63c280746706 0812b930-113b-029e-7c5a-63c280746706}" next=1 2024-06-11T14:39:57.883Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: pipelining replication: peer="{Voter server-0812b930-113b-029e-7c5a-63c280746706 0812b930-113b-029e-7c5a-63c280746706}" 2024-06-11T14:39:57.883Z [WARN] server-142eec7f-123d-2176-2921-cfed9c4f5d7c: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2024-06-11T14:39:57.883Z [WARN] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: appendEntries rejected, sending older logs: peer="{Voter server-142eec7f-123d-2176-2921-cfed9c4f5d7c 142eec7f-123d-2176-2921-cfed9c4f5d7c}" next=1 2024-06-11T14:39:57.884Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: pipelining replication: peer="{Voter server-142eec7f-123d-2176-2921-cfed9c4f5d7c 142eec7f-123d-2176-2921-cfed9c4f5d7c}" 2024-06-11T14:39:57.989Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:39:57.884123139 +0000 UTC m=+4.467202091 (2 nodes), 60.663971ms from start of poll, 61.24651ms from cluster start. Timeout at 2024-06-11 14:39:57.989425139 +0000 UTC m=+4.572504051, 105.30196ms after stability 2024-06-11T14:39:57.989Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:39:58.094Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:39:57.989589093 +0000 UTC m=+4.572668085 (1 nodes), 15.919µs from start of poll, 166.712504ms from cluster start. Timeout at 2024-06-11 14:39:58.094908213 +0000 UTC m=+4.677987165, 105.31908ms after stability 2024-06-11T14:39:58.106Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: aborting pipeline replication: peer="{Voter server-142eec7f-123d-2176-2921-cfed9c4f5d7c 142eec7f-123d-2176-2921-cfed9c4f5d7c}" 2024-06-11T14:39:58.106Z [INFO] server-33e4e1a5-d5c8-c0b7-de37-94ecc5fb14b4: aborting pipeline replication: peer="{Voter server-0812b930-113b-029e-7c5a-63c280746706 0812b930-113b-029e-7c5a-63c280746706}" --- PASS: TestRaft_LiveBootstrap (0.29s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2024-06-11T14:39:58.108Z [INFO] server-7a7babfc-36d0-2d4e-275c-e6517404a819: initial configuration: index=0 servers=[] 2024-06-11T14:39:58.108Z [INFO] server-6ffabf80-af98-00fa-9c9b-9a88fa8a9c46: initial configuration: index=0 servers=[] 2024-06-11T14:39:58.108Z [INFO] server-7a7babfc-36d0-2d4e-275c-e6517404a819: entering follower state: follower="Node at 7a7babfc-36d0-2d4e-275c-e6517404a819 [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2024-06-11T14:39:58.109Z [INFO] server-523a6bbe-ffda-75c4-9c04-fea1995e43be: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2024-06-11T14:39:58.111Z [INFO] server-209ed2ae-2800-edf8-a063-29aaf70ce8e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 Address:209ed2ae-2800-edf8-a063-29aaf70ce8e9} {Suffrage:Voter ID:server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc Address:eceec754-fb2a-d1f2-231f-efab3ea3e8dc} {Suffrage:Voter ID:server-a8a86e35-5e00-ef88-9461-84ede2582a5c Address:a8a86e35-5e00-ef88-9461-84ede2582a5c}]" 2024-06-11T14:39:58.111Z [INFO] server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 Address:209ed2ae-2800-edf8-a063-29aaf70ce8e9} {Suffrage:Voter ID:server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc Address:eceec754-fb2a-d1f2-231f-efab3ea3e8dc} {Suffrage:Voter ID:server-a8a86e35-5e00-ef88-9461-84ede2582a5c Address:a8a86e35-5e00-ef88-9461-84ede2582a5c}]" 2024-06-11T14:39:58.111Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 Address:209ed2ae-2800-edf8-a063-29aaf70ce8e9} {Suffrage:Voter ID:server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc Address:eceec754-fb2a-d1f2-231f-efab3ea3e8dc} {Suffrage:Voter ID:server-a8a86e35-5e00-ef88-9461-84ede2582a5c Address:a8a86e35-5e00-ef88-9461-84ede2582a5c}]" 2024-06-11T14:39:58.111Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:39:58.111Z [INFO] server-209ed2ae-2800-edf8-a063-29aaf70ce8e9: entering follower state: follower="Node at 209ed2ae-2800-edf8-a063-29aaf70ce8e9 [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.111Z [INFO] server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc: entering follower state: follower="Node at eceec754-fb2a-d1f2-231f-efab3ea3e8dc [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.111Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: entering follower state: follower="Node at a8a86e35-5e00-ef88-9461-84ede2582a5c [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.172Z [WARN] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:58.172Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: entering candidate state: node="Node at a8a86e35-5e00-ef88-9461-84ede2582a5c [Candidate]" term=2 2024-06-11T14:39:58.172Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: election won: term=2 tally=2 2024-06-11T14:39:58.172Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: entering leader state: leader="Node at a8a86e35-5e00-ef88-9461-84ede2582a5c [Leader]" 2024-06-11T14:39:58.172Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: added peer, starting replication: peer=server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 2024-06-11T14:39:58.172Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: added peer, starting replication: peer=server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc 2024-06-11T14:39:58.172Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: pipelining replication: peer="{Voter server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc eceec754-fb2a-d1f2-231f-efab3ea3e8dc}" 2024-06-11T14:39:58.172Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: pipelining replication: peer="{Voter server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 209ed2ae-2800-edf8-a063-29aaf70ce8e9}" 2024-06-11T14:39:58.278Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:39:58.172778483 +0000 UTC m=+4.755857435 (1 nodes), 61.072676ms from start of poll, 61.718693ms from cluster start. Timeout at 2024-06-11 14:39:58.27847315 +0000 UTC m=+4.861552062, 105.694627ms after stability 2024-06-11T14:39:58.278Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: aborting pipeline replication: peer="{Voter server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc eceec754-fb2a-d1f2-231f-efab3ea3e8dc}" 2024-06-11T14:39:58.278Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: aborting pipeline replication: peer="{Voter server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 209ed2ae-2800-edf8-a063-29aaf70ce8e9}" 2024-06-11T14:39:58.278Z [INFO] creating new snapshot: path=/tmp/raft3528178340/snapshots/2-2-1718116798278.tmp 2024-06-11T14:39:58.280Z [INFO] server-209ed2ae-2800-edf8-a063-29aaf70ce8e9: starting restore from snapshot: id=2-2-1718116798278 last-index=2 last-term=2 size-in-bytes=1 2024-06-11T14:39:58.280Z [INFO] server-209ed2ae-2800-edf8-a063-29aaf70ce8e9: snapshot restore progress: id=2-2-1718116798278 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2024-06-11T14:39:58.280Z [INFO] server-209ed2ae-2800-edf8-a063-29aaf70ce8e9: restored from snapshot: id=2-2-1718116798278 last-index=2 last-term=2 size-in-bytes=1 2024-06-11T14:39:58.280Z [INFO] server-209ed2ae-2800-edf8-a063-29aaf70ce8e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 Address:209ed2ae-2800-edf8-a063-29aaf70ce8e9} {Suffrage:Voter ID:server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc Address:eceec754-fb2a-d1f2-231f-efab3ea3e8dc} {Suffrage:Voter ID:server-a8a86e35-5e00-ef88-9461-84ede2582a5c Address:a8a86e35-5e00-ef88-9461-84ede2582a5c}]" 2024-06-11T14:39:58.280Z [INFO] creating new snapshot: path=/tmp/raft1082860114/snapshots/2-2-1718116798280.tmp 2024-06-11T14:39:58.281Z [INFO] server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc: starting restore from snapshot: id=2-2-1718116798280 last-index=2 last-term=2 size-in-bytes=1 2024-06-11T14:39:58.281Z [INFO] server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc: snapshot restore progress: id=2-2-1718116798280 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2024-06-11T14:39:58.281Z [INFO] server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc: restored from snapshot: id=2-2-1718116798280 last-index=2 last-term=2 size-in-bytes=1 2024-06-11T14:39:58.281Z [INFO] server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 Address:209ed2ae-2800-edf8-a063-29aaf70ce8e9} {Suffrage:Voter ID:server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc Address:eceec754-fb2a-d1f2-231f-efab3ea3e8dc} {Suffrage:Voter ID:server-a8a86e35-5e00-ef88-9461-84ede2582a5c Address:a8a86e35-5e00-ef88-9461-84ede2582a5c}]" 2024-06-11T14:39:58.281Z [INFO] creating new snapshot: path=/tmp/raft3225609305/snapshots/2-2-1718116798281.tmp 2024-06-11T14:39:58.282Z [INFO] server-209ed2ae-2800-edf8-a063-29aaf70ce8e9: entering follower state: follower="Node at 209ed2ae-2800-edf8-a063-29aaf70ce8e9 [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.282Z [INFO] server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc: entering follower state: follower="Node at eceec754-fb2a-d1f2-231f-efab3ea3e8dc [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.283Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: starting restore from snapshot: id=2-2-1718116798281 last-index=2 last-term=2 size-in-bytes=1 2024-06-11T14:39:58.283Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: snapshot restore progress: id=2-2-1718116798281 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2024-06-11T14:39:58.283Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: restored from snapshot: id=2-2-1718116798281 last-index=2 last-term=2 size-in-bytes=1 2024-06-11T14:39:58.283Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 Address:209ed2ae-2800-edf8-a063-29aaf70ce8e9} {Suffrage:Voter ID:server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc Address:eceec754-fb2a-d1f2-231f-efab3ea3e8dc} {Suffrage:Voter ID:server-a8a86e35-5e00-ef88-9461-84ede2582a5c Address:a8a86e35-5e00-ef88-9461-84ede2582a5c}]" 2024-06-11T14:39:58.283Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: entering follower state: follower="Node at a8a86e35-5e00-ef88-9461-84ede2582a5c [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.335Z [WARN] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:58.335Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: entering candidate state: node="Node at a8a86e35-5e00-ef88-9461-84ede2582a5c [Candidate]" term=3 2024-06-11T14:39:58.335Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: election won: term=3 tally=2 2024-06-11T14:39:58.335Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: entering leader state: leader="Node at a8a86e35-5e00-ef88-9461-84ede2582a5c [Leader]" 2024-06-11T14:39:58.336Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: added peer, starting replication: peer=server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 2024-06-11T14:39:58.336Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: added peer, starting replication: peer=server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc 2024-06-11T14:39:58.336Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: pipelining replication: peer="{Voter server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc eceec754-fb2a-d1f2-231f-efab3ea3e8dc}" 2024-06-11T14:39:58.336Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: pipelining replication: peer="{Voter server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 209ed2ae-2800-edf8-a063-29aaf70ce8e9}" 2024-06-11T14:39:58.599Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:39:58.705Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:39:58.599768716 +0000 UTC m=+5.182847628 (1 nodes), 40.318µs from start of poll, 488.708886ms from cluster start. Timeout at 2024-06-11 14:39:58.705073997 +0000 UTC m=+5.288152909, 105.305281ms after stability 2024-06-11T14:39:58.705Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: aborting pipeline replication: peer="{Voter server-eceec754-fb2a-d1f2-231f-efab3ea3e8dc eceec754-fb2a-d1f2-231f-efab3ea3e8dc}" 2024-06-11T14:39:58.705Z [INFO] server-a8a86e35-5e00-ef88-9461-84ede2582a5c: aborting pipeline replication: peer="{Voter server-209ed2ae-2800-edf8-a063-29aaf70ce8e9 209ed2ae-2800-edf8-a063-29aaf70ce8e9}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2024-06-11T14:39:58.711Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 Address:29f4d260-c27c-57b7-4783-30aa8b3e17d9} {Suffrage:Voter ID:server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab Address:54d3a3dc-4ea3-f491-0b93-6597fb1acdab} {Suffrage:Voter ID:server-61fee36d-f632-87a7-3c51-c1464762d1fc Address:61fee36d-f632-87a7-3c51-c1464762d1fc}]" 2024-06-11T14:39:58.711Z [INFO] server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 Address:29f4d260-c27c-57b7-4783-30aa8b3e17d9} {Suffrage:Voter ID:server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab Address:54d3a3dc-4ea3-f491-0b93-6597fb1acdab} {Suffrage:Voter ID:server-61fee36d-f632-87a7-3c51-c1464762d1fc Address:61fee36d-f632-87a7-3c51-c1464762d1fc}]" 2024-06-11T14:39:58.711Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 Address:29f4d260-c27c-57b7-4783-30aa8b3e17d9} {Suffrage:Voter ID:server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab Address:54d3a3dc-4ea3-f491-0b93-6597fb1acdab} {Suffrage:Voter ID:server-61fee36d-f632-87a7-3c51-c1464762d1fc Address:61fee36d-f632-87a7-3c51-c1464762d1fc}]" 2024-06-11T14:39:58.711Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:39:58.711Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: entering follower state: follower="Node at 29f4d260-c27c-57b7-4783-30aa8b3e17d9 [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.711Z [INFO] server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab: entering follower state: follower="Node at 54d3a3dc-4ea3-f491-0b93-6597fb1acdab [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.711Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: entering follower state: follower="Node at 61fee36d-f632-87a7-3c51-c1464762d1fc [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.791Z [WARN] server-61fee36d-f632-87a7-3c51-c1464762d1fc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:58.791Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: entering candidate state: node="Node at 61fee36d-f632-87a7-3c51-c1464762d1fc [Candidate]" term=2 2024-06-11T14:39:58.792Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: election won: term=2 tally=2 2024-06-11T14:39:58.792Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: entering leader state: leader="Node at 61fee36d-f632-87a7-3c51-c1464762d1fc [Leader]" 2024-06-11T14:39:58.792Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: added peer, starting replication: peer=server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 2024-06-11T14:39:58.792Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: added peer, starting replication: peer=server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab 2024-06-11T14:39:58.792Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: pipelining replication: peer="{Voter server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab 54d3a3dc-4ea3-f491-0b93-6597fb1acdab}" 2024-06-11T14:39:58.792Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: pipelining replication: peer="{Voter server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 29f4d260-c27c-57b7-4783-30aa8b3e17d9}" 2024-06-11T14:39:58.898Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:39:58.792673676 +0000 UTC m=+5.375752588 (1 nodes), 80.93032ms from start of poll, 81.588056ms from cluster start. Timeout at 2024-06-11 14:39:58.897976276 +0000 UTC m=+5.481055188, 105.3026ms after stability 2024-06-11T14:39:58.899Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: aborting pipeline replication: peer="{Voter server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab 54d3a3dc-4ea3-f491-0b93-6597fb1acdab}" 2024-06-11T14:39:58.899Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: aborting pipeline replication: peer="{Voter server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 29f4d260-c27c-57b7-4783-30aa8b3e17d9}" 2024-06-11T14:39:58.899Z [INFO] creating new snapshot: path=/tmp/raft1448038950/snapshots/2-6-1718116798899.tmp 2024-06-11T14:39:58.900Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: starting restore from snapshot: id=2-6-1718116798899 last-index=6 last-term=2 size-in-bytes=25 2024-06-11T14:39:58.900Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: snapshot restore progress: id=2-6-1718116798899 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2024-06-11T14:39:58.900Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: restored from snapshot: id=2-6-1718116798899 last-index=6 last-term=2 size-in-bytes=25 2024-06-11T14:39:58.901Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 Address:29f4d260-c27c-57b7-4783-30aa8b3e17d9} {Suffrage:Voter ID:server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab Address:54d3a3dc-4ea3-f491-0b93-6597fb1acdab} {Suffrage:Voter ID:server-61fee36d-f632-87a7-3c51-c1464762d1fc Address:61fee36d-f632-87a7-3c51-c1464762d1fc}]" 2024-06-11T14:39:58.901Z [INFO] creating new snapshot: path=/tmp/raft1975820548/snapshots/2-6-1718116798901.tmp 2024-06-11T14:39:58.902Z [INFO] server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab: starting restore from snapshot: id=2-6-1718116798901 last-index=6 last-term=2 size-in-bytes=25 2024-06-11T14:39:58.904Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: entering follower state: follower="Node at 29f4d260-c27c-57b7-4783-30aa8b3e17d9 [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.905Z [INFO] server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab: snapshot restore progress: id=2-6-1718116798901 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2024-06-11T14:39:58.905Z [INFO] server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab: restored from snapshot: id=2-6-1718116798901 last-index=6 last-term=2 size-in-bytes=25 2024-06-11T14:39:58.905Z [INFO] server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 Address:29f4d260-c27c-57b7-4783-30aa8b3e17d9} {Suffrage:Voter ID:server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab Address:54d3a3dc-4ea3-f491-0b93-6597fb1acdab} {Suffrage:Voter ID:server-61fee36d-f632-87a7-3c51-c1464762d1fc Address:61fee36d-f632-87a7-3c51-c1464762d1fc}]" 2024-06-11T14:39:58.905Z [INFO] creating new snapshot: path=/tmp/raft3904940657/snapshots/2-6-1718116798905.tmp 2024-06-11T14:39:58.906Z [INFO] server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab: entering follower state: follower="Node at 54d3a3dc-4ea3-f491-0b93-6597fb1acdab [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.907Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: starting restore from snapshot: id=2-6-1718116798905 last-index=6 last-term=2 size-in-bytes=25 2024-06-11T14:39:58.907Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: snapshot restore progress: id=2-6-1718116798905 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2024-06-11T14:39:58.907Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: restored from snapshot: id=2-6-1718116798905 last-index=6 last-term=2 size-in-bytes=25 2024-06-11T14:39:58.907Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f4d260-c27c-57b7-4783-30aa8b3e17d9 Address:29f4d260-c27c-57b7-4783-30aa8b3e17d9} {Suffrage:Voter ID:server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab Address:54d3a3dc-4ea3-f491-0b93-6597fb1acdab} {Suffrage:Voter ID:server-61fee36d-f632-87a7-3c51-c1464762d1fc Address:61fee36d-f632-87a7-3c51-c1464762d1fc}]" 2024-06-11T14:39:58.907Z [INFO] server-61fee36d-f632-87a7-3c51-c1464762d1fc: entering follower state: follower="Node at 61fee36d-f632-87a7-3c51-c1464762d1fc [Follower]" leader-address= leader-id= 2024-06-11T14:39:58.967Z [WARN] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:58.967Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: entering candidate state: node="Node at 29f4d260-c27c-57b7-4783-30aa8b3e17d9 [Candidate]" term=3 2024-06-11T14:39:58.968Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: election won: term=3 tally=2 2024-06-11T14:39:58.968Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: entering leader state: leader="Node at 29f4d260-c27c-57b7-4783-30aa8b3e17d9 [Leader]" 2024-06-11T14:39:58.968Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: added peer, starting replication: peer=server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab 2024-06-11T14:39:58.968Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: added peer, starting replication: peer=server-61fee36d-f632-87a7-3c51-c1464762d1fc 2024-06-11T14:39:58.968Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: pipelining replication: peer="{Voter server-61fee36d-f632-87a7-3c51-c1464762d1fc 61fee36d-f632-87a7-3c51-c1464762d1fc}" 2024-06-11T14:39:58.968Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: pipelining replication: peer="{Voter server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab 54d3a3dc-4ea3-f491-0b93-6597fb1acdab}" 2024-06-11T14:39:59.223Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:39:59.328Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:39:59.22322142 +0000 UTC m=+5.806300372 (1 nodes), 116.475µs from start of poll, 512.13584ms from cluster start. Timeout at 2024-06-11 14:39:59.32827163 +0000 UTC m=+5.911350582, 105.05021ms after stability 2024-06-11T14:39:59.328Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: aborting pipeline replication: peer="{Voter server-61fee36d-f632-87a7-3c51-c1464762d1fc 61fee36d-f632-87a7-3c51-c1464762d1fc}" 2024-06-11T14:39:59.328Z [INFO] server-29f4d260-c27c-57b7-4783-30aa8b3e17d9: aborting pipeline replication: peer="{Voter server-54d3a3dc-4ea3-f491-0b93-6597fb1acdab 54d3a3dc-4ea3-f491-0b93-6597fb1acdab}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2024-06-11T14:39:59.331Z [INFO] server-50b7de4c-3878-4061-13f3-ada5edf727c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b7de4c-3878-4061-13f3-ada5edf727c4 Address:50b7de4c-3878-4061-13f3-ada5edf727c4} {Suffrage:Voter ID:server-5eff50fc-4359-2351-7e6f-20830d266ea5 Address:5eff50fc-4359-2351-7e6f-20830d266ea5} {Suffrage:Voter ID:server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 Address:2f92d7be-58f4-afae-39f6-f09c5b3e9319}]" 2024-06-11T14:39:59.331Z [INFO] server-50b7de4c-3878-4061-13f3-ada5edf727c4: entering follower state: follower="Node at 50b7de4c-3878-4061-13f3-ada5edf727c4 [Follower]" leader-address= leader-id= 2024-06-11T14:39:59.331Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b7de4c-3878-4061-13f3-ada5edf727c4 Address:50b7de4c-3878-4061-13f3-ada5edf727c4} {Suffrage:Voter ID:server-5eff50fc-4359-2351-7e6f-20830d266ea5 Address:5eff50fc-4359-2351-7e6f-20830d266ea5} {Suffrage:Voter ID:server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 Address:2f92d7be-58f4-afae-39f6-f09c5b3e9319}]" 2024-06-11T14:39:59.331Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: entering follower state: follower="Node at 5eff50fc-4359-2351-7e6f-20830d266ea5 [Follower]" leader-address= leader-id= 2024-06-11T14:39:59.331Z [INFO] server-2f92d7be-58f4-afae-39f6-f09c5b3e9319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b7de4c-3878-4061-13f3-ada5edf727c4 Address:50b7de4c-3878-4061-13f3-ada5edf727c4} {Suffrage:Voter ID:server-5eff50fc-4359-2351-7e6f-20830d266ea5 Address:5eff50fc-4359-2351-7e6f-20830d266ea5} {Suffrage:Voter ID:server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 Address:2f92d7be-58f4-afae-39f6-f09c5b3e9319}]" 2024-06-11T14:39:59.331Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:39:59.332Z [INFO] server-2f92d7be-58f4-afae-39f6-f09c5b3e9319: entering follower state: follower="Node at 2f92d7be-58f4-afae-39f6-f09c5b3e9319 [Follower]" leader-address= leader-id= 2024-06-11T14:39:59.393Z [WARN] server-5eff50fc-4359-2351-7e6f-20830d266ea5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:59.393Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: entering candidate state: node="Node at 5eff50fc-4359-2351-7e6f-20830d266ea5 [Candidate]" term=2 2024-06-11T14:39:59.394Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: election won: term=2 tally=2 2024-06-11T14:39:59.394Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: entering leader state: leader="Node at 5eff50fc-4359-2351-7e6f-20830d266ea5 [Leader]" 2024-06-11T14:39:59.394Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: added peer, starting replication: peer=server-50b7de4c-3878-4061-13f3-ada5edf727c4 2024-06-11T14:39:59.394Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: added peer, starting replication: peer=server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 2024-06-11T14:39:59.394Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: pipelining replication: peer="{Voter server-50b7de4c-3878-4061-13f3-ada5edf727c4 50b7de4c-3878-4061-13f3-ada5edf727c4}" 2024-06-11T14:39:59.394Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: pipelining replication: peer="{Voter server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 2f92d7be-58f4-afae-39f6-f09c5b3e9319}" 2024-06-11T14:39:59.499Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:39:59.394179732 +0000 UTC m=+5.977258684 (1 nodes), 62.164197ms from start of poll, 62.894251ms from cluster start. Timeout at 2024-06-11 14:39:59.499627647 +0000 UTC m=+6.082706559, 105.447875ms after stability 2024-06-11T14:39:59.505Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: aborting pipeline replication: peer="{Voter server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 2f92d7be-58f4-afae-39f6-f09c5b3e9319}" 2024-06-11T14:39:59.505Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: aborting pipeline replication: peer="{Voter server-50b7de4c-3878-4061-13f3-ada5edf727c4 50b7de4c-3878-4061-13f3-ada5edf727c4}" 2024-06-11T14:39:59.506Z [INFO] creating new snapshot: path=/tmp/raft4026219514/snapshots/2-27-1718116799506.tmp 2024-06-11T14:39:59.507Z [INFO] server-50b7de4c-3878-4061-13f3-ada5edf727c4: starting restore from snapshot: id=2-27-1718116799506 last-index=27 last-term=2 size-in-bytes=168 2024-06-11T14:39:59.507Z [INFO] server-50b7de4c-3878-4061-13f3-ada5edf727c4: snapshot restore progress: id=2-27-1718116799506 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2024-06-11T14:39:59.507Z [INFO] server-50b7de4c-3878-4061-13f3-ada5edf727c4: restored from snapshot: id=2-27-1718116799506 last-index=27 last-term=2 size-in-bytes=168 2024-06-11T14:39:59.507Z [INFO] server-50b7de4c-3878-4061-13f3-ada5edf727c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b7de4c-3878-4061-13f3-ada5edf727c4 Address:50b7de4c-3878-4061-13f3-ada5edf727c4} {Suffrage:Voter ID:server-5eff50fc-4359-2351-7e6f-20830d266ea5 Address:5eff50fc-4359-2351-7e6f-20830d266ea5} {Suffrage:Voter ID:server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 Address:2f92d7be-58f4-afae-39f6-f09c5b3e9319}]" 2024-06-11T14:39:59.508Z [INFO] server-50b7de4c-3878-4061-13f3-ada5edf727c4: entering follower state: follower="Node at 50b7de4c-3878-4061-13f3-ada5edf727c4 [Follower]" leader-address= leader-id= 2024-06-11T14:39:59.508Z [INFO] creating new snapshot: path=/tmp/raft3076031012/snapshots/2-27-1718116799508.tmp 2024-06-11T14:39:59.510Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: starting restore from snapshot: id=2-27-1718116799508 last-index=27 last-term=2 size-in-bytes=168 2024-06-11T14:39:59.510Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: snapshot restore progress: id=2-27-1718116799508 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2024-06-11T14:39:59.511Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: restored from snapshot: id=2-27-1718116799508 last-index=27 last-term=2 size-in-bytes=168 2024-06-11T14:39:59.511Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b7de4c-3878-4061-13f3-ada5edf727c4 Address:50b7de4c-3878-4061-13f3-ada5edf727c4} {Suffrage:Voter ID:server-5eff50fc-4359-2351-7e6f-20830d266ea5 Address:5eff50fc-4359-2351-7e6f-20830d266ea5} {Suffrage:Voter ID:server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 Address:2f92d7be-58f4-afae-39f6-f09c5b3e9319}]" 2024-06-11T14:39:59.511Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: entering follower state: follower="Node at 5eff50fc-4359-2351-7e6f-20830d266ea5 [Follower]" leader-address= leader-id= 2024-06-11T14:39:59.511Z [INFO] creating new snapshot: path=/tmp/raft2305386892/snapshots/2-27-1718116799511.tmp 2024-06-11T14:39:59.513Z [INFO] server-2f92d7be-58f4-afae-39f6-f09c5b3e9319: starting restore from snapshot: id=2-27-1718116799511 last-index=27 last-term=2 size-in-bytes=168 2024-06-11T14:39:59.513Z [INFO] server-2f92d7be-58f4-afae-39f6-f09c5b3e9319: snapshot restore progress: id=2-27-1718116799511 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2024-06-11T14:39:59.513Z [INFO] server-2f92d7be-58f4-afae-39f6-f09c5b3e9319: restored from snapshot: id=2-27-1718116799511 last-index=27 last-term=2 size-in-bytes=168 2024-06-11T14:39:59.513Z [INFO] server-2f92d7be-58f4-afae-39f6-f09c5b3e9319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b7de4c-3878-4061-13f3-ada5edf727c4 Address:50b7de4c-3878-4061-13f3-ada5edf727c4} {Suffrage:Voter ID:server-5eff50fc-4359-2351-7e6f-20830d266ea5 Address:5eff50fc-4359-2351-7e6f-20830d266ea5} {Suffrage:Voter ID:server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 Address:2f92d7be-58f4-afae-39f6-f09c5b3e9319}]" 2024-06-11T14:39:59.513Z [INFO] server-2f92d7be-58f4-afae-39f6-f09c5b3e9319: entering follower state: follower="Node at 2f92d7be-58f4-afae-39f6-f09c5b3e9319 [Follower]" leader-address= leader-id= 2024-06-11T14:39:59.570Z [WARN] server-5eff50fc-4359-2351-7e6f-20830d266ea5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:59.571Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: entering candidate state: node="Node at 5eff50fc-4359-2351-7e6f-20830d266ea5 [Candidate]" term=3 2024-06-11T14:39:59.571Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: election won: term=3 tally=2 2024-06-11T14:39:59.571Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: entering leader state: leader="Node at 5eff50fc-4359-2351-7e6f-20830d266ea5 [Leader]" 2024-06-11T14:39:59.571Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: added peer, starting replication: peer=server-50b7de4c-3878-4061-13f3-ada5edf727c4 2024-06-11T14:39:59.571Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: added peer, starting replication: peer=server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 2024-06-11T14:39:59.571Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: pipelining replication: peer="{Voter server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 2f92d7be-58f4-afae-39f6-f09c5b3e9319}" 2024-06-11T14:39:59.571Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: pipelining replication: peer="{Voter server-50b7de4c-3878-4061-13f3-ada5edf727c4 50b7de4c-3878-4061-13f3-ada5edf727c4}" 2024-06-11T14:39:59.828Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:39:59.934Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:39:59.828805369 +0000 UTC m=+6.411884321 (1 nodes), 64.837µs from start of poll, 497.519888ms from cluster start. Timeout at 2024-06-11 14:39:59.934400999 +0000 UTC m=+6.517479951, 105.59563ms after stability 2024-06-11T14:39:59.934Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: aborting pipeline replication: peer="{Voter server-2f92d7be-58f4-afae-39f6-f09c5b3e9319 2f92d7be-58f4-afae-39f6-f09c5b3e9319}" 2024-06-11T14:39:59.934Z [INFO] server-5eff50fc-4359-2351-7e6f-20830d266ea5: aborting pipeline replication: peer="{Voter server-50b7de4c-3878-4061-13f3-ada5edf727c4 50b7de4c-3878-4061-13f3-ada5edf727c4}" --- PASS: TestRaft_RecoverCluster (1.83s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.62s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.61s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2024-06-11T14:39:59.936Z [INFO] server-ef1306b9-b716-f7b0-067c-c441767a6439: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef1306b9-b716-f7b0-067c-c441767a6439 Address:ef1306b9-b716-f7b0-067c-c441767a6439}]" 2024-06-11T14:39:59.936Z [INFO] server-ef1306b9-b716-f7b0-067c-c441767a6439: entering follower state: follower="Node at ef1306b9-b716-f7b0-067c-c441767a6439 [Follower]" leader-address= leader-id= 2024-06-11T14:39:59.997Z [WARN] server-ef1306b9-b716-f7b0-067c-c441767a6439: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:39:59.997Z [INFO] server-ef1306b9-b716-f7b0-067c-c441767a6439: entering candidate state: node="Node at ef1306b9-b716-f7b0-067c-c441767a6439 [Candidate]" term=2 2024-06-11T14:39:59.997Z [INFO] server-ef1306b9-b716-f7b0-067c-c441767a6439: election won: term=2 tally=1 2024-06-11T14:39:59.997Z [INFO] server-ef1306b9-b716-f7b0-067c-c441767a6439: entering leader state: leader="Node at ef1306b9-b716-f7b0-067c-c441767a6439 [Leader]" --- PASS: TestRaft_SingleNode (0.06s) === RUN TestRaft_TripleNode 2024-06-11T14:39:59.999Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f209674-06b7-fe72-5cf1-96651365946c Address:9f209674-06b7-fe72-5cf1-96651365946c} {Suffrage:Voter ID:server-2d300aa5-282f-9808-d086-11c07f5e9786 Address:2d300aa5-282f-9808-d086-11c07f5e9786} {Suffrage:Voter ID:server-4ee0f7d4-5b8d-114d-2b43-a522565c0943 Address:4ee0f7d4-5b8d-114d-2b43-a522565c0943}]" 2024-06-11T14:39:59.999Z [INFO] server-2d300aa5-282f-9808-d086-11c07f5e9786: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f209674-06b7-fe72-5cf1-96651365946c Address:9f209674-06b7-fe72-5cf1-96651365946c} {Suffrage:Voter ID:server-2d300aa5-282f-9808-d086-11c07f5e9786 Address:2d300aa5-282f-9808-d086-11c07f5e9786} {Suffrage:Voter ID:server-4ee0f7d4-5b8d-114d-2b43-a522565c0943 Address:4ee0f7d4-5b8d-114d-2b43-a522565c0943}]" 2024-06-11T14:40:00.000Z [INFO] server-4ee0f7d4-5b8d-114d-2b43-a522565c0943: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f209674-06b7-fe72-5cf1-96651365946c Address:9f209674-06b7-fe72-5cf1-96651365946c} {Suffrage:Voter ID:server-2d300aa5-282f-9808-d086-11c07f5e9786 Address:2d300aa5-282f-9808-d086-11c07f5e9786} {Suffrage:Voter ID:server-4ee0f7d4-5b8d-114d-2b43-a522565c0943 Address:4ee0f7d4-5b8d-114d-2b43-a522565c0943}]" 2024-06-11T14:40:00.000Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:00.000Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: entering follower state: follower="Node at 9f209674-06b7-fe72-5cf1-96651365946c [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.000Z [INFO] server-2d300aa5-282f-9808-d086-11c07f5e9786: entering follower state: follower="Node at 2d300aa5-282f-9808-d086-11c07f5e9786 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.000Z [INFO] server-4ee0f7d4-5b8d-114d-2b43-a522565c0943: entering follower state: follower="Node at 4ee0f7d4-5b8d-114d-2b43-a522565c0943 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.079Z [WARN] server-9f209674-06b7-fe72-5cf1-96651365946c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:00.079Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: entering candidate state: node="Node at 9f209674-06b7-fe72-5cf1-96651365946c [Candidate]" term=2 2024-06-11T14:40:00.079Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: election won: term=2 tally=2 2024-06-11T14:40:00.079Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: entering leader state: leader="Node at 9f209674-06b7-fe72-5cf1-96651365946c [Leader]" 2024-06-11T14:40:00.079Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: added peer, starting replication: peer=server-2d300aa5-282f-9808-d086-11c07f5e9786 2024-06-11T14:40:00.079Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: added peer, starting replication: peer=server-4ee0f7d4-5b8d-114d-2b43-a522565c0943 2024-06-11T14:40:00.079Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: pipelining replication: peer="{Voter server-4ee0f7d4-5b8d-114d-2b43-a522565c0943 4ee0f7d4-5b8d-114d-2b43-a522565c0943}" 2024-06-11T14:40:00.079Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: pipelining replication: peer="{Voter server-2d300aa5-282f-9808-d086-11c07f5e9786 2d300aa5-282f-9808-d086-11c07f5e9786}" 2024-06-11T14:40:00.185Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:00.079915188 +0000 UTC m=+6.662994140 (2 nodes), 79.690644ms from start of poll, 80.305902ms from cluster start. Timeout at 2024-06-11 14:40:00.185307546 +0000 UTC m=+6.768386458, 105.392318ms after stability 2024-06-11T14:40:00.185Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:00.291Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:00.185435061 +0000 UTC m=+6.768513973 (1 nodes), 1.12µs from start of poll, 185.825735ms from cluster start. Timeout at 2024-06-11 14:40:00.291487394 +0000 UTC m=+6.874566346, 106.052373ms after stability 2024-06-11T14:40:00.303Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: aborting pipeline replication: peer="{Voter server-4ee0f7d4-5b8d-114d-2b43-a522565c0943 4ee0f7d4-5b8d-114d-2b43-a522565c0943}" 2024-06-11T14:40:00.303Z [INFO] server-9f209674-06b7-fe72-5cf1-96651365946c: aborting pipeline replication: peer="{Voter server-2d300aa5-282f-9808-d086-11c07f5e9786 2d300aa5-282f-9808-d086-11c07f5e9786}" --- PASS: TestRaft_TripleNode (0.31s) === RUN TestRaft_LeaderFail 2024-06-11T14:40:00.305Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c30b7dd8-a013-8674-e7e7-f52446b53437 Address:c30b7dd8-a013-8674-e7e7-f52446b53437} {Suffrage:Voter ID:server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 Address:d2e072ff-c445-db6c-c9cf-58b95ef6a518} {Suffrage:Voter ID:server-d80330b7-0ce8-8921-035b-9484ecc74ad0 Address:d80330b7-0ce8-8921-035b-9484ecc74ad0}]" 2024-06-11T14:40:00.305Z [INFO] server-d2e072ff-c445-db6c-c9cf-58b95ef6a518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c30b7dd8-a013-8674-e7e7-f52446b53437 Address:c30b7dd8-a013-8674-e7e7-f52446b53437} {Suffrage:Voter ID:server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 Address:d2e072ff-c445-db6c-c9cf-58b95ef6a518} {Suffrage:Voter ID:server-d80330b7-0ce8-8921-035b-9484ecc74ad0 Address:d80330b7-0ce8-8921-035b-9484ecc74ad0}]" 2024-06-11T14:40:00.306Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c30b7dd8-a013-8674-e7e7-f52446b53437 Address:c30b7dd8-a013-8674-e7e7-f52446b53437} {Suffrage:Voter ID:server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 Address:d2e072ff-c445-db6c-c9cf-58b95ef6a518} {Suffrage:Voter ID:server-d80330b7-0ce8-8921-035b-9484ecc74ad0 Address:d80330b7-0ce8-8921-035b-9484ecc74ad0}]" 2024-06-11T14:40:00.306Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:00.306Z [INFO] server-d2e072ff-c445-db6c-c9cf-58b95ef6a518: entering follower state: follower="Node at d2e072ff-c445-db6c-c9cf-58b95ef6a518 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.306Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: entering follower state: follower="Node at d80330b7-0ce8-8921-035b-9484ecc74ad0 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.306Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: entering follower state: follower="Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.372Z [WARN] server-c30b7dd8-a013-8674-e7e7-f52446b53437: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:00.372Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: entering candidate state: node="Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Candidate]" term=2 2024-06-11T14:40:00.372Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: election won: term=2 tally=2 2024-06-11T14:40:00.372Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: entering leader state: leader="Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Leader]" 2024-06-11T14:40:00.372Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: added peer, starting replication: peer=server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 2024-06-11T14:40:00.372Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: added peer, starting replication: peer=server-d80330b7-0ce8-8921-035b-9484ecc74ad0 2024-06-11T14:40:00.372Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: pipelining replication: peer="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" 2024-06-11T14:40:00.372Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: pipelining replication: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" 2024-06-11T14:40:00.479Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:00.37307329 +0000 UTC m=+6.956152202 (2 nodes), 66.970623ms from start of poll, 67.581001ms from cluster start. Timeout at 2024-06-11 14:40:00.478977749 +0000 UTC m=+7.062056701, 105.904499ms after stability 2024-06-11T14:40:00.479Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:00.584Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:00.479092105 +0000 UTC m=+7.062171057 (1 nodes), 1.28µs from start of poll, 173.599856ms from cluster start. Timeout at 2024-06-11 14:40:00.584684935 +0000 UTC m=+7.167763887, 105.59283ms after stability raft_test.go:378: [INFO] Disconnecting Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Leader] 2024-06-11T14:40:00.598Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to heartbeat to: peer=d2e072ff-c445-db6c-c9cf-58b95ef6a518 backoff time=10ms error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" 2024-06-11T14:40:00.598Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:00.599Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to pipeline appendEntries: peer="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" error="append pipeline closed" 2024-06-11T14:40:00.599Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: aborting pipeline replication: peer="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" 2024-06-11T14:40:00.600Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to pipeline appendEntries: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" error="append pipeline closed" 2024-06-11T14:40:00.600Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: aborting pipeline replication: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" 2024-06-11T14:40:00.603Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to heartbeat to: peer=d80330b7-0ce8-8921-035b-9484ecc74ad0 backoff time=10ms error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" 2024-06-11T14:40:00.607Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to appendEntries to: peer="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" 2024-06-11T14:40:00.610Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to appendEntries to: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" 2024-06-11T14:40:00.617Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to heartbeat to: peer=d2e072ff-c445-db6c-c9cf-58b95ef6a518 backoff time=10ms error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" 2024-06-11T14:40:00.622Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to heartbeat to: peer=d80330b7-0ce8-8921-035b-9484ecc74ad0 backoff time=10ms error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" 2024-06-11T14:40:00.626Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to appendEntries to: peer="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" 2024-06-11T14:40:00.630Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to appendEntries to: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" 2024-06-11T14:40:00.635Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to heartbeat to: peer=d2e072ff-c445-db6c-c9cf-58b95ef6a518 backoff time=10ms error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" 2024-06-11T14:40:00.641Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to heartbeat to: peer=d80330b7-0ce8-8921-035b-9484ecc74ad0 backoff time=10ms error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" 2024-06-11T14:40:00.642Z [WARN] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to contact: server-id=server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 time=50.241707ms 2024-06-11T14:40:00.646Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to appendEntries to: peer="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" 2024-06-11T14:40:00.647Z [WARN] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: heartbeat timeout reached, starting election: last-leader-addr=c30b7dd8-a013-8674-e7e7-f52446b53437 last-leader-id=server-c30b7dd8-a013-8674-e7e7-f52446b53437 2024-06-11T14:40:00.647Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: entering candidate state: node="Node at d80330b7-0ce8-8921-035b-9484ecc74ad0 [Candidate]" term=3 2024-06-11T14:40:00.647Z [WARN] server-d2e072ff-c445-db6c-c9cf-58b95ef6a518: rejecting vote request since we have a leader: from=d80330b7-0ce8-8921-035b-9484ecc74ad0 leader=c30b7dd8-a013-8674-e7e7-f52446b53437 leader-id=server-c30b7dd8-a013-8674-e7e7-f52446b53437 2024-06-11T14:40:00.647Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to make requestVote RPC: target="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" term=3 2024-06-11T14:40:00.650Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to appendEntries to: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" 2024-06-11T14:40:00.653Z [WARN] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to contact: server-id=server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 time=60.990359ms 2024-06-11T14:40:00.653Z [WARN] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to contact: server-id=server-d80330b7-0ce8-8921-035b-9484ecc74ad0 time=59.793723ms 2024-06-11T14:40:00.653Z [WARN] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to contact quorum of nodes, stepping down 2024-06-11T14:40:00.653Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: entering follower state: follower="Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.706Z [WARN] server-d2e072ff-c445-db6c-c9cf-58b95ef6a518: heartbeat timeout reached, starting election: last-leader-addr=c30b7dd8-a013-8674-e7e7-f52446b53437 last-leader-id=server-c30b7dd8-a013-8674-e7e7-f52446b53437 2024-06-11T14:40:00.706Z [INFO] server-d2e072ff-c445-db6c-c9cf-58b95ef6a518: entering candidate state: node="Node at d2e072ff-c445-db6c-c9cf-58b95ef6a518 [Candidate]" term=3 2024-06-11T14:40:00.706Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: duplicate requestVote for same term: term=3 2024-06-11T14:40:00.706Z [ERROR] server-d2e072ff-c445-db6c-c9cf-58b95ef6a518: failed to make requestVote RPC: target="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" term=3 2024-06-11T14:40:00.711Z [WARN] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: Election timeout reached, restarting election 2024-06-11T14:40:00.711Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: entering candidate state: node="Node at d80330b7-0ce8-8921-035b-9484ecc74ad0 [Candidate]" term=4 2024-06-11T14:40:00.711Z [INFO] server-d2e072ff-c445-db6c-c9cf-58b95ef6a518: entering follower state: follower="Node at d2e072ff-c445-db6c-c9cf-58b95ef6a518 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.711Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: election won: term=4 tally=2 2024-06-11T14:40:00.711Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: entering leader state: leader="Node at d80330b7-0ce8-8921-035b-9484ecc74ad0 [Leader]" 2024-06-11T14:40:00.712Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: added peer, starting replication: peer=server-c30b7dd8-a013-8674-e7e7-f52446b53437 2024-06-11T14:40:00.712Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: added peer, starting replication: peer=server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 2024-06-11T14:40:00.712Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: pipelining replication: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" 2024-06-11T14:40:00.712Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to make requestVote RPC: target="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" term=4 2024-06-11T14:40:00.712Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to appendEntries to: peer="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.721Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to heartbeat to: peer=c30b7dd8-a013-8674-e7e7-f52446b53437 backoff time=10ms error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.730Z [WARN] server-c30b7dd8-a013-8674-e7e7-f52446b53437: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:00.730Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: entering candidate state: node="Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Candidate]" term=3 2024-06-11T14:40:00.730Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to make requestVote RPC: target="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" term=3 2024-06-11T14:40:00.730Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to make requestVote RPC: target="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" term=3 2024-06-11T14:40:00.733Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to appendEntries to: peer="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.741Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to heartbeat to: peer=c30b7dd8-a013-8674-e7e7-f52446b53437 backoff time=10ms error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.753Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to appendEntries to: peer="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.760Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to heartbeat to: peer=c30b7dd8-a013-8674-e7e7-f52446b53437 backoff time=10ms error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.762Z [WARN] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to contact: server-id=server-c30b7dd8-a013-8674-e7e7-f52446b53437 time=50.911163ms 2024-06-11T14:40:00.777Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to heartbeat to: peer=c30b7dd8-a013-8674-e7e7-f52446b53437 backoff time=20ms error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.781Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to appendEntries to: peer="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.806Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to heartbeat to: peer=c30b7dd8-a013-8674-e7e7-f52446b53437 backoff time=25ms error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.807Z [WARN] server-c30b7dd8-a013-8674-e7e7-f52446b53437: Election timeout reached, restarting election 2024-06-11T14:40:00.807Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: entering candidate state: node="Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Candidate]" term=4 2024-06-11T14:40:00.807Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to make requestVote RPC: target="{Voter server-d80330b7-0ce8-8921-035b-9484ecc74ad0 d80330b7-0ce8-8921-035b-9484ecc74ad0}" error="failed to connect to peer: d80330b7-0ce8-8921-035b-9484ecc74ad0" term=4 2024-06-11T14:40:00.808Z [ERROR] server-c30b7dd8-a013-8674-e7e7-f52446b53437: failed to make requestVote RPC: target="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" error="failed to connect to peer: d2e072ff-c445-db6c-c9cf-58b95ef6a518" term=4 2024-06-11T14:40:00.809Z [WARN] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to contact: server-id=server-c30b7dd8-a013-8674-e7e7-f52446b53437 time=97.217452ms 2024-06-11T14:40:00.827Z [ERROR] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: failed to appendEntries to: peer="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" error="failed to connect to peer: c30b7dd8-a013-8674-e7e7-f52446b53437" 2024-06-11T14:40:00.835Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:00.730672947 +0000 UTC m=+7.313751899 (1 nodes), 132.373943ms from start of poll, 425.180698ms from cluster start. Timeout at 2024-06-11 14:40:00.835844272 +0000 UTC m=+7.418923224, 105.171325ms after stability raft_test.go:413: [INFO] Reconnecting Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Candidate] 2024-06-11T14:40:00.838Z [INFO] server-c30b7dd8-a013-8674-e7e7-f52446b53437: entering follower state: follower="Node at c30b7dd8-a013-8674-e7e7-f52446b53437 [Follower]" leader-address=d80330b7-0ce8-8921-035b-9484ecc74ad0 leader-id=server-d80330b7-0ce8-8921-035b-9484ecc74ad0 2024-06-11T14:40:00.915Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: pipelining replication: peer="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" 2024-06-11T14:40:00.917Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: aborting pipeline replication: peer="{Voter server-d2e072ff-c445-db6c-c9cf-58b95ef6a518 d2e072ff-c445-db6c-c9cf-58b95ef6a518}" 2024-06-11T14:40:00.918Z [INFO] server-d80330b7-0ce8-8921-035b-9484ecc74ad0: aborting pipeline replication: peer="{Voter server-c30b7dd8-a013-8674-e7e7-f52446b53437 c30b7dd8-a013-8674-e7e7-f52446b53437}" --- PASS: TestRaft_LeaderFail (0.61s) === RUN TestRaft_BehindFollower 2024-06-11T14:40:00.920Z [INFO] server-6e4a2416-99ae-eca6-383b-c67797624f78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e4a2416-99ae-eca6-383b-c67797624f78 Address:6e4a2416-99ae-eca6-383b-c67797624f78} {Suffrage:Voter ID:server-de746fd8-8d65-2e17-1e63-6fb1b6970662 Address:de746fd8-8d65-2e17-1e63-6fb1b6970662} {Suffrage:Voter ID:server-6018ad6a-4a85-caec-80aa-6308e07649c2 Address:6018ad6a-4a85-caec-80aa-6308e07649c2}]" 2024-06-11T14:40:00.920Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e4a2416-99ae-eca6-383b-c67797624f78 Address:6e4a2416-99ae-eca6-383b-c67797624f78} {Suffrage:Voter ID:server-de746fd8-8d65-2e17-1e63-6fb1b6970662 Address:de746fd8-8d65-2e17-1e63-6fb1b6970662} {Suffrage:Voter ID:server-6018ad6a-4a85-caec-80aa-6308e07649c2 Address:6018ad6a-4a85-caec-80aa-6308e07649c2}]" 2024-06-11T14:40:00.920Z [INFO] server-6018ad6a-4a85-caec-80aa-6308e07649c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e4a2416-99ae-eca6-383b-c67797624f78 Address:6e4a2416-99ae-eca6-383b-c67797624f78} {Suffrage:Voter ID:server-de746fd8-8d65-2e17-1e63-6fb1b6970662 Address:de746fd8-8d65-2e17-1e63-6fb1b6970662} {Suffrage:Voter ID:server-6018ad6a-4a85-caec-80aa-6308e07649c2 Address:6018ad6a-4a85-caec-80aa-6308e07649c2}]" 2024-06-11T14:40:00.920Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:00.920Z [INFO] server-6e4a2416-99ae-eca6-383b-c67797624f78: entering follower state: follower="Node at 6e4a2416-99ae-eca6-383b-c67797624f78 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.920Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: entering follower state: follower="Node at de746fd8-8d65-2e17-1e63-6fb1b6970662 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.920Z [INFO] server-6018ad6a-4a85-caec-80aa-6308e07649c2: entering follower state: follower="Node at 6018ad6a-4a85-caec-80aa-6308e07649c2 [Follower]" leader-address= leader-id= 2024-06-11T14:40:00.985Z [WARN] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:00.985Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: entering candidate state: node="Node at de746fd8-8d65-2e17-1e63-6fb1b6970662 [Candidate]" term=2 2024-06-11T14:40:00.985Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: election won: term=2 tally=2 2024-06-11T14:40:00.985Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: entering leader state: leader="Node at de746fd8-8d65-2e17-1e63-6fb1b6970662 [Leader]" 2024-06-11T14:40:00.985Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: added peer, starting replication: peer=server-6e4a2416-99ae-eca6-383b-c67797624f78 2024-06-11T14:40:00.985Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: added peer, starting replication: peer=server-6018ad6a-4a85-caec-80aa-6308e07649c2 2024-06-11T14:40:00.985Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: pipelining replication: peer="{Voter server-6018ad6a-4a85-caec-80aa-6308e07649c2 6018ad6a-4a85-caec-80aa-6308e07649c2}" 2024-06-11T14:40:00.986Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: pipelining replication: peer="{Voter server-6e4a2416-99ae-eca6-383b-c67797624f78 6e4a2416-99ae-eca6-383b-c67797624f78}" 2024-06-11T14:40:01.091Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:00.98638976 +0000 UTC m=+7.569468712 (1 nodes), 65.899782ms from start of poll, 66.50632ms from cluster start. Timeout at 2024-06-11 14:40:01.091666442 +0000 UTC m=+7.674745354, 105.276642ms after stability 2024-06-11T14:40:01.091Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:01.197Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:01.091831196 +0000 UTC m=+7.674910148 (2 nodes), 1.6µs from start of poll, 171.947756ms from cluster start. Timeout at 2024-06-11 14:40:01.197332189 +0000 UTC m=+7.780411101, 105.500953ms after stability 2024-06-11T14:40:01.197Z [ERROR] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: failed to pipeline appendEntries: peer="{Voter server-6e4a2416-99ae-eca6-383b-c67797624f78 6e4a2416-99ae-eca6-383b-c67797624f78}" error="append pipeline closed" 2024-06-11T14:40:01.197Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: aborting pipeline replication: peer="{Voter server-6e4a2416-99ae-eca6-383b-c67797624f78 6e4a2416-99ae-eca6-383b-c67797624f78}" 2024-06-11T14:40:01.197Z [ERROR] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: failed to appendEntries to: peer="{Voter server-6e4a2416-99ae-eca6-383b-c67797624f78 6e4a2416-99ae-eca6-383b-c67797624f78}" error="failed to connect to peer: 6e4a2416-99ae-eca6-383b-c67797624f78" 2024-06-11T14:40:01.198Z [ERROR] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: failed to heartbeat to: peer=6e4a2416-99ae-eca6-383b-c67797624f78 backoff time=10ms error="failed to connect to peer: 6e4a2416-99ae-eca6-383b-c67797624f78" raft_test.go:459: [INFO] Finished apply without behind follower 2024-06-11T14:40:01.209Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: pipelining replication: peer="{Voter server-6e4a2416-99ae-eca6-383b-c67797624f78 6e4a2416-99ae-eca6-383b-c67797624f78}" 2024-06-11T14:40:01.217Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:01.322Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:01.217441143 +0000 UTC m=+7.800520095 (1 nodes), 1.32µs from start of poll, 297.557703ms from cluster start. Timeout at 2024-06-11 14:40:01.322821301 +0000 UTC m=+7.905900213, 105.380118ms after stability 2024-06-11T14:40:01.322Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: aborting pipeline replication: peer="{Voter server-6018ad6a-4a85-caec-80aa-6308e07649c2 6018ad6a-4a85-caec-80aa-6308e07649c2}" 2024-06-11T14:40:01.323Z [INFO] server-de746fd8-8d65-2e17-1e63-6fb1b6970662: aborting pipeline replication: peer="{Voter server-6e4a2416-99ae-eca6-383b-c67797624f78 6e4a2416-99ae-eca6-383b-c67797624f78}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader 2024-06-11T14:40:01.325Z [INFO] server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc Address:458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc} {Suffrage:Voter ID:server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2 Address:c70f50b9-e36b-c0b3-335f-3f0b84a83bf2} {Suffrage:Voter ID:server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066 Address:ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066}]" 2024-06-11T14:40:01.325Z [INFO] server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc Address:458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc} {Suffrage:Voter ID:server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2 Address:c70f50b9-e36b-c0b3-335f-3f0b84a83bf2} {Suffrage:Voter ID:server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066 Address:ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066}]" 2024-06-11T14:40:01.325Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc Address:458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc} {Suffrage:Voter ID:server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2 Address:c70f50b9-e36b-c0b3-335f-3f0b84a83bf2} {Suffrage:Voter ID:server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066 Address:ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066}]" 2024-06-11T14:40:01.325Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:01.325Z [INFO] server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc: entering follower state: follower="Node at 458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc [Follower]" leader-address= leader-id= 2024-06-11T14:40:01.325Z [INFO] server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2: entering follower state: follower="Node at c70f50b9-e36b-c0b3-335f-3f0b84a83bf2 [Follower]" leader-address= leader-id= 2024-06-11T14:40:01.325Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: entering follower state: follower="Node at ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066 [Follower]" leader-address= leader-id= 2024-06-11T14:40:01.378Z [WARN] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:01.378Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: entering candidate state: node="Node at ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066 [Candidate]" term=2 2024-06-11T14:40:01.378Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: election won: term=2 tally=2 2024-06-11T14:40:01.378Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: entering leader state: leader="Node at ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066 [Leader]" 2024-06-11T14:40:01.378Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: added peer, starting replication: peer=server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc 2024-06-11T14:40:01.378Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: added peer, starting replication: peer=server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2 2024-06-11T14:40:01.378Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: pipelining replication: peer="{Voter server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2 c70f50b9-e36b-c0b3-335f-3f0b84a83bf2}" 2024-06-11T14:40:01.379Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: pipelining replication: peer="{Voter server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc 458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc}" 2024-06-11T14:40:01.485Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:01.379273304 +0000 UTC m=+7.962352256 (1 nodes), 53.659624ms from start of poll, 54.276042ms from cluster start. Timeout at 2024-06-11 14:40:01.485436033 +0000 UTC m=+8.068514945, 106.162689ms after stability 2024-06-11T14:40:01.485Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:01.590Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:01.485551469 +0000 UTC m=+8.068630421 (2 nodes), 1.72µs from start of poll, 160.554207ms from cluster start. Timeout at 2024-06-11 14:40:01.590882948 +0000 UTC m=+8.173961860, 105.331439ms after stability 2024-06-11T14:40:01.591Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: aborting pipeline replication: peer="{Voter server-458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc 458ba1af-e3df-0dc9-f0b4-1047c5e7b2cc}" 2024-06-11T14:40:01.591Z [INFO] server-ce6a2dfb-9c59-bbaf-fbfd-4f5333af4066: aborting pipeline replication: peer="{Voter server-c70f50b9-e36b-c0b3-335f-3f0b84a83bf2 c70f50b9-e36b-c0b3-335f-3f0b84a83bf2}" --- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent 2024-06-11T14:40:01.593Z [INFO] server-be349841-4d17-ca74-c37b-402a8440e7d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be349841-4d17-ca74-c37b-402a8440e7d2 Address:be349841-4d17-ca74-c37b-402a8440e7d2} {Suffrage:Voter ID:server-266c8975-b5d6-8b91-8c4e-22be9ba965b5 Address:266c8975-b5d6-8b91-8c4e-22be9ba965b5} {Suffrage:Voter ID:server-307b00df-d955-f3ba-a8e3-30bb894a778a Address:307b00df-d955-f3ba-a8e3-30bb894a778a}]" 2024-06-11T14:40:01.593Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be349841-4d17-ca74-c37b-402a8440e7d2 Address:be349841-4d17-ca74-c37b-402a8440e7d2} {Suffrage:Voter ID:server-266c8975-b5d6-8b91-8c4e-22be9ba965b5 Address:266c8975-b5d6-8b91-8c4e-22be9ba965b5} {Suffrage:Voter ID:server-307b00df-d955-f3ba-a8e3-30bb894a778a Address:307b00df-d955-f3ba-a8e3-30bb894a778a}]" 2024-06-11T14:40:01.593Z [INFO] server-307b00df-d955-f3ba-a8e3-30bb894a778a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be349841-4d17-ca74-c37b-402a8440e7d2 Address:be349841-4d17-ca74-c37b-402a8440e7d2} {Suffrage:Voter ID:server-266c8975-b5d6-8b91-8c4e-22be9ba965b5 Address:266c8975-b5d6-8b91-8c4e-22be9ba965b5} {Suffrage:Voter ID:server-307b00df-d955-f3ba-a8e3-30bb894a778a Address:307b00df-d955-f3ba-a8e3-30bb894a778a}]" 2024-06-11T14:40:01.593Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:01.593Z [INFO] server-be349841-4d17-ca74-c37b-402a8440e7d2: entering follower state: follower="Node at be349841-4d17-ca74-c37b-402a8440e7d2 [Follower]" leader-address= leader-id= 2024-06-11T14:40:01.593Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: entering follower state: follower="Node at 266c8975-b5d6-8b91-8c4e-22be9ba965b5 [Follower]" leader-address= leader-id= 2024-06-11T14:40:01.593Z [INFO] server-307b00df-d955-f3ba-a8e3-30bb894a778a: entering follower state: follower="Node at 307b00df-d955-f3ba-a8e3-30bb894a778a [Follower]" leader-address= leader-id= 2024-06-11T14:40:01.701Z [WARN] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:01.701Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: entering candidate state: node="Node at 266c8975-b5d6-8b91-8c4e-22be9ba965b5 [Candidate]" term=2 2024-06-11T14:40:01.701Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: election won: term=2 tally=2 2024-06-11T14:40:01.701Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: entering leader state: leader="Node at 266c8975-b5d6-8b91-8c4e-22be9ba965b5 [Leader]" 2024-06-11T14:40:01.701Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: added peer, starting replication: peer=server-be349841-4d17-ca74-c37b-402a8440e7d2 2024-06-11T14:40:01.701Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: added peer, starting replication: peer=server-307b00df-d955-f3ba-a8e3-30bb894a778a 2024-06-11T14:40:01.702Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: pipelining replication: peer="{Voter server-307b00df-d955-f3ba-a8e3-30bb894a778a 307b00df-d955-f3ba-a8e3-30bb894a778a}" 2024-06-11T14:40:01.702Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: pipelining replication: peer="{Voter server-be349841-4d17-ca74-c37b-402a8440e7d2 be349841-4d17-ca74-c37b-402a8440e7d2}" 2024-06-11T14:40:01.908Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:01.702466202 +0000 UTC m=+8.285545154 (1 nodes), 108.856952ms from start of poll, 109.47789ms from cluster start. Timeout at 2024-06-11 14:40:01.908316534 +0000 UTC m=+8.491395486, 205.850332ms after stability 2024-06-11T14:40:01.927Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: aborting pipeline replication: peer="{Voter server-307b00df-d955-f3ba-a8e3-30bb894a778a 307b00df-d955-f3ba-a8e3-30bb894a778a}" 2024-06-11T14:40:01.927Z [INFO] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: aborting pipeline replication: peer="{Voter server-be349841-4d17-ca74-c37b-402a8440e7d2 be349841-4d17-ca74-c37b-402a8440e7d2}" 2024-06-11T14:40:02.428Z [ERROR] server-266c8975-b5d6-8b91-8c4e-22be9ba965b5: failed to heartbeat to: peer=be349841-4d17-ca74-c37b-402a8440e7d2 backoff time=10ms error="command timed out" --- PASS: TestRaft_ApplyConcurrent (0.84s) === RUN TestRaft_ApplyConcurrent_Timeout 2024-06-11T14:40:02.430Z [INFO] server-fe4cb622-5b9e-6ca8-74e9-067e658f35e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe4cb622-5b9e-6ca8-74e9-067e658f35e7 Address:fe4cb622-5b9e-6ca8-74e9-067e658f35e7}]" 2024-06-11T14:40:02.430Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:02.430Z [INFO] server-fe4cb622-5b9e-6ca8-74e9-067e658f35e7: entering follower state: follower="Node at fe4cb622-5b9e-6ca8-74e9-067e658f35e7 [Follower]" leader-address= leader-id= 2024-06-11T14:40:02.612Z [WARN] server-fe4cb622-5b9e-6ca8-74e9-067e658f35e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:02.612Z [INFO] server-fe4cb622-5b9e-6ca8-74e9-067e658f35e7: entering candidate state: node="Node at fe4cb622-5b9e-6ca8-74e9-067e658f35e7 [Candidate]" term=2 2024-06-11T14:40:02.612Z [INFO] server-fe4cb622-5b9e-6ca8-74e9-067e658f35e7: election won: term=2 tally=1 2024-06-11T14:40:02.612Z [INFO] server-fe4cb622-5b9e-6ca8-74e9-067e658f35e7: entering leader state: leader="Node at fe4cb622-5b9e-6ca8-74e9-067e658f35e7 [Leader]" 2024-06-11T14:40:02.813Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:02.612427368 +0000 UTC m=+9.195506320 (1 nodes), 182.08679ms from start of poll, 182.447017ms from cluster start. Timeout at 2024-06-11 14:40:02.813521792 +0000 UTC m=+9.396600744, 201.094424ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.39s) === RUN TestRaft_JoinNode 2024-06-11T14:40:02.821Z [INFO] server-fd0f304b-979a-5c10-f988-66d58df8070c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd0f304b-979a-5c10-f988-66d58df8070c Address:fd0f304b-979a-5c10-f988-66d58df8070c} {Suffrage:Voter ID:server-8e679f23-4520-2321-5f05-4e3a2b759f8e Address:8e679f23-4520-2321-5f05-4e3a2b759f8e}]" 2024-06-11T14:40:02.822Z [INFO] server-fd0f304b-979a-5c10-f988-66d58df8070c: entering follower state: follower="Node at fd0f304b-979a-5c10-f988-66d58df8070c [Follower]" leader-address= leader-id= 2024-06-11T14:40:02.822Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd0f304b-979a-5c10-f988-66d58df8070c Address:fd0f304b-979a-5c10-f988-66d58df8070c} {Suffrage:Voter ID:server-8e679f23-4520-2321-5f05-4e3a2b759f8e Address:8e679f23-4520-2321-5f05-4e3a2b759f8e}]" 2024-06-11T14:40:02.822Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: entering follower state: follower="Node at 8e679f23-4520-2321-5f05-4e3a2b759f8e [Follower]" leader-address= leader-id= 2024-06-11T14:40:02.822Z [INFO] server-d360f881-e634-bf79-ef63-2dfe5ca15f40: initial configuration: index=0 servers=[] 2024-06-11T14:40:02.822Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:02.822Z [INFO] server-d360f881-e634-bf79-ef63-2dfe5ca15f40: entering follower state: follower="Node at d360f881-e634-bf79-ef63-2dfe5ca15f40 [Follower]" leader-address= leader-id= 2024-06-11T14:40:02.893Z [WARN] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:02.893Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: entering candidate state: node="Node at 8e679f23-4520-2321-5f05-4e3a2b759f8e [Candidate]" term=2 2024-06-11T14:40:02.893Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: election won: term=2 tally=2 2024-06-11T14:40:02.893Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: entering leader state: leader="Node at 8e679f23-4520-2321-5f05-4e3a2b759f8e [Leader]" 2024-06-11T14:40:02.893Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: added peer, starting replication: peer=server-fd0f304b-979a-5c10-f988-66d58df8070c 2024-06-11T14:40:02.894Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: pipelining replication: peer="{Voter server-fd0f304b-979a-5c10-f988-66d58df8070c fd0f304b-979a-5c10-f988-66d58df8070c}" 2024-06-11T14:40:02.923Z [WARN] server-d360f881-e634-bf79-ef63-2dfe5ca15f40: no known peers, aborting election 2024-06-11T14:40:02.999Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:02.894112444 +0000 UTC m=+9.477191356 (1 nodes), 71.340066ms from start of poll, 72.448346ms from cluster start. Timeout at 2024-06-11 14:40:02.999614797 +0000 UTC m=+9.582693749, 105.502393ms after stability 2024-06-11T14:40:02.999Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: updating configuration: command=AddVoter server-id=server-d360f881-e634-bf79-ef63-2dfe5ca15f40 server-addr=d360f881-e634-bf79-ef63-2dfe5ca15f40 servers="[{Suffrage:Voter ID:server-fd0f304b-979a-5c10-f988-66d58df8070c Address:fd0f304b-979a-5c10-f988-66d58df8070c} {Suffrage:Voter ID:server-8e679f23-4520-2321-5f05-4e3a2b759f8e Address:8e679f23-4520-2321-5f05-4e3a2b759f8e} {Suffrage:Voter ID:server-d360f881-e634-bf79-ef63-2dfe5ca15f40 Address:d360f881-e634-bf79-ef63-2dfe5ca15f40}]" 2024-06-11T14:40:02.999Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: added peer, starting replication: peer=server-d360f881-e634-bf79-ef63-2dfe5ca15f40 2024-06-11T14:40:03.000Z [WARN] server-d360f881-e634-bf79-ef63-2dfe5ca15f40: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2024-06-11T14:40:03.000Z [WARN] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: appendEntries rejected, sending older logs: peer="{Voter server-d360f881-e634-bf79-ef63-2dfe5ca15f40 d360f881-e634-bf79-ef63-2dfe5ca15f40}" next=1 2024-06-11T14:40:03.000Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:03.000Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: pipelining replication: peer="{Voter server-d360f881-e634-bf79-ef63-2dfe5ca15f40 d360f881-e634-bf79-ef63-2dfe5ca15f40}" 2024-06-11T14:40:03.105Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:03.000297253 +0000 UTC m=+9.583376205 (1 nodes), 1.32µs from start of poll, 178.633195ms from cluster start. Timeout at 2024-06-11 14:40:03.1053933 +0000 UTC m=+9.688472212, 105.096007ms after stability 2024-06-11T14:40:03.105Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: aborting pipeline replication: peer="{Voter server-d360f881-e634-bf79-ef63-2dfe5ca15f40 d360f881-e634-bf79-ef63-2dfe5ca15f40}" 2024-06-11T14:40:03.105Z [INFO] server-8e679f23-4520-2321-5f05-4e3a2b759f8e: aborting pipeline replication: peer="{Voter server-fd0f304b-979a-5c10-f988-66d58df8070c fd0f304b-979a-5c10-f988-66d58df8070c}" --- PASS: TestRaft_JoinNode (0.29s) === RUN TestRaft_JoinNode_ConfigStore 2024-06-11T14:40:03.107Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aaf465c4-890a-08e5-0b62-c733fccbcbda Address:aaf465c4-890a-08e5-0b62-c733fccbcbda}]" 2024-06-11T14:40:03.107Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: entering follower state: follower="Node at aaf465c4-890a-08e5-0b62-c733fccbcbda [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.108Z [INFO] server-4b8db6a8-a1f5-2367-037f-6477bba82815: initial configuration: index=0 servers=[] 2024-06-11T14:40:03.108Z [INFO] server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278: initial configuration: index=0 servers=[] 2024-06-11T14:40:03.108Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:03.108Z [INFO] server-4b8db6a8-a1f5-2367-037f-6477bba82815: entering follower state: follower="Node at 4b8db6a8-a1f5-2367-037f-6477bba82815 [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.108Z [INFO] server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278: entering follower state: follower="Node at f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.199Z [WARN] server-4b8db6a8-a1f5-2367-037f-6477bba82815: no known peers, aborting election 2024-06-11T14:40:03.202Z [WARN] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:03.202Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: entering candidate state: node="Node at aaf465c4-890a-08e5-0b62-c733fccbcbda [Candidate]" term=2 2024-06-11T14:40:03.202Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: election won: term=2 tally=1 2024-06-11T14:40:03.202Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: entering leader state: leader="Node at aaf465c4-890a-08e5-0b62-c733fccbcbda [Leader]" 2024-06-11T14:40:03.208Z [WARN] server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278: no known peers, aborting election 2024-06-11T14:40:03.314Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:03.202908862 +0000 UTC m=+9.785987774 (1 nodes), 94.395394ms from start of poll, 95.533113ms from cluster start. Timeout at 2024-06-11 14:40:03.314233765 +0000 UTC m=+9.897312717, 111.324943ms after stability 2024-06-11T14:40:03.314Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: updating configuration: command=AddVoter server-id=server-4b8db6a8-a1f5-2367-037f-6477bba82815 server-addr=4b8db6a8-a1f5-2367-037f-6477bba82815 servers="[{Suffrage:Voter ID:server-aaf465c4-890a-08e5-0b62-c733fccbcbda Address:aaf465c4-890a-08e5-0b62-c733fccbcbda} {Suffrage:Voter ID:server-4b8db6a8-a1f5-2367-037f-6477bba82815 Address:4b8db6a8-a1f5-2367-037f-6477bba82815}]" 2024-06-11T14:40:03.314Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: added peer, starting replication: peer=server-4b8db6a8-a1f5-2367-037f-6477bba82815 2024-06-11T14:40:03.314Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:03.314Z [WARN] server-4b8db6a8-a1f5-2367-037f-6477bba82815: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2024-06-11T14:40:03.314Z [WARN] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: appendEntries rejected, sending older logs: peer="{Voter server-4b8db6a8-a1f5-2367-037f-6477bba82815 4b8db6a8-a1f5-2367-037f-6477bba82815}" next=1 2024-06-11T14:40:03.315Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: pipelining replication: peer="{Voter server-4b8db6a8-a1f5-2367-037f-6477bba82815 4b8db6a8-a1f5-2367-037f-6477bba82815}" 2024-06-11T14:40:03.420Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:03.314814224 +0000 UTC m=+9.897893136 (1 nodes), 1.48µs from start of poll, 207.438475ms from cluster start. Timeout at 2024-06-11 14:40:03.420737242 +0000 UTC m=+10.003816154, 105.923018ms after stability 2024-06-11T14:40:03.420Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: updating configuration: command=AddVoter server-id=server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 server-addr=f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 servers="[{Suffrage:Voter ID:server-aaf465c4-890a-08e5-0b62-c733fccbcbda Address:aaf465c4-890a-08e5-0b62-c733fccbcbda} {Suffrage:Voter ID:server-4b8db6a8-a1f5-2367-037f-6477bba82815 Address:4b8db6a8-a1f5-2367-037f-6477bba82815} {Suffrage:Voter ID:server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 Address:f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278}]" 2024-06-11T14:40:03.421Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: added peer, starting replication: peer=server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 2024-06-11T14:40:03.421Z [WARN] server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2024-06-11T14:40:03.421Z [WARN] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: appendEntries rejected, sending older logs: peer="{Voter server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278}" next=1 2024-06-11T14:40:03.421Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: pipelining replication: peer="{Voter server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278}" 2024-06-11T14:40:03.421Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:03.526Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:03.421690248 +0000 UTC m=+10.004769200 (1 nodes), 1.36µs from start of poll, 314.314539ms from cluster start. Timeout at 2024-06-11 14:40:03.526876972 +0000 UTC m=+10.109955924, 105.186724ms after stability 2024-06-11T14:40:03.527Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: aborting pipeline replication: peer="{Voter server-f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278 f2cc8e5a-52ac-f1b6-a6b6-1bdc82962278}" 2024-06-11T14:40:03.527Z [INFO] server-aaf465c4-890a-08e5-0b62-c733fccbcbda: aborting pipeline replication: peer="{Voter server-4b8db6a8-a1f5-2367-037f-6477bba82815 4b8db6a8-a1f5-2367-037f-6477bba82815}" --- PASS: TestRaft_JoinNode_ConfigStore (0.42s) === RUN TestRaft_RemoveFollower 2024-06-11T14:40:03.529Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79359c09-1f09-d8b2-64d7-f3153348b3a2 Address:79359c09-1f09-d8b2-64d7-f3153348b3a2} {Suffrage:Voter ID:server-c98b01b7-3106-bc41-8f70-8f7064f0e056 Address:c98b01b7-3106-bc41-8f70-8f7064f0e056} {Suffrage:Voter ID:server-7102f276-046f-bff8-b4fc-c30b39264e74 Address:7102f276-046f-bff8-b4fc-c30b39264e74}]" 2024-06-11T14:40:03.529Z [INFO] server-c98b01b7-3106-bc41-8f70-8f7064f0e056: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79359c09-1f09-d8b2-64d7-f3153348b3a2 Address:79359c09-1f09-d8b2-64d7-f3153348b3a2} {Suffrage:Voter ID:server-c98b01b7-3106-bc41-8f70-8f7064f0e056 Address:c98b01b7-3106-bc41-8f70-8f7064f0e056} {Suffrage:Voter ID:server-7102f276-046f-bff8-b4fc-c30b39264e74 Address:7102f276-046f-bff8-b4fc-c30b39264e74}]" 2024-06-11T14:40:03.529Z [INFO] server-7102f276-046f-bff8-b4fc-c30b39264e74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79359c09-1f09-d8b2-64d7-f3153348b3a2 Address:79359c09-1f09-d8b2-64d7-f3153348b3a2} {Suffrage:Voter ID:server-c98b01b7-3106-bc41-8f70-8f7064f0e056 Address:c98b01b7-3106-bc41-8f70-8f7064f0e056} {Suffrage:Voter ID:server-7102f276-046f-bff8-b4fc-c30b39264e74 Address:7102f276-046f-bff8-b4fc-c30b39264e74}]" 2024-06-11T14:40:03.529Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:03.529Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: entering follower state: follower="Node at 79359c09-1f09-d8b2-64d7-f3153348b3a2 [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.529Z [INFO] server-c98b01b7-3106-bc41-8f70-8f7064f0e056: entering follower state: follower="Node at c98b01b7-3106-bc41-8f70-8f7064f0e056 [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.529Z [INFO] server-7102f276-046f-bff8-b4fc-c30b39264e74: entering follower state: follower="Node at 7102f276-046f-bff8-b4fc-c30b39264e74 [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.594Z [WARN] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:03.594Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: entering candidate state: node="Node at 79359c09-1f09-d8b2-64d7-f3153348b3a2 [Candidate]" term=2 2024-06-11T14:40:03.594Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: election won: term=2 tally=2 2024-06-11T14:40:03.594Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: entering leader state: leader="Node at 79359c09-1f09-d8b2-64d7-f3153348b3a2 [Leader]" 2024-06-11T14:40:03.594Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: added peer, starting replication: peer=server-c98b01b7-3106-bc41-8f70-8f7064f0e056 2024-06-11T14:40:03.594Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: added peer, starting replication: peer=server-7102f276-046f-bff8-b4fc-c30b39264e74 2024-06-11T14:40:03.594Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: pipelining replication: peer="{Voter server-7102f276-046f-bff8-b4fc-c30b39264e74 7102f276-046f-bff8-b4fc-c30b39264e74}" 2024-06-11T14:40:03.594Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: pipelining replication: peer="{Voter server-c98b01b7-3106-bc41-8f70-8f7064f0e056 c98b01b7-3106-bc41-8f70-8f7064f0e056}" 2024-06-11T14:40:03.700Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:03.594967595 +0000 UTC m=+10.178046507 (1 nodes), 65.236086ms from start of poll, 65.836264ms from cluster start. Timeout at 2024-06-11 14:40:03.700724259 +0000 UTC m=+10.283803211, 105.756704ms after stability 2024-06-11T14:40:03.706Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:03.811Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:03.706268179 +0000 UTC m=+10.289347091 (2 nodes), 1.68µs from start of poll, 177.136848ms from cluster start. Timeout at 2024-06-11 14:40:03.811481543 +0000 UTC m=+10.394560455, 105.213364ms after stability 2024-06-11T14:40:03.811Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: updating configuration: command=RemoveServer server-id=server-c98b01b7-3106-bc41-8f70-8f7064f0e056 server-addr= servers="[{Suffrage:Voter ID:server-79359c09-1f09-d8b2-64d7-f3153348b3a2 Address:79359c09-1f09-d8b2-64d7-f3153348b3a2} {Suffrage:Voter ID:server-7102f276-046f-bff8-b4fc-c30b39264e74 Address:7102f276-046f-bff8-b4fc-c30b39264e74}]" 2024-06-11T14:40:03.811Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: removed peer, stopping replication: peer=server-c98b01b7-3106-bc41-8f70-8f7064f0e056 last-index=3 2024-06-11T14:40:03.811Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: aborting pipeline replication: peer="{Voter server-c98b01b7-3106-bc41-8f70-8f7064f0e056 c98b01b7-3106-bc41-8f70-8f7064f0e056}" 2024-06-11T14:40:03.867Z [WARN] server-c98b01b7-3106-bc41-8f70-8f7064f0e056: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2024-06-11T14:40:03.917Z [INFO] server-79359c09-1f09-d8b2-64d7-f3153348b3a2: aborting pipeline replication: peer="{Voter server-7102f276-046f-bff8-b4fc-c30b39264e74 7102f276-046f-bff8-b4fc-c30b39264e74}" --- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader 2024-06-11T14:40:03.922Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a Address:0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a} {Suffrage:Voter ID:server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 Address:ef7f00ef-c05b-8990-8edd-1c953bbf3bf5} {Suffrage:Voter ID:server-c73d1d24-9212-86cf-e97d-353ee57be48f Address:c73d1d24-9212-86cf-e97d-353ee57be48f}]" 2024-06-11T14:40:03.922Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a Address:0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a} {Suffrage:Voter ID:server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 Address:ef7f00ef-c05b-8990-8edd-1c953bbf3bf5} {Suffrage:Voter ID:server-c73d1d24-9212-86cf-e97d-353ee57be48f Address:c73d1d24-9212-86cf-e97d-353ee57be48f}]" 2024-06-11T14:40:03.923Z [INFO] server-c73d1d24-9212-86cf-e97d-353ee57be48f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a Address:0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a} {Suffrage:Voter ID:server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 Address:ef7f00ef-c05b-8990-8edd-1c953bbf3bf5} {Suffrage:Voter ID:server-c73d1d24-9212-86cf-e97d-353ee57be48f Address:c73d1d24-9212-86cf-e97d-353ee57be48f}]" 2024-06-11T14:40:03.923Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:03.923Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: entering follower state: follower="Node at 0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.923Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: entering follower state: follower="Node at ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 [Follower]" leader-address= leader-id= 2024-06-11T14:40:03.923Z [INFO] server-c73d1d24-9212-86cf-e97d-353ee57be48f: entering follower state: follower="Node at c73d1d24-9212-86cf-e97d-353ee57be48f [Follower]" leader-address= leader-id= 2024-06-11T14:40:04.006Z [WARN] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:04.006Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: entering candidate state: node="Node at ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 [Candidate]" term=2 2024-06-11T14:40:04.006Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: election won: term=2 tally=2 2024-06-11T14:40:04.006Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: entering leader state: leader="Node at ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 [Leader]" 2024-06-11T14:40:04.006Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: added peer, starting replication: peer=server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a 2024-06-11T14:40:04.006Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: added peer, starting replication: peer=server-c73d1d24-9212-86cf-e97d-353ee57be48f 2024-06-11T14:40:04.006Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: pipelining replication: peer="{Voter server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a 0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a}" 2024-06-11T14:40:04.007Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: pipelining replication: peer="{Voter server-c73d1d24-9212-86cf-e97d-353ee57be48f c73d1d24-9212-86cf-e97d-353ee57be48f}" 2024-06-11T14:40:04.112Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:04.006799295 +0000 UTC m=+10.589878247 (1 nodes), 83.657381ms from start of poll, 84.322197ms from cluster start. Timeout at 2024-06-11 14:40:04.112377686 +0000 UTC m=+10.695456638, 105.578391ms after stability 2024-06-11T14:40:04.118Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:04.223Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:04.118404588 +0000 UTC m=+10.701483540 (2 nodes), 1.76µs from start of poll, 195.92749ms from cluster start. Timeout at 2024-06-11 14:40:04.223685989 +0000 UTC m=+10.806764941, 105.281401ms after stability 2024-06-11T14:40:04.223Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: updating configuration: command=RemoveServer server-id=server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 server-addr= servers="[{Suffrage:Voter ID:server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a Address:0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a} {Suffrage:Voter ID:server-c73d1d24-9212-86cf-e97d-353ee57be48f Address:c73d1d24-9212-86cf-e97d-353ee57be48f}]" 2024-06-11T14:40:04.224Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: removed ourself, shutting down 2024-06-11T14:40:04.224Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: aborting pipeline replication: peer="{Voter server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a 0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a}" 2024-06-11T14:40:04.224Z [INFO] server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5: aborting pipeline replication: peer="{Voter server-c73d1d24-9212-86cf-e97d-353ee57be48f c73d1d24-9212-86cf-e97d-353ee57be48f}" 2024-06-11T14:40:04.284Z [WARN] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: heartbeat timeout reached, starting election: last-leader-addr=ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 last-leader-id=server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 2024-06-11T14:40:04.284Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: entering candidate state: node="Node at 0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a [Candidate]" term=3 2024-06-11T14:40:04.284Z [WARN] server-c73d1d24-9212-86cf-e97d-353ee57be48f: rejecting vote request since we have a leader: from=0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a leader=ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 leader-id=server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 2024-06-11T14:40:04.298Z [WARN] server-c73d1d24-9212-86cf-e97d-353ee57be48f: heartbeat timeout reached, starting election: last-leader-addr=ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 last-leader-id=server-ef7f00ef-c05b-8990-8edd-1c953bbf3bf5 2024-06-11T14:40:04.298Z [INFO] server-c73d1d24-9212-86cf-e97d-353ee57be48f: entering candidate state: node="Node at c73d1d24-9212-86cf-e97d-353ee57be48f [Candidate]" term=3 2024-06-11T14:40:04.298Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: duplicate requestVote for same term: term=3 2024-06-11T14:40:04.346Z [WARN] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: Election timeout reached, restarting election 2024-06-11T14:40:04.346Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: entering candidate state: node="Node at 0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a [Candidate]" term=4 2024-06-11T14:40:04.347Z [INFO] server-c73d1d24-9212-86cf-e97d-353ee57be48f: entering follower state: follower="Node at c73d1d24-9212-86cf-e97d-353ee57be48f [Follower]" leader-address= leader-id= 2024-06-11T14:40:04.347Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: election won: term=4 tally=2 2024-06-11T14:40:04.347Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: entering leader state: leader="Node at 0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a [Leader]" 2024-06-11T14:40:04.347Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: added peer, starting replication: peer=server-c73d1d24-9212-86cf-e97d-353ee57be48f 2024-06-11T14:40:04.347Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: pipelining replication: peer="{Voter server-c73d1d24-9212-86cf-e97d-353ee57be48f c73d1d24-9212-86cf-e97d-353ee57be48f}" 2024-06-11T14:40:04.435Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:04.541Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:04.435984609 +0000 UTC m=+11.019063561 (1 nodes), 87.957µs from start of poll, 513.507511ms from cluster start. Timeout at 2024-06-11 14:40:04.541099536 +0000 UTC m=+11.124178488, 105.114927ms after stability 2024-06-11T14:40:04.541Z [INFO] server-0838c0f1-9f31-bd5b-1ea7-9a582f0fca1a: aborting pipeline replication: peer="{Voter server-c73d1d24-9212-86cf-e97d-353ee57be48f c73d1d24-9212-86cf-e97d-353ee57be48f}" --- PASS: TestRaft_RemoveLeader (0.62s) === RUN TestRaft_RemoveLeader_NoShutdown 2024-06-11T14:40:04.547Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17 Address:e2c9762c-fc4e-46ff-5287-6d6bfd126d17} {Suffrage:Voter ID:server-c25cc0f7-224b-03d3-94d0-9bf10813a862 Address:c25cc0f7-224b-03d3-94d0-9bf10813a862} {Suffrage:Voter ID:server-4980e142-7f5d-1c09-f767-57f42771991e Address:4980e142-7f5d-1c09-f767-57f42771991e}]" 2024-06-11T14:40:04.547Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17 Address:e2c9762c-fc4e-46ff-5287-6d6bfd126d17} {Suffrage:Voter ID:server-c25cc0f7-224b-03d3-94d0-9bf10813a862 Address:c25cc0f7-224b-03d3-94d0-9bf10813a862} {Suffrage:Voter ID:server-4980e142-7f5d-1c09-f767-57f42771991e Address:4980e142-7f5d-1c09-f767-57f42771991e}]" 2024-06-11T14:40:04.547Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: entering follower state: follower="Node at e2c9762c-fc4e-46ff-5287-6d6bfd126d17 [Follower]" leader-address= leader-id= 2024-06-11T14:40:04.547Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: entering follower state: follower="Node at c25cc0f7-224b-03d3-94d0-9bf10813a862 [Follower]" leader-address= leader-id= 2024-06-11T14:40:04.547Z [INFO] server-4980e142-7f5d-1c09-f767-57f42771991e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17 Address:e2c9762c-fc4e-46ff-5287-6d6bfd126d17} {Suffrage:Voter ID:server-c25cc0f7-224b-03d3-94d0-9bf10813a862 Address:c25cc0f7-224b-03d3-94d0-9bf10813a862} {Suffrage:Voter ID:server-4980e142-7f5d-1c09-f767-57f42771991e Address:4980e142-7f5d-1c09-f767-57f42771991e}]" 2024-06-11T14:40:04.547Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:04.547Z [INFO] server-4980e142-7f5d-1c09-f767-57f42771991e: entering follower state: follower="Node at 4980e142-7f5d-1c09-f767-57f42771991e [Follower]" leader-address= leader-id= 2024-06-11T14:40:04.600Z [WARN] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:04.600Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: entering candidate state: node="Node at c25cc0f7-224b-03d3-94d0-9bf10813a862 [Candidate]" term=2 2024-06-11T14:40:04.600Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: election won: term=2 tally=2 2024-06-11T14:40:04.600Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: entering leader state: leader="Node at c25cc0f7-224b-03d3-94d0-9bf10813a862 [Leader]" 2024-06-11T14:40:04.600Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: added peer, starting replication: peer=server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17 2024-06-11T14:40:04.600Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: added peer, starting replication: peer=server-4980e142-7f5d-1c09-f767-57f42771991e 2024-06-11T14:40:04.600Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: pipelining replication: peer="{Voter server-4980e142-7f5d-1c09-f767-57f42771991e 4980e142-7f5d-1c09-f767-57f42771991e}" 2024-06-11T14:40:04.601Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: pipelining replication: peer="{Voter server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17 e2c9762c-fc4e-46ff-5287-6d6bfd126d17}" 2024-06-11T14:40:04.707Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:04.601238886 +0000 UTC m=+11.184317838 (2 nodes), 53.321516ms from start of poll, 54.019251ms from cluster start. Timeout at 2024-06-11 14:40:04.707031389 +0000 UTC m=+11.290110301, 105.792463ms after stability 2024-06-11T14:40:04.707Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:04.812Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:04.707170144 +0000 UTC m=+11.290249056 (1 nodes), 2.12µs from start of poll, 159.950469ms from cluster start. Timeout at 2024-06-11 14:40:04.812903689 +0000 UTC m=+11.395982601, 105.733545ms after stability 2024-06-11T14:40:04.820Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: updating configuration: command=RemoveServer server-id=server-c25cc0f7-224b-03d3-94d0-9bf10813a862 server-addr= servers="[{Suffrage:Voter ID:server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17 Address:e2c9762c-fc4e-46ff-5287-6d6bfd126d17} {Suffrage:Voter ID:server-4980e142-7f5d-1c09-f767-57f42771991e Address:4980e142-7f5d-1c09-f767-57f42771991e}]" 2024-06-11T14:40:04.820Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: removed ourself, transitioning to follower 2024-06-11T14:40:04.820Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: entering follower state: follower="Node at c25cc0f7-224b-03d3-94d0-9bf10813a862 [Follower]" leader-address= leader-id= 2024-06-11T14:40:04.820Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: aborting pipeline replication: peer="{Voter server-4980e142-7f5d-1c09-f767-57f42771991e 4980e142-7f5d-1c09-f767-57f42771991e}" 2024-06-11T14:40:04.820Z [INFO] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: aborting pipeline replication: peer="{Voter server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17 e2c9762c-fc4e-46ff-5287-6d6bfd126d17}" 2024-06-11T14:40:04.904Z [WARN] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: heartbeat timeout reached, starting election: last-leader-addr=c25cc0f7-224b-03d3-94d0-9bf10813a862 last-leader-id=server-c25cc0f7-224b-03d3-94d0-9bf10813a862 2024-06-11T14:40:04.904Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: entering candidate state: node="Node at e2c9762c-fc4e-46ff-5287-6d6bfd126d17 [Candidate]" term=3 2024-06-11T14:40:04.904Z [WARN] server-4980e142-7f5d-1c09-f767-57f42771991e: rejecting vote request since we have a leader: from=e2c9762c-fc4e-46ff-5287-6d6bfd126d17 leader=c25cc0f7-224b-03d3-94d0-9bf10813a862 leader-id=server-c25cc0f7-224b-03d3-94d0-9bf10813a862 2024-06-11T14:40:04.912Z [WARN] server-c25cc0f7-224b-03d3-94d0-9bf10813a862: not part of stable configuration, aborting election 2024-06-11T14:40:04.916Z [WARN] server-4980e142-7f5d-1c09-f767-57f42771991e: heartbeat timeout reached, starting election: last-leader-addr=c25cc0f7-224b-03d3-94d0-9bf10813a862 last-leader-id=server-c25cc0f7-224b-03d3-94d0-9bf10813a862 2024-06-11T14:40:04.916Z [INFO] server-4980e142-7f5d-1c09-f767-57f42771991e: entering candidate state: node="Node at 4980e142-7f5d-1c09-f767-57f42771991e [Candidate]" term=3 2024-06-11T14:40:04.916Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: duplicate requestVote for same term: term=3 2024-06-11T14:40:04.926Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:04.955Z [WARN] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: Election timeout reached, restarting election 2024-06-11T14:40:04.955Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: entering candidate state: node="Node at e2c9762c-fc4e-46ff-5287-6d6bfd126d17 [Candidate]" term=4 2024-06-11T14:40:04.955Z [INFO] server-4980e142-7f5d-1c09-f767-57f42771991e: entering follower state: follower="Node at 4980e142-7f5d-1c09-f767-57f42771991e [Follower]" leader-address= leader-id= 2024-06-11T14:40:04.955Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: election won: term=4 tally=2 2024-06-11T14:40:04.955Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: entering leader state: leader="Node at e2c9762c-fc4e-46ff-5287-6d6bfd126d17 [Leader]" 2024-06-11T14:40:04.955Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: added peer, starting replication: peer=server-4980e142-7f5d-1c09-f767-57f42771991e 2024-06-11T14:40:04.955Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: pipelining replication: peer="{Voter server-4980e142-7f5d-1c09-f767-57f42771991e 4980e142-7f5d-1c09-f767-57f42771991e}" 2024-06-11T14:40:05.061Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:04.955974806 +0000 UTC m=+11.539053758 (1 nodes), 29.073751ms from start of poll, 408.755171ms from cluster start. Timeout at 2024-06-11 14:40:05.061483439 +0000 UTC m=+11.644562351, 105.508593ms after stability 2024-06-11T14:40:05.167Z [INFO] server-e2c9762c-fc4e-46ff-5287-6d6bfd126d17: aborting pipeline replication: peer="{Voter server-4980e142-7f5d-1c09-f767-57f42771991e 4980e142-7f5d-1c09-f767-57f42771991e}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s) === RUN TestRaft_RemoveFollower_SplitCluster 2024-06-11T14:40:05.169Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a Address:74a5cb2f-f4dd-97f7-a533-b4877ac1706a} {Suffrage:Voter ID:server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 Address:f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8} {Suffrage:Voter ID:server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd Address:20338e0d-3242-e29e-da2d-d92cfa5cc0bd} {Suffrage:Voter ID:server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c Address:5f9af617-1f7d-5bce-fe2a-2c0d0935181c}]" 2024-06-11T14:40:05.170Z [INFO] server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a Address:74a5cb2f-f4dd-97f7-a533-b4877ac1706a} {Suffrage:Voter ID:server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 Address:f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8} {Suffrage:Voter ID:server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd Address:20338e0d-3242-e29e-da2d-d92cfa5cc0bd} {Suffrage:Voter ID:server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c Address:5f9af617-1f7d-5bce-fe2a-2c0d0935181c}]" 2024-06-11T14:40:05.170Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: entering follower state: follower="Node at 74a5cb2f-f4dd-97f7-a533-b4877ac1706a [Follower]" leader-address= leader-id= 2024-06-11T14:40:05.170Z [INFO] server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8: entering follower state: follower="Node at f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 [Follower]" leader-address= leader-id= 2024-06-11T14:40:05.170Z [INFO] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a Address:74a5cb2f-f4dd-97f7-a533-b4877ac1706a} {Suffrage:Voter ID:server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 Address:f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8} {Suffrage:Voter ID:server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd Address:20338e0d-3242-e29e-da2d-d92cfa5cc0bd} {Suffrage:Voter ID:server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c Address:5f9af617-1f7d-5bce-fe2a-2c0d0935181c}]" 2024-06-11T14:40:05.170Z [INFO] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: entering follower state: follower="Node at 20338e0d-3242-e29e-da2d-d92cfa5cc0bd [Follower]" leader-address= leader-id= 2024-06-11T14:40:05.170Z [INFO] server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a Address:74a5cb2f-f4dd-97f7-a533-b4877ac1706a} {Suffrage:Voter ID:server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 Address:f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8} {Suffrage:Voter ID:server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd Address:20338e0d-3242-e29e-da2d-d92cfa5cc0bd} {Suffrage:Voter ID:server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c Address:5f9af617-1f7d-5bce-fe2a-2c0d0935181c}]" 2024-06-11T14:40:05.170Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:05.170Z [INFO] server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c: entering follower state: follower="Node at 5f9af617-1f7d-5bce-fe2a-2c0d0935181c [Follower]" leader-address= leader-id= 2024-06-11T14:40:05.238Z [WARN] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:05.238Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: entering candidate state: node="Node at 74a5cb2f-f4dd-97f7-a533-b4877ac1706a [Candidate]" term=2 2024-06-11T14:40:05.238Z [WARN] server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:05.239Z [INFO] server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8: entering candidate state: node="Node at f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 [Candidate]" term=2 2024-06-11T14:40:05.239Z [INFO] server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c: duplicate requestVote for same term: term=2 2024-06-11T14:40:05.239Z [INFO] server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8: duplicate requestVote for same term: term=2 2024-06-11T14:40:05.239Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: election won: term=2 tally=3 2024-06-11T14:40:05.239Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: entering leader state: leader="Node at 74a5cb2f-f4dd-97f7-a533-b4877ac1706a [Leader]" 2024-06-11T14:40:05.239Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: added peer, starting replication: peer=server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 2024-06-11T14:40:05.239Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: added peer, starting replication: peer=server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd 2024-06-11T14:40:05.239Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: added peer, starting replication: peer=server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c 2024-06-11T14:40:05.239Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: pipelining replication: peer="{Voter server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c 5f9af617-1f7d-5bce-fe2a-2c0d0935181c}" 2024-06-11T14:40:05.239Z [WARN] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: rejecting vote request since we have a leader: from=f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 leader=74a5cb2f-f4dd-97f7-a533-b4877ac1706a leader-id=server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a 2024-06-11T14:40:05.239Z [INFO] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: duplicate requestVote for same term: term=2 2024-06-11T14:40:05.240Z [INFO] server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8: entering follower state: follower="Node at f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 [Follower]" leader-address=74a5cb2f-f4dd-97f7-a533-b4877ac1706a leader-id=server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a 2024-06-11T14:40:05.240Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: pipelining replication: peer="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" 2024-06-11T14:40:05.240Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: pipelining replication: peer="{Voter server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd 20338e0d-3242-e29e-da2d-d92cfa5cc0bd}" 2024-06-11T14:40:05.345Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:05.24021219 +0000 UTC m=+11.823291142 (1 nodes), 69.484373ms from start of poll, 70.471097ms from cluster start. Timeout at 2024-06-11 14:40:05.34552783 +0000 UTC m=+11.928606782, 105.31564ms after stability 2024-06-11T14:40:05.451Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:05.557Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:05.451597043 +0000 UTC m=+12.034675955 (3 nodes), 2.08µs from start of poll, 281.85591ms from cluster start. Timeout at 2024-06-11 14:40:05.557047718 +0000 UTC m=+12.140126670, 105.450715ms after stability 2024-06-11T14:40:05.557Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: updating configuration: command=RemoveServer server-id=server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c server-addr= servers="[{Suffrage:Voter ID:server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a Address:74a5cb2f-f4dd-97f7-a533-b4877ac1706a} {Suffrage:Voter ID:server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 Address:f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8} {Suffrage:Voter ID:server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd Address:20338e0d-3242-e29e-da2d-d92cfa5cc0bd}]" 2024-06-11T14:40:05.557Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: removed peer, stopping replication: peer=server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c last-index=3 2024-06-11T14:40:05.557Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: aborting pipeline replication: peer="{Voter server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c 5f9af617-1f7d-5bce-fe2a-2c0d0935181c}" 2024-06-11T14:40:05.557Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to pipeline appendEntries: peer="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" error="append pipeline closed" 2024-06-11T14:40:05.557Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: aborting pipeline replication: peer="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" 2024-06-11T14:40:05.557Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to pipeline appendEntries: peer="{Voter server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd 20338e0d-3242-e29e-da2d-d92cfa5cc0bd}" error="append pipeline closed" 2024-06-11T14:40:05.557Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: aborting pipeline replication: peer="{Voter server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd 20338e0d-3242-e29e-da2d-d92cfa5cc0bd}" 2024-06-11T14:40:05.557Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to heartbeat to: peer=20338e0d-3242-e29e-da2d-d92cfa5cc0bd backoff time=10ms error="failed to connect to peer: 20338e0d-3242-e29e-da2d-d92cfa5cc0bd" 2024-06-11T14:40:05.565Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to appendEntries to: peer="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" error="failed to connect to peer: f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8" 2024-06-11T14:40:05.567Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to heartbeat to: peer=f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 backoff time=10ms error="failed to connect to peer: f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8" 2024-06-11T14:40:05.567Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to appendEntries to: peer="{Voter server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd 20338e0d-3242-e29e-da2d-d92cfa5cc0bd}" error="failed to connect to peer: 20338e0d-3242-e29e-da2d-d92cfa5cc0bd" 2024-06-11T14:40:05.576Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to heartbeat to: peer=20338e0d-3242-e29e-da2d-d92cfa5cc0bd backoff time=10ms error="failed to connect to peer: 20338e0d-3242-e29e-da2d-d92cfa5cc0bd" 2024-06-11T14:40:05.581Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to appendEntries to: peer="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" error="failed to connect to peer: f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8" 2024-06-11T14:40:05.586Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to appendEntries to: peer="{Voter server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd 20338e0d-3242-e29e-da2d-d92cfa5cc0bd}" error="failed to connect to peer: 20338e0d-3242-e29e-da2d-d92cfa5cc0bd" 2024-06-11T14:40:05.586Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to heartbeat to: peer=f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 backoff time=10ms error="failed to connect to peer: f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8" 2024-06-11T14:40:05.593Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to heartbeat to: peer=20338e0d-3242-e29e-da2d-d92cfa5cc0bd backoff time=10ms error="failed to connect to peer: 20338e0d-3242-e29e-da2d-d92cfa5cc0bd" 2024-06-11T14:40:05.601Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to appendEntries to: peer="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" error="failed to connect to peer: f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8" 2024-06-11T14:40:05.603Z [WARN] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to contact: server-id=server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd time=50.478779ms 2024-06-11T14:40:05.604Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to heartbeat to: peer=f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 backoff time=10ms error="failed to connect to peer: f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8" 2024-06-11T14:40:05.605Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to appendEntries to: peer="{Voter server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd 20338e0d-3242-e29e-da2d-d92cfa5cc0bd}" error="failed to connect to peer: 20338e0d-3242-e29e-da2d-d92cfa5cc0bd" 2024-06-11T14:40:05.613Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to heartbeat to: peer=20338e0d-3242-e29e-da2d-d92cfa5cc0bd backoff time=20ms error="failed to connect to peer: 20338e0d-3242-e29e-da2d-d92cfa5cc0bd" 2024-06-11T14:40:05.613Z [WARN] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to contact: server-id=server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 time=56.039418ms 2024-06-11T14:40:05.613Z [WARN] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to contact: server-id=server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd time=60.585814ms 2024-06-11T14:40:05.613Z [WARN] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to contact quorum of nodes, stepping down 2024-06-11T14:40:05.613Z [INFO] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: entering follower state: follower="Node at 74a5cb2f-f4dd-97f7-a533-b4877ac1706a [Follower]" leader-address= leader-id= 2024-06-11T14:40:05.613Z [WARN] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: heartbeat timeout reached, starting election: last-leader-addr=74a5cb2f-f4dd-97f7-a533-b4877ac1706a last-leader-id=server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a 2024-06-11T14:40:05.613Z [INFO] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: entering candidate state: node="Node at 20338e0d-3242-e29e-da2d-d92cfa5cc0bd [Shutdown]" term=3 2024-06-11T14:40:05.613Z [ERROR] server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a: failed to appendEntries to: peer="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" error="failed to connect to peer: f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8" 2024-06-11T14:40:05.613Z [ERROR] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: failed to make requestVote RPC: target="{Voter server-5f9af617-1f7d-5bce-fe2a-2c0d0935181c 5f9af617-1f7d-5bce-fe2a-2c0d0935181c}" error="failed to connect to peer: 5f9af617-1f7d-5bce-fe2a-2c0d0935181c" term=3 2024-06-11T14:40:05.613Z [ERROR] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: failed to make requestVote RPC: target="{Voter server-74a5cb2f-f4dd-97f7-a533-b4877ac1706a 74a5cb2f-f4dd-97f7-a533-b4877ac1706a}" error="failed to connect to peer: 74a5cb2f-f4dd-97f7-a533-b4877ac1706a" term=3 2024-06-11T14:40:06.114Z [ERROR] server-20338e0d-3242-e29e-da2d-d92cfa5cc0bd: failed to make requestVote RPC: target="{Voter server-f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8 f6f9cae7-0cde-e95a-e6ab-c3241f1b93d8}" error="command timed out" term=3 --- PASS: TestRaft_RemoveFollower_SplitCluster (0.95s) === RUN TestRaft_AddKnownPeer 2024-06-11T14:40:06.116Z [INFO] server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 Address:9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4} {Suffrage:Voter ID:server-dd127375-e665-0c20-92c0-52bbfbec8add Address:dd127375-e665-0c20-92c0-52bbfbec8add} {Suffrage:Voter ID:server-a3ffed20-747d-f9ec-a596-0a0114f04366 Address:a3ffed20-747d-f9ec-a596-0a0114f04366}]" 2024-06-11T14:40:06.116Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 Address:9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4} {Suffrage:Voter ID:server-dd127375-e665-0c20-92c0-52bbfbec8add Address:dd127375-e665-0c20-92c0-52bbfbec8add} {Suffrage:Voter ID:server-a3ffed20-747d-f9ec-a596-0a0114f04366 Address:a3ffed20-747d-f9ec-a596-0a0114f04366}]" 2024-06-11T14:40:06.117Z [INFO] server-a3ffed20-747d-f9ec-a596-0a0114f04366: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 Address:9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4} {Suffrage:Voter ID:server-dd127375-e665-0c20-92c0-52bbfbec8add Address:dd127375-e665-0c20-92c0-52bbfbec8add} {Suffrage:Voter ID:server-a3ffed20-747d-f9ec-a596-0a0114f04366 Address:a3ffed20-747d-f9ec-a596-0a0114f04366}]" 2024-06-11T14:40:06.117Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:06.117Z [INFO] server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4: entering follower state: follower="Node at 9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.117Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: entering follower state: follower="Node at dd127375-e665-0c20-92c0-52bbfbec8add [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.117Z [INFO] server-a3ffed20-747d-f9ec-a596-0a0114f04366: entering follower state: follower="Node at a3ffed20-747d-f9ec-a596-0a0114f04366 [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.169Z [WARN] server-dd127375-e665-0c20-92c0-52bbfbec8add: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:06.170Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: entering candidate state: node="Node at dd127375-e665-0c20-92c0-52bbfbec8add [Candidate]" term=2 2024-06-11T14:40:06.170Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: election won: term=2 tally=2 2024-06-11T14:40:06.170Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: entering leader state: leader="Node at dd127375-e665-0c20-92c0-52bbfbec8add [Leader]" 2024-06-11T14:40:06.170Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: added peer, starting replication: peer=server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 2024-06-11T14:40:06.170Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: added peer, starting replication: peer=server-a3ffed20-747d-f9ec-a596-0a0114f04366 2024-06-11T14:40:06.170Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: pipelining replication: peer="{Voter server-a3ffed20-747d-f9ec-a596-0a0114f04366 a3ffed20-747d-f9ec-a596-0a0114f04366}" 2024-06-11T14:40:06.170Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: pipelining replication: peer="{Voter server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4}" 2024-06-11T14:40:06.276Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:06.170935127 +0000 UTC m=+12.754014079 (1 nodes), 53.689943ms from start of poll, 54.399797ms from cluster start. Timeout at 2024-06-11 14:40:06.27617345 +0000 UTC m=+12.859252362, 105.238283ms after stability 2024-06-11T14:40:06.276Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:06.382Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:06.276320884 +0000 UTC m=+12.859399836 (2 nodes), 1.6µs from start of poll, 159.785554ms from cluster start. Timeout at 2024-06-11 14:40:06.382251422 +0000 UTC m=+12.965330374, 105.930538ms after stability 2024-06-11T14:40:06.382Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: updating configuration: command=AddVoter server-id=server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 server-addr=9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 servers="[{Suffrage:Voter ID:server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 Address:9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4} {Suffrage:Voter ID:server-dd127375-e665-0c20-92c0-52bbfbec8add Address:dd127375-e665-0c20-92c0-52bbfbec8add} {Suffrage:Voter ID:server-a3ffed20-747d-f9ec-a596-0a0114f04366 Address:a3ffed20-747d-f9ec-a596-0a0114f04366}]" 2024-06-11T14:40:06.383Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: aborting pipeline replication: peer="{Voter server-9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4 9ced21c1-ca3d-6190-6abf-9ac22c9c7aa4}" 2024-06-11T14:40:06.383Z [INFO] server-dd127375-e665-0c20-92c0-52bbfbec8add: aborting pipeline replication: peer="{Voter server-a3ffed20-747d-f9ec-a596-0a0114f04366 a3ffed20-747d-f9ec-a596-0a0114f04366}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer 2024-06-11T14:40:06.385Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2 Address:f682f44b-013a-cce6-5bbd-9eb215b4a4f2} {Suffrage:Voter ID:server-b94dbc13-bf25-e097-b812-70d9f350db57 Address:b94dbc13-bf25-e097-b812-70d9f350db57} {Suffrage:Voter ID:server-1c641893-2e66-c5a3-9383-eff7a54c2e1b Address:1c641893-2e66-c5a3-9383-eff7a54c2e1b}]" 2024-06-11T14:40:06.385Z [INFO] server-b94dbc13-bf25-e097-b812-70d9f350db57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2 Address:f682f44b-013a-cce6-5bbd-9eb215b4a4f2} {Suffrage:Voter ID:server-b94dbc13-bf25-e097-b812-70d9f350db57 Address:b94dbc13-bf25-e097-b812-70d9f350db57} {Suffrage:Voter ID:server-1c641893-2e66-c5a3-9383-eff7a54c2e1b Address:1c641893-2e66-c5a3-9383-eff7a54c2e1b}]" 2024-06-11T14:40:06.385Z [INFO] server-1c641893-2e66-c5a3-9383-eff7a54c2e1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2 Address:f682f44b-013a-cce6-5bbd-9eb215b4a4f2} {Suffrage:Voter ID:server-b94dbc13-bf25-e097-b812-70d9f350db57 Address:b94dbc13-bf25-e097-b812-70d9f350db57} {Suffrage:Voter ID:server-1c641893-2e66-c5a3-9383-eff7a54c2e1b Address:1c641893-2e66-c5a3-9383-eff7a54c2e1b}]" 2024-06-11T14:40:06.385Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:06.385Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: entering follower state: follower="Node at f682f44b-013a-cce6-5bbd-9eb215b4a4f2 [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.385Z [INFO] server-b94dbc13-bf25-e097-b812-70d9f350db57: entering follower state: follower="Node at b94dbc13-bf25-e097-b812-70d9f350db57 [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.385Z [INFO] server-1c641893-2e66-c5a3-9383-eff7a54c2e1b: entering follower state: follower="Node at 1c641893-2e66-c5a3-9383-eff7a54c2e1b [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.444Z [WARN] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:06.444Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: entering candidate state: node="Node at f682f44b-013a-cce6-5bbd-9eb215b4a4f2 [Candidate]" term=2 2024-06-11T14:40:06.444Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: election won: term=2 tally=2 2024-06-11T14:40:06.444Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: entering leader state: leader="Node at f682f44b-013a-cce6-5bbd-9eb215b4a4f2 [Leader]" 2024-06-11T14:40:06.444Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: added peer, starting replication: peer=server-b94dbc13-bf25-e097-b812-70d9f350db57 2024-06-11T14:40:06.444Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: added peer, starting replication: peer=server-1c641893-2e66-c5a3-9383-eff7a54c2e1b 2024-06-11T14:40:06.444Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: pipelining replication: peer="{Voter server-1c641893-2e66-c5a3-9383-eff7a54c2e1b 1c641893-2e66-c5a3-9383-eff7a54c2e1b}" 2024-06-11T14:40:06.444Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: pipelining replication: peer="{Voter server-b94dbc13-bf25-e097-b812-70d9f350db57 b94dbc13-bf25-e097-b812-70d9f350db57}" 2024-06-11T14:40:06.550Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:06.445058916 +0000 UTC m=+13.028137828 (1 nodes), 59.378098ms from start of poll, 60.089272ms from cluster start. Timeout at 2024-06-11 14:40:06.550757782 +0000 UTC m=+13.133836694, 105.698866ms after stability 2024-06-11T14:40:06.550Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: updating configuration: command=RemoveServer server-id=a488ab5e-6eec-3fd8-6afa-7cd4f264b485 server-addr= servers="[{Suffrage:Voter ID:server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2 Address:f682f44b-013a-cce6-5bbd-9eb215b4a4f2} {Suffrage:Voter ID:server-b94dbc13-bf25-e097-b812-70d9f350db57 Address:b94dbc13-bf25-e097-b812-70d9f350db57} {Suffrage:Voter ID:server-1c641893-2e66-c5a3-9383-eff7a54c2e1b Address:1c641893-2e66-c5a3-9383-eff7a54c2e1b}]" 2024-06-11T14:40:06.551Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: aborting pipeline replication: peer="{Voter server-1c641893-2e66-c5a3-9383-eff7a54c2e1b 1c641893-2e66-c5a3-9383-eff7a54c2e1b}" 2024-06-11T14:40:06.551Z [INFO] server-f682f44b-013a-cce6-5bbd-9eb215b4a4f2: aborting pipeline replication: peer="{Voter server-b94dbc13-bf25-e097-b812-70d9f350db57 b94dbc13-bf25-e097-b812-70d9f350db57}" --- PASS: TestRaft_RemoveUnknownPeer (0.17s) === RUN TestRaft_SnapshotRestore 2024-06-11T14:40:06.553Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cc6e111-c41c-b0be-7a35-99fc4becbf41 Address:4cc6e111-c41c-b0be-7a35-99fc4becbf41}]" 2024-06-11T14:40:06.553Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:06.553Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: entering follower state: follower="Node at 4cc6e111-c41c-b0be-7a35-99fc4becbf41 [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.610Z [WARN] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:06.610Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: entering candidate state: node="Node at 4cc6e111-c41c-b0be-7a35-99fc4becbf41 [Candidate]" term=2 2024-06-11T14:40:06.610Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: election won: term=2 tally=1 2024-06-11T14:40:06.610Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: entering leader state: leader="Node at 4cc6e111-c41c-b0be-7a35-99fc4becbf41 [Leader]" 2024-06-11T14:40:06.716Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:06.610842134 +0000 UTC m=+13.193921086 (1 nodes), 57.403929ms from start of poll, 57.799675ms from cluster start. Timeout at 2024-06-11 14:40:06.716039418 +0000 UTC m=+13.299118330, 105.197244ms after stability 2024-06-11T14:40:06.721Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: starting snapshot up to: index=102 2024-06-11T14:40:06.721Z [INFO] creating new snapshot: path=/tmp/raft3345761062/snapshots/2-102-1718116806721.tmp 2024-06-11T14:40:06.722Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: compacting logs: from=1 to=92 2024-06-11T14:40:06.722Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: snapshot complete up to: index=102 2024-06-11T14:40:06.722Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: starting restore from snapshot: id=2-102-1718116806721 last-index=102 last-term=2 size-in-bytes=693 2024-06-11T14:40:06.723Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: snapshot restore progress: id=2-102-1718116806721 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-11T14:40:06.723Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: restored from snapshot: id=2-102-1718116806721 last-index=102 last-term=2 size-in-bytes=693 2024-06-11T14:40:06.723Z [INFO] server-4cc6e111-c41c-b0be-7a35-99fc4becbf41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cc6e111-c41c-b0be-7a35-99fc4becbf41 Address:4cc6e111-c41c-b0be-7a35-99fc4becbf41}]" --- PASS: TestRaft_SnapshotRestore (0.17s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2024-06-11T14:40:06.726Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60cf1b80-311d-e3cf-5afa-12127b084359 Address:60cf1b80-311d-e3cf-5afa-12127b084359}]" 2024-06-11T14:40:06.726Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:06.726Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: entering follower state: follower="Node at 60cf1b80-311d-e3cf-5afa-12127b084359 [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.782Z [WARN] server-60cf1b80-311d-e3cf-5afa-12127b084359: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:06.782Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: entering candidate state: node="Node at 60cf1b80-311d-e3cf-5afa-12127b084359 [Candidate]" term=2 2024-06-11T14:40:06.782Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: election won: term=2 tally=1 2024-06-11T14:40:06.782Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: entering leader state: leader="Node at 60cf1b80-311d-e3cf-5afa-12127b084359 [Leader]" 2024-06-11T14:40:06.887Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:06.782375344 +0000 UTC m=+13.365454296 (1 nodes), 55.96554ms from start of poll, 56.25609ms from cluster start. Timeout at 2024-06-11 14:40:06.887791421 +0000 UTC m=+13.470870373, 105.416077ms after stability 2024-06-11T14:40:06.893Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: starting snapshot up to: index=102 2024-06-11T14:40:06.893Z [INFO] creating new snapshot: path=/tmp/raft4174435870/snapshots/2-102-1718116806893.tmp 2024-06-11T14:40:06.894Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: compacting logs: from=1 to=92 2024-06-11T14:40:06.894Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: snapshot complete up to: index=102 2024-06-11T14:40:06.895Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: starting restore from snapshot: id=2-102-1718116806893 last-index=102 last-term=2 size-in-bytes=693 2024-06-11T14:40:06.895Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: snapshot restore progress: id=2-102-1718116806893 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-11T14:40:06.895Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: restored from snapshot: id=2-102-1718116806893 last-index=102 last-term=2 size-in-bytes=693 2024-06-11T14:40:06.895Z [INFO] server-60cf1b80-311d-e3cf-5afa-12127b084359: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60cf1b80-311d-e3cf-5afa-12127b084359 Address:60cf1b80-311d-e3cf-5afa-12127b084359}]" --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.17s) === RUN TestRaft_SnapshotRestore_Progress 2024-06-11T14:40:06.896Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf056052-d28a-2d93-2a30-7156b313fbd0 Address:bf056052-d28a-2d93-2a30-7156b313fbd0}]" 2024-06-11T14:40:06.896Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:06.896Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: entering follower state: follower="Node at bf056052-d28a-2d93-2a30-7156b313fbd0 [Follower]" leader-address= leader-id= 2024-06-11T14:40:06.953Z [WARN] server-bf056052-d28a-2d93-2a30-7156b313fbd0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:06.954Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: entering candidate state: node="Node at bf056052-d28a-2d93-2a30-7156b313fbd0 [Candidate]" term=2 2024-06-11T14:40:06.954Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: election won: term=2 tally=1 2024-06-11T14:40:06.954Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: entering leader state: leader="Node at bf056052-d28a-2d93-2a30-7156b313fbd0 [Leader]" 2024-06-11T14:40:07.059Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:06.954218584 +0000 UTC m=+13.537297496 (1 nodes), 57.488286ms from start of poll, 57.691319ms from cluster start. Timeout at 2024-06-11 14:40:07.059799614 +0000 UTC m=+13.642878526, 105.58103ms after stability 2024-06-11T14:40:07.064Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: starting snapshot up to: index=102 2024-06-11T14:40:07.064Z [INFO] creating new snapshot: path=/tmp/raft4008813457/snapshots/2-102-1718116807064.tmp 2024-06-11T14:40:07.065Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: compacting logs: from=1 to=92 2024-06-11T14:40:07.065Z [INFO] server-bf056052-d28a-2d93-2a30-7156b313fbd0: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.17s) === RUN TestRaft_NoRestoreOnStart 2024-06-11T14:40:07.068Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36cd7489-c701-6c59-d479-fc1b63b6faff Address:36cd7489-c701-6c59-d479-fc1b63b6faff}]" 2024-06-11T14:40:07.068Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:07.068Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: entering follower state: follower="Node at 36cd7489-c701-6c59-d479-fc1b63b6faff [Follower]" leader-address= leader-id= 2024-06-11T14:40:07.139Z [WARN] server-36cd7489-c701-6c59-d479-fc1b63b6faff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:07.139Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: entering candidate state: node="Node at 36cd7489-c701-6c59-d479-fc1b63b6faff [Candidate]" term=2 2024-06-11T14:40:07.140Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: election won: term=2 tally=1 2024-06-11T14:40:07.140Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: entering leader state: leader="Node at 36cd7489-c701-6c59-d479-fc1b63b6faff [Leader]" 2024-06-11T14:40:07.245Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:07.140176954 +0000 UTC m=+13.723255906 (1 nodes), 71.78465ms from start of poll, 72.026201ms from cluster start. Timeout at 2024-06-11 14:40:07.245778863 +0000 UTC m=+13.828857815, 105.601909ms after stability 2024-06-11T14:40:07.250Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: starting snapshot up to: index=102 2024-06-11T14:40:07.251Z [INFO] creating new snapshot: path=/tmp/raft2101308665/snapshots/2-102-1718116807250.tmp 2024-06-11T14:40:07.252Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: compacting logs: from=1 to=92 2024-06-11T14:40:07.252Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: snapshot complete up to: index=102 2024-06-11T14:40:07.252Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36cd7489-c701-6c59-d479-fc1b63b6faff Address:36cd7489-c701-6c59-d479-fc1b63b6faff}]" --- PASS: TestRaft_NoRestoreOnStart (0.18s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2024-06-11T14:40:07.253Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90b4fc87-3a7c-1ba2-5672-a5571481050a Address:90b4fc87-3a7c-1ba2-5672-a5571481050a}]" 2024-06-11T14:40:07.253Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:07.253Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: entering follower state: follower="Node at 36cd7489-c701-6c59-d479-fc1b63b6faff [Follower]" leader-address= leader-id= 2024-06-11T14:40:07.253Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: entering follower state: follower="Node at 90b4fc87-3a7c-1ba2-5672-a5571481050a [Follower]" leader-address= leader-id= 2024-06-11T14:40:07.322Z [WARN] server-36cd7489-c701-6c59-d479-fc1b63b6faff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:07.322Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: entering candidate state: node="Node at 36cd7489-c701-6c59-d479-fc1b63b6faff [Candidate]" term=3 2024-06-11T14:40:07.322Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: election won: term=3 tally=1 2024-06-11T14:40:07.322Z [INFO] server-36cd7489-c701-6c59-d479-fc1b63b6faff: entering leader state: leader="Node at 36cd7489-c701-6c59-d479-fc1b63b6faff [Leader]" 2024-06-11T14:40:07.329Z [WARN] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:07.329Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: entering candidate state: node="Node at 90b4fc87-3a7c-1ba2-5672-a5571481050a [Candidate]" term=2 2024-06-11T14:40:07.329Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: election won: term=2 tally=1 2024-06-11T14:40:07.329Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: entering leader state: leader="Node at 90b4fc87-3a7c-1ba2-5672-a5571481050a [Leader]" 2024-06-11T14:40:07.435Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:07.329832431 +0000 UTC m=+13.912911383 (1 nodes), 76.565678ms from start of poll, 76.794029ms from cluster start. Timeout at 2024-06-11 14:40:07.435267746 +0000 UTC m=+14.018346698, 105.435315ms after stability 2024-06-11T14:40:07.438Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: starting snapshot up to: index=58 2024-06-11T14:40:07.438Z [INFO] creating new snapshot: path=/tmp/raft2064945944/snapshots/2-58-1718116807438.tmp 2024-06-11T14:40:07.439Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: compacting logs: from=1 to=49 2024-06-11T14:40:07.439Z [INFO] server-90b4fc87-3a7c-1ba2-5672-a5571481050a: snapshot complete up to: index=58 --- PASS: TestRaft_AutoSnapshot (0.30s) === RUN TestRaft_UserSnapshot 2024-06-11T14:40:07.551Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5838ecac-7d4a-5767-9da8-bfb941cb809e Address:5838ecac-7d4a-5767-9da8-bfb941cb809e}]" 2024-06-11T14:40:07.551Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:07.551Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: entering follower state: follower="Node at 5838ecac-7d4a-5767-9da8-bfb941cb809e [Follower]" leader-address= leader-id= 2024-06-11T14:40:07.619Z [WARN] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:07.619Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: entering candidate state: node="Node at 5838ecac-7d4a-5767-9da8-bfb941cb809e [Candidate]" term=2 2024-06-11T14:40:07.619Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: election won: term=2 tally=1 2024-06-11T14:40:07.619Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: entering leader state: leader="Node at 5838ecac-7d4a-5767-9da8-bfb941cb809e [Leader]" 2024-06-11T14:40:07.725Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:07.619635734 +0000 UTC m=+14.202714686 (1 nodes), 68.476969ms from start of poll, 68.796718ms from cluster start. Timeout at 2024-06-11 14:40:07.725703906 +0000 UTC m=+14.308782858, 106.068172ms after stability 2024-06-11T14:40:07.725Z [ERROR] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: failed to take snapshot: error="nothing new to snapshot" 2024-06-11T14:40:07.726Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: starting snapshot up to: index=12 2024-06-11T14:40:07.726Z [INFO] creating new snapshot: path=/tmp/raft2990397408/snapshots/2-12-1718116807726.tmp 2024-06-11T14:40:07.727Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: compacting logs: from=1 to=2 2024-06-11T14:40:07.728Z [INFO] server-5838ecac-7d4a-5767-9da8-bfb941cb809e: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.18s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2024-06-11T14:40:07.730Z [INFO] server-9ca4de2f-c510-09c7-3108-987f870af915: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ca4de2f-c510-09c7-3108-987f870af915 Address:9ca4de2f-c510-09c7-3108-987f870af915} {Suffrage:Voter ID:server-0f966915-d942-127a-f9be-864682c8c679 Address:0f966915-d942-127a-f9be-864682c8c679} {Suffrage:Voter ID:server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95 Address:f3ffd050-0129-0dc2-98ca-a83b9c3b5d95}]" 2024-06-11T14:40:07.730Z [INFO] server-9ca4de2f-c510-09c7-3108-987f870af915: entering follower state: follower="Node at 9ca4de2f-c510-09c7-3108-987f870af915 [Follower]" leader-address= leader-id= 2024-06-11T14:40:07.730Z [INFO] server-0f966915-d942-127a-f9be-864682c8c679: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ca4de2f-c510-09c7-3108-987f870af915 Address:9ca4de2f-c510-09c7-3108-987f870af915} {Suffrage:Voter ID:server-0f966915-d942-127a-f9be-864682c8c679 Address:0f966915-d942-127a-f9be-864682c8c679} {Suffrage:Voter ID:server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95 Address:f3ffd050-0129-0dc2-98ca-a83b9c3b5d95}]" 2024-06-11T14:40:07.730Z [INFO] server-0f966915-d942-127a-f9be-864682c8c679: entering follower state: follower="Node at 0f966915-d942-127a-f9be-864682c8c679 [Follower]" leader-address= leader-id= 2024-06-11T14:40:07.730Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ca4de2f-c510-09c7-3108-987f870af915 Address:9ca4de2f-c510-09c7-3108-987f870af915} {Suffrage:Voter ID:server-0f966915-d942-127a-f9be-864682c8c679 Address:0f966915-d942-127a-f9be-864682c8c679} {Suffrage:Voter ID:server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95 Address:f3ffd050-0129-0dc2-98ca-a83b9c3b5d95}]" 2024-06-11T14:40:07.730Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:07.730Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: entering follower state: follower="Node at f3ffd050-0129-0dc2-98ca-a83b9c3b5d95 [Follower]" leader-address= leader-id= 2024-06-11T14:40:08.278Z [WARN] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:08.278Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: entering candidate state: node="Node at f3ffd050-0129-0dc2-98ca-a83b9c3b5d95 [Candidate]" term=2 2024-06-11T14:40:08.279Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: election won: term=2 tally=2 2024-06-11T14:40:08.279Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: entering leader state: leader="Node at f3ffd050-0129-0dc2-98ca-a83b9c3b5d95 [Leader]" 2024-06-11T14:40:08.279Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: added peer, starting replication: peer=server-9ca4de2f-c510-09c7-3108-987f870af915 2024-06-11T14:40:08.279Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: added peer, starting replication: peer=server-0f966915-d942-127a-f9be-864682c8c679 2024-06-11T14:40:08.279Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: pipelining replication: peer="{Voter server-0f966915-d942-127a-f9be-864682c8c679 0f966915-d942-127a-f9be-864682c8c679}" 2024-06-11T14:40:08.279Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: pipelining replication: peer="{Voter server-9ca4de2f-c510-09c7-3108-987f870af915 9ca4de2f-c510-09c7-3108-987f870af915}" 2024-06-11T14:40:09.284Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:08.279706556 +0000 UTC m=+14.862785468 (1 nodes), 548.921193ms from start of poll, 549.785122ms from cluster start. Timeout at 2024-06-11 14:40:09.284875927 +0000 UTC m=+15.867954839, 1.005169371s after stability 2024-06-11T14:40:09.286Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: starting snapshot up to: index=12 2024-06-11T14:40:09.286Z [INFO] creating new snapshot: path=/tmp/raft701144930/snapshots/2-12-1718116809286.tmp 2024-06-11T14:40:09.287Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: snapshot complete up to: index=12 2024-06-11T14:40:09.289Z [INFO] creating new snapshot: path=/tmp/raft701144930/snapshots/2-23-1718116809289.tmp 2024-06-11T14:40:09.291Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: copied to local snapshot: bytes=71 2024-06-11T14:40:09.291Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: snapshot restore progress: id=2-23-1718116809289 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:09.291Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: restored user snapshot: index=23 2024-06-11T14:40:09.291Z [ERROR] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: failed to get log: index=23 error="log not found" 2024-06-11T14:40:09.291Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: aborting pipeline replication: peer="{Voter server-0f966915-d942-127a-f9be-864682c8c679 0f966915-d942-127a-f9be-864682c8c679}" 2024-06-11T14:40:09.291Z [ERROR] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: failed to get log: index=23 error="log not found" 2024-06-11T14:40:09.291Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: aborting pipeline replication: peer="{Voter server-9ca4de2f-c510-09c7-3108-987f870af915 9ca4de2f-c510-09c7-3108-987f870af915}" 2024-06-11T14:40:09.299Z [ERROR] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: failed to get log: index=23 error="log not found" 2024-06-11T14:40:09.300Z [INFO] creating new snapshot: path=/tmp/raft2784752993/snapshots/2-23-1718116809300.tmp 2024-06-11T14:40:09.300Z [INFO] server-0f966915-d942-127a-f9be-864682c8c679: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:09.301Z [INFO] server-0f966915-d942-127a-f9be-864682c8c679: copied to local snapshot: bytes=71 2024-06-11T14:40:09.301Z [INFO] server-0f966915-d942-127a-f9be-864682c8c679: snapshot restore progress: id=2-23-1718116809300 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:09.301Z [ERROR] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: failed to get log: index=23 error="log not found" 2024-06-11T14:40:09.302Z [INFO] creating new snapshot: path=/tmp/raft3219025380/snapshots/2-23-1718116809302.tmp 2024-06-11T14:40:09.302Z [INFO] server-0f966915-d942-127a-f9be-864682c8c679: Installed remote snapshot 2024-06-11T14:40:09.302Z [WARN] server-0f966915-d942-127a-f9be-864682c8c679: skipping application of old log: index=22 2024-06-11T14:40:09.302Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: pipelining replication: peer="{Voter server-0f966915-d942-127a-f9be-864682c8c679 0f966915-d942-127a-f9be-864682c8c679}" 2024-06-11T14:40:09.302Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:09.303Z [INFO] server-9ca4de2f-c510-09c7-3108-987f870af915: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:09.304Z [INFO] server-9ca4de2f-c510-09c7-3108-987f870af915: copied to local snapshot: bytes=71 2024-06-11T14:40:09.306Z [INFO] server-9ca4de2f-c510-09c7-3108-987f870af915: snapshot restore progress: id=2-23-1718116809302 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:09.306Z [INFO] server-9ca4de2f-c510-09c7-3108-987f870af915: Installed remote snapshot 2024-06-11T14:40:09.307Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: pipelining replication: peer="{Voter server-9ca4de2f-c510-09c7-3108-987f870af915 9ca4de2f-c510-09c7-3108-987f870af915}" 2024-06-11T14:40:10.307Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:09.302616367 +0000 UTC m=+15.885695279 (2 nodes), 1.28µs from start of poll, 1.572694933s from cluster start. Timeout at 2024-06-11 14:40:10.30770954 +0000 UTC m=+16.890788492, 1.005093213s after stability 2024-06-11T14:40:10.320Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: aborting pipeline replication: peer="{Voter server-0f966915-d942-127a-f9be-864682c8c679 0f966915-d942-127a-f9be-864682c8c679}" 2024-06-11T14:40:10.321Z [INFO] server-f3ffd050-0129-0dc2-98ca-a83b9c3b5d95: aborting pipeline replication: peer="{Voter server-9ca4de2f-c510-09c7-3108-987f870af915 9ca4de2f-c510-09c7-3108-987f870af915}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2024-06-11T14:40:10.323Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be4ddad8-5723-266d-47ef-8da1ff18de84 Address:be4ddad8-5723-266d-47ef-8da1ff18de84} {Suffrage:Voter ID:server-368de480-8bd8-a228-8bfd-6efab6088d64 Address:368de480-8bd8-a228-8bfd-6efab6088d64} {Suffrage:Voter ID:server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6 Address:5bc8756a-e78c-d97b-8acc-ba3d28d23ea6}]" 2024-06-11T14:40:10.323Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be4ddad8-5723-266d-47ef-8da1ff18de84 Address:be4ddad8-5723-266d-47ef-8da1ff18de84} {Suffrage:Voter ID:server-368de480-8bd8-a228-8bfd-6efab6088d64 Address:368de480-8bd8-a228-8bfd-6efab6088d64} {Suffrage:Voter ID:server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6 Address:5bc8756a-e78c-d97b-8acc-ba3d28d23ea6}]" 2024-06-11T14:40:10.324Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be4ddad8-5723-266d-47ef-8da1ff18de84 Address:be4ddad8-5723-266d-47ef-8da1ff18de84} {Suffrage:Voter ID:server-368de480-8bd8-a228-8bfd-6efab6088d64 Address:368de480-8bd8-a228-8bfd-6efab6088d64} {Suffrage:Voter ID:server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6 Address:5bc8756a-e78c-d97b-8acc-ba3d28d23ea6}]" 2024-06-11T14:40:10.324Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:10.324Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: entering follower state: follower="Node at be4ddad8-5723-266d-47ef-8da1ff18de84 [Follower]" leader-address= leader-id= 2024-06-11T14:40:10.324Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: entering follower state: follower="Node at 368de480-8bd8-a228-8bfd-6efab6088d64 [Follower]" leader-address= leader-id= 2024-06-11T14:40:10.324Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: entering follower state: follower="Node at 5bc8756a-e78c-d97b-8acc-ba3d28d23ea6 [Follower]" leader-address= leader-id= 2024-06-11T14:40:10.850Z [WARN] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:10.850Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: entering candidate state: node="Node at 5bc8756a-e78c-d97b-8acc-ba3d28d23ea6 [Candidate]" term=2 2024-06-11T14:40:10.850Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: election won: term=2 tally=2 2024-06-11T14:40:10.850Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: entering leader state: leader="Node at 5bc8756a-e78c-d97b-8acc-ba3d28d23ea6 [Leader]" 2024-06-11T14:40:10.850Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: added peer, starting replication: peer=server-be4ddad8-5723-266d-47ef-8da1ff18de84 2024-06-11T14:40:10.850Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: added peer, starting replication: peer=server-368de480-8bd8-a228-8bfd-6efab6088d64 2024-06-11T14:40:10.851Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: pipelining replication: peer="{Voter server-368de480-8bd8-a228-8bfd-6efab6088d64 368de480-8bd8-a228-8bfd-6efab6088d64}" 2024-06-11T14:40:10.851Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: pipelining replication: peer="{Voter server-be4ddad8-5723-266d-47ef-8da1ff18de84 be4ddad8-5723-266d-47ef-8da1ff18de84}" 2024-06-11T14:40:11.857Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:10.851432841 +0000 UTC m=+17.434511793 (1 nodes), 527.246775ms from start of poll, 527.813755ms from cluster start. Timeout at 2024-06-11 14:40:11.856966879 +0000 UTC m=+18.440045791, 1.005533998s after stability 2024-06-11T14:40:11.858Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: starting snapshot up to: index=12 2024-06-11T14:40:11.858Z [INFO] creating new snapshot: path=/tmp/raft1195170909/snapshots/2-12-1718116811858.tmp 2024-06-11T14:40:11.859Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: snapshot complete up to: index=12 2024-06-11T14:40:11.861Z [INFO] creating new snapshot: path=/tmp/raft1195170909/snapshots/2-23-1718116811861.tmp 2024-06-11T14:40:11.863Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: copied to local snapshot: bytes=71 2024-06-11T14:40:11.863Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: snapshot restore progress: id=2-23-1718116811861 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:11.863Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: removing all old logs from log store 2024-06-11T14:40:11.863Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: compacting logs: from=1 to=22 2024-06-11T14:40:11.863Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: restored user snapshot: index=23 2024-06-11T14:40:11.863Z [ERROR] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: failed to get log: index=22 error="log not found" 2024-06-11T14:40:11.863Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: aborting pipeline replication: peer="{Voter server-368de480-8bd8-a228-8bfd-6efab6088d64 368de480-8bd8-a228-8bfd-6efab6088d64}" 2024-06-11T14:40:11.863Z [ERROR] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: failed to get log: index=22 error="log not found" 2024-06-11T14:40:11.863Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: aborting pipeline replication: peer="{Voter server-be4ddad8-5723-266d-47ef-8da1ff18de84 be4ddad8-5723-266d-47ef-8da1ff18de84}" 2024-06-11T14:40:11.869Z [ERROR] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: failed to get log: index=22 error="log not found" 2024-06-11T14:40:11.870Z [INFO] creating new snapshot: path=/tmp/raft1843089783/snapshots/2-23-1718116811869.tmp 2024-06-11T14:40:11.870Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:11.871Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: copied to local snapshot: bytes=71 2024-06-11T14:40:11.871Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: snapshot restore progress: id=2-23-1718116811869 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:11.871Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: removing all old logs from log store 2024-06-11T14:40:11.871Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: compacting logs: from=1 to=22 2024-06-11T14:40:11.871Z [INFO] server-be4ddad8-5723-266d-47ef-8da1ff18de84: Installed remote snapshot 2024-06-11T14:40:11.871Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: pipelining replication: peer="{Voter server-be4ddad8-5723-266d-47ef-8da1ff18de84 be4ddad8-5723-266d-47ef-8da1ff18de84}" 2024-06-11T14:40:11.871Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:11.874Z [ERROR] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: failed to get log: index=22 error="log not found" 2024-06-11T14:40:11.874Z [INFO] creating new snapshot: path=/tmp/raft2101468261/snapshots/2-23-1718116811874.tmp 2024-06-11T14:40:11.875Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:11.875Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: copied to local snapshot: bytes=71 2024-06-11T14:40:11.876Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: snapshot restore progress: id=2-23-1718116811874 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:11.876Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: removing all old logs from log store 2024-06-11T14:40:11.876Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: compacting logs: from=1 to=22 2024-06-11T14:40:11.876Z [INFO] server-368de480-8bd8-a228-8bfd-6efab6088d64: Installed remote snapshot 2024-06-11T14:40:11.876Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: pipelining replication: peer="{Voter server-368de480-8bd8-a228-8bfd-6efab6088d64 368de480-8bd8-a228-8bfd-6efab6088d64}" 2024-06-11T14:40:12.877Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:11.871825543 +0000 UTC m=+18.454904455 (2 nodes), 1.4µs from start of poll, 1.548206417s from cluster start. Timeout at 2024-06-11 14:40:12.877271623 +0000 UTC m=+19.460350575, 1.00544612s after stability 2024-06-11T14:40:12.890Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: aborting pipeline replication: peer="{Voter server-368de480-8bd8-a228-8bfd-6efab6088d64 368de480-8bd8-a228-8bfd-6efab6088d64}" 2024-06-11T14:40:12.890Z [INFO] server-5bc8756a-e78c-d97b-8acc-ba3d28d23ea6: aborting pipeline replication: peer="{Voter server-be4ddad8-5723-266d-47ef-8da1ff18de84 be4ddad8-5723-266d-47ef-8da1ff18de84}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2024-06-11T14:40:12.893Z [INFO] server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57 Address:5873c42e-56ea-86d9-dbaf-7cd6658fbc57} {Suffrage:Voter ID:server-9d519251-69ce-d247-9b54-eeb3ce1a0a25 Address:9d519251-69ce-d247-9b54-eeb3ce1a0a25} {Suffrage:Voter ID:server-090f4352-1e9f-7126-8cad-f9d497b866dc Address:090f4352-1e9f-7126-8cad-f9d497b866dc}]" 2024-06-11T14:40:12.893Z [INFO] server-9d519251-69ce-d247-9b54-eeb3ce1a0a25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57 Address:5873c42e-56ea-86d9-dbaf-7cd6658fbc57} {Suffrage:Voter ID:server-9d519251-69ce-d247-9b54-eeb3ce1a0a25 Address:9d519251-69ce-d247-9b54-eeb3ce1a0a25} {Suffrage:Voter ID:server-090f4352-1e9f-7126-8cad-f9d497b866dc Address:090f4352-1e9f-7126-8cad-f9d497b866dc}]" 2024-06-11T14:40:12.893Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57 Address:5873c42e-56ea-86d9-dbaf-7cd6658fbc57} {Suffrage:Voter ID:server-9d519251-69ce-d247-9b54-eeb3ce1a0a25 Address:9d519251-69ce-d247-9b54-eeb3ce1a0a25} {Suffrage:Voter ID:server-090f4352-1e9f-7126-8cad-f9d497b866dc Address:090f4352-1e9f-7126-8cad-f9d497b866dc}]" 2024-06-11T14:40:12.893Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:12.893Z [INFO] server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57: entering follower state: follower="Node at 5873c42e-56ea-86d9-dbaf-7cd6658fbc57 [Follower]" leader-address= leader-id= 2024-06-11T14:40:12.893Z [INFO] server-9d519251-69ce-d247-9b54-eeb3ce1a0a25: entering follower state: follower="Node at 9d519251-69ce-d247-9b54-eeb3ce1a0a25 [Follower]" leader-address= leader-id= 2024-06-11T14:40:12.894Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: entering follower state: follower="Node at 090f4352-1e9f-7126-8cad-f9d497b866dc [Follower]" leader-address= leader-id= 2024-06-11T14:40:13.575Z [WARN] server-090f4352-1e9f-7126-8cad-f9d497b866dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:13.575Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: entering candidate state: node="Node at 090f4352-1e9f-7126-8cad-f9d497b866dc [Candidate]" term=2 2024-06-11T14:40:13.575Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: election won: term=2 tally=2 2024-06-11T14:40:13.575Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: entering leader state: leader="Node at 090f4352-1e9f-7126-8cad-f9d497b866dc [Leader]" 2024-06-11T14:40:13.575Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: added peer, starting replication: peer=server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57 2024-06-11T14:40:13.575Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: added peer, starting replication: peer=server-9d519251-69ce-d247-9b54-eeb3ce1a0a25 2024-06-11T14:40:13.576Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: pipelining replication: peer="{Voter server-9d519251-69ce-d247-9b54-eeb3ce1a0a25 9d519251-69ce-d247-9b54-eeb3ce1a0a25}" 2024-06-11T14:40:13.576Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: pipelining replication: peer="{Voter server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57 5873c42e-56ea-86d9-dbaf-7cd6658fbc57}" 2024-06-11T14:40:14.581Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:13.576388517 +0000 UTC m=+20.159467429 (1 nodes), 682.502413ms from start of poll, 683.131591ms from cluster start. Timeout at 2024-06-11 14:40:14.581411973 +0000 UTC m=+21.164490925, 1.005023496s after stability 2024-06-11T14:40:14.582Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: starting snapshot up to: index=12 2024-06-11T14:40:14.582Z [INFO] creating new snapshot: path=/tmp/raft2337306104/snapshots/2-12-1718116814582.tmp 2024-06-11T14:40:14.584Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: snapshot complete up to: index=12 2024-06-11T14:40:14.586Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c Address:a0e933a5-df18-5d92-5d19-4e02dfb9ff4c} {Suffrage:Voter ID:server-cfa40785-9e2f-98ef-dad9-7d852813e03e Address:cfa40785-9e2f-98ef-dad9-7d852813e03e} {Suffrage:Voter ID:server-45f9323a-295a-7845-ab6a-ffc99fc84cb5 Address:45f9323a-295a-7845-ab6a-ffc99fc84cb5}]" 2024-06-11T14:40:14.587Z [INFO] server-cfa40785-9e2f-98ef-dad9-7d852813e03e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c Address:a0e933a5-df18-5d92-5d19-4e02dfb9ff4c} {Suffrage:Voter ID:server-cfa40785-9e2f-98ef-dad9-7d852813e03e Address:cfa40785-9e2f-98ef-dad9-7d852813e03e} {Suffrage:Voter ID:server-45f9323a-295a-7845-ab6a-ffc99fc84cb5 Address:45f9323a-295a-7845-ab6a-ffc99fc84cb5}]" 2024-06-11T14:40:14.587Z [INFO] server-45f9323a-295a-7845-ab6a-ffc99fc84cb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c Address:a0e933a5-df18-5d92-5d19-4e02dfb9ff4c} {Suffrage:Voter ID:server-cfa40785-9e2f-98ef-dad9-7d852813e03e Address:cfa40785-9e2f-98ef-dad9-7d852813e03e} {Suffrage:Voter ID:server-45f9323a-295a-7845-ab6a-ffc99fc84cb5 Address:45f9323a-295a-7845-ab6a-ffc99fc84cb5}]" 2024-06-11T14:40:14.587Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:14.587Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: entering follower state: follower="Node at a0e933a5-df18-5d92-5d19-4e02dfb9ff4c [Follower]" leader-address= leader-id= 2024-06-11T14:40:14.587Z [INFO] server-cfa40785-9e2f-98ef-dad9-7d852813e03e: entering follower state: follower="Node at cfa40785-9e2f-98ef-dad9-7d852813e03e [Follower]" leader-address= leader-id= 2024-06-11T14:40:14.587Z [INFO] server-45f9323a-295a-7845-ab6a-ffc99fc84cb5: entering follower state: follower="Node at 45f9323a-295a-7845-ab6a-ffc99fc84cb5 [Follower]" leader-address= leader-id= 2024-06-11T14:40:15.219Z [WARN] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:15.219Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: entering candidate state: node="Node at a0e933a5-df18-5d92-5d19-4e02dfb9ff4c [Candidate]" term=2 2024-06-11T14:40:15.219Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: election won: term=2 tally=2 2024-06-11T14:40:15.219Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: entering leader state: leader="Node at a0e933a5-df18-5d92-5d19-4e02dfb9ff4c [Leader]" 2024-06-11T14:40:15.219Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: added peer, starting replication: peer=server-cfa40785-9e2f-98ef-dad9-7d852813e03e 2024-06-11T14:40:15.219Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: added peer, starting replication: peer=server-45f9323a-295a-7845-ab6a-ffc99fc84cb5 2024-06-11T14:40:15.219Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: pipelining replication: peer="{Voter server-45f9323a-295a-7845-ab6a-ffc99fc84cb5 45f9323a-295a-7845-ab6a-ffc99fc84cb5}" 2024-06-11T14:40:15.220Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: pipelining replication: peer="{Voter server-cfa40785-9e2f-98ef-dad9-7d852813e03e cfa40785-9e2f-98ef-dad9-7d852813e03e}" 2024-06-11T14:40:16.226Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:15.220265681 +0000 UTC m=+21.803344633 (1 nodes), 632.939481ms from start of poll, 634.231675ms from cluster start. Timeout at 2024-06-11 14:40:16.225987912 +0000 UTC m=+22.809066824, 1.005722191s after stability 2024-06-11T14:40:16.226Z [INFO] creating new snapshot: path=/tmp/raft760836910/snapshots/2-13-1718116816226.tmp 2024-06-11T14:40:16.227Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: copied to local snapshot: bytes=71 2024-06-11T14:40:16.228Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: snapshot restore progress: id=2-13-1718116816226 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:16.228Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: restored user snapshot: index=13 2024-06-11T14:40:16.228Z [ERROR] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: failed to get log: index=3 error="log not found" 2024-06-11T14:40:16.228Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: aborting pipeline replication: peer="{Voter server-45f9323a-295a-7845-ab6a-ffc99fc84cb5 45f9323a-295a-7845-ab6a-ffc99fc84cb5}" 2024-06-11T14:40:16.228Z [ERROR] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: failed to get log: index=3 error="log not found" 2024-06-11T14:40:16.228Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: aborting pipeline replication: peer="{Voter server-cfa40785-9e2f-98ef-dad9-7d852813e03e cfa40785-9e2f-98ef-dad9-7d852813e03e}" 2024-06-11T14:40:16.235Z [ERROR] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: failed to get log: index=3 error="log not found" 2024-06-11T14:40:16.239Z [ERROR] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: failed to get log: index=3 error="log not found" 2024-06-11T14:40:16.240Z [INFO] creating new snapshot: path=/tmp/raft2227942313/snapshots/2-13-1718116816240.tmp 2024-06-11T14:40:16.241Z [INFO] server-cfa40785-9e2f-98ef-dad9-7d852813e03e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:16.242Z [INFO] server-cfa40785-9e2f-98ef-dad9-7d852813e03e: copied to local snapshot: bytes=71 2024-06-11T14:40:16.242Z [INFO] server-cfa40785-9e2f-98ef-dad9-7d852813e03e: snapshot restore progress: id=2-13-1718116816240 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:16.242Z [INFO] server-cfa40785-9e2f-98ef-dad9-7d852813e03e: Installed remote snapshot 2024-06-11T14:40:16.243Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: pipelining replication: peer="{Voter server-cfa40785-9e2f-98ef-dad9-7d852813e03e cfa40785-9e2f-98ef-dad9-7d852813e03e}" 2024-06-11T14:40:16.243Z [INFO] creating new snapshot: path=/tmp/raft301687321/snapshots/2-13-1718116816243.tmp 2024-06-11T14:40:16.243Z [INFO] server-45f9323a-295a-7845-ab6a-ffc99fc84cb5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:16.244Z [INFO] server-45f9323a-295a-7845-ab6a-ffc99fc84cb5: copied to local snapshot: bytes=71 2024-06-11T14:40:16.244Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:16.244Z [INFO] server-45f9323a-295a-7845-ab6a-ffc99fc84cb5: snapshot restore progress: id=2-13-1718116816243 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:16.244Z [INFO] server-45f9323a-295a-7845-ab6a-ffc99fc84cb5: Installed remote snapshot 2024-06-11T14:40:16.244Z [INFO] server-a0e933a5-df18-5d92-5d19-4e02dfb9ff4c: pipelining replication: peer="{Voter server-45f9323a-295a-7845-ab6a-ffc99fc84cb5 45f9323a-295a-7845-ab6a-ffc99fc84cb5}" 2024-06-11T14:40:17.249Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:16.244357489 +0000 UTC m=+22.827436441 (2 nodes), 1.6µs from start of poll, 1.658323483s from cluster start. Timeout at 2024-06-11 14:40:17.249877007 +0000 UTC m=+23.832955959, 1.005519518s after stability 2024-06-11T14:40:17.263Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: aborting pipeline replication: peer="{Voter server-5873c42e-56ea-86d9-dbaf-7cd6658fbc57 5873c42e-56ea-86d9-dbaf-7cd6658fbc57}" 2024-06-11T14:40:17.263Z [INFO] server-090f4352-1e9f-7126-8cad-f9d497b866dc: aborting pipeline replication: peer="{Voter server-9d519251-69ce-d247-9b54-eeb3ce1a0a25 9d519251-69ce-d247-9b54-eeb3ce1a0a25}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2024-06-11T14:40:17.265Z [INFO] server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3 Address:cce5f04b-68a6-32c8-1a6c-7d37061cd0c3} {Suffrage:Voter ID:server-4d7237a9-73d1-99de-09de-84096dfefb70 Address:4d7237a9-73d1-99de-09de-84096dfefb70} {Suffrage:Voter ID:server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89 Address:62fcfd35-f89b-87e0-7ae9-4159f00b6b89}]" 2024-06-11T14:40:17.265Z [INFO] server-4d7237a9-73d1-99de-09de-84096dfefb70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3 Address:cce5f04b-68a6-32c8-1a6c-7d37061cd0c3} {Suffrage:Voter ID:server-4d7237a9-73d1-99de-09de-84096dfefb70 Address:4d7237a9-73d1-99de-09de-84096dfefb70} {Suffrage:Voter ID:server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89 Address:62fcfd35-f89b-87e0-7ae9-4159f00b6b89}]" 2024-06-11T14:40:17.266Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3 Address:cce5f04b-68a6-32c8-1a6c-7d37061cd0c3} {Suffrage:Voter ID:server-4d7237a9-73d1-99de-09de-84096dfefb70 Address:4d7237a9-73d1-99de-09de-84096dfefb70} {Suffrage:Voter ID:server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89 Address:62fcfd35-f89b-87e0-7ae9-4159f00b6b89}]" 2024-06-11T14:40:17.266Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:17.266Z [INFO] server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3: entering follower state: follower="Node at cce5f04b-68a6-32c8-1a6c-7d37061cd0c3 [Follower]" leader-address= leader-id= 2024-06-11T14:40:17.266Z [INFO] server-4d7237a9-73d1-99de-09de-84096dfefb70: entering follower state: follower="Node at 4d7237a9-73d1-99de-09de-84096dfefb70 [Follower]" leader-address= leader-id= 2024-06-11T14:40:17.266Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: entering follower state: follower="Node at 62fcfd35-f89b-87e0-7ae9-4159f00b6b89 [Follower]" leader-address= leader-id= 2024-06-11T14:40:17.836Z [WARN] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:17.836Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: entering candidate state: node="Node at 62fcfd35-f89b-87e0-7ae9-4159f00b6b89 [Candidate]" term=2 2024-06-11T14:40:17.837Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: election won: term=2 tally=2 2024-06-11T14:40:17.837Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: entering leader state: leader="Node at 62fcfd35-f89b-87e0-7ae9-4159f00b6b89 [Leader]" 2024-06-11T14:40:17.837Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: added peer, starting replication: peer=server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3 2024-06-11T14:40:17.837Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: added peer, starting replication: peer=server-4d7237a9-73d1-99de-09de-84096dfefb70 2024-06-11T14:40:17.837Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: pipelining replication: peer="{Voter server-4d7237a9-73d1-99de-09de-84096dfefb70 4d7237a9-73d1-99de-09de-84096dfefb70}" 2024-06-11T14:40:17.837Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: pipelining replication: peer="{Voter server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3 cce5f04b-68a6-32c8-1a6c-7d37061cd0c3}" 2024-06-11T14:40:18.843Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:17.837642759 +0000 UTC m=+24.420721711 (1 nodes), 571.560176ms from start of poll, 572.184954ms from cluster start. Timeout at 2024-06-11 14:40:18.843774375 +0000 UTC m=+25.426853327, 1.006131616s after stability 2024-06-11T14:40:18.845Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: starting snapshot up to: index=12 2024-06-11T14:40:18.845Z [INFO] creating new snapshot: path=/tmp/raft3551193293/snapshots/2-12-1718116818845.tmp 2024-06-11T14:40:18.846Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: snapshot complete up to: index=12 2024-06-11T14:40:18.848Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76edade-e8b5-63fe-fe3f-ca601fc384e4 Address:f76edade-e8b5-63fe-fe3f-ca601fc384e4} {Suffrage:Voter ID:server-e64079de-ba6b-f0e0-a862-737b6c593cba Address:e64079de-ba6b-f0e0-a862-737b6c593cba} {Suffrage:Voter ID:server-60c93417-6c40-b4ad-7288-4800cca2d783 Address:60c93417-6c40-b4ad-7288-4800cca2d783}]" 2024-06-11T14:40:18.848Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76edade-e8b5-63fe-fe3f-ca601fc384e4 Address:f76edade-e8b5-63fe-fe3f-ca601fc384e4} {Suffrage:Voter ID:server-e64079de-ba6b-f0e0-a862-737b6c593cba Address:e64079de-ba6b-f0e0-a862-737b6c593cba} {Suffrage:Voter ID:server-60c93417-6c40-b4ad-7288-4800cca2d783 Address:60c93417-6c40-b4ad-7288-4800cca2d783}]" 2024-06-11T14:40:18.848Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76edade-e8b5-63fe-fe3f-ca601fc384e4 Address:f76edade-e8b5-63fe-fe3f-ca601fc384e4} {Suffrage:Voter ID:server-e64079de-ba6b-f0e0-a862-737b6c593cba Address:e64079de-ba6b-f0e0-a862-737b6c593cba} {Suffrage:Voter ID:server-60c93417-6c40-b4ad-7288-4800cca2d783 Address:60c93417-6c40-b4ad-7288-4800cca2d783}]" 2024-06-11T14:40:18.848Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:18.848Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: entering follower state: follower="Node at f76edade-e8b5-63fe-fe3f-ca601fc384e4 [Follower]" leader-address= leader-id= 2024-06-11T14:40:18.849Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: entering follower state: follower="Node at e64079de-ba6b-f0e0-a862-737b6c593cba [Follower]" leader-address= leader-id= 2024-06-11T14:40:18.849Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: entering follower state: follower="Node at 60c93417-6c40-b4ad-7288-4800cca2d783 [Follower]" leader-address= leader-id= 2024-06-11T14:40:19.434Z [WARN] server-e64079de-ba6b-f0e0-a862-737b6c593cba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:19.434Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: entering candidate state: node="Node at e64079de-ba6b-f0e0-a862-737b6c593cba [Candidate]" term=2 2024-06-11T14:40:19.434Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: election won: term=2 tally=2 2024-06-11T14:40:19.434Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: entering leader state: leader="Node at e64079de-ba6b-f0e0-a862-737b6c593cba [Leader]" 2024-06-11T14:40:19.434Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: added peer, starting replication: peer=server-f76edade-e8b5-63fe-fe3f-ca601fc384e4 2024-06-11T14:40:19.434Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: added peer, starting replication: peer=server-60c93417-6c40-b4ad-7288-4800cca2d783 2024-06-11T14:40:19.434Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: pipelining replication: peer="{Voter server-60c93417-6c40-b4ad-7288-4800cca2d783 60c93417-6c40-b4ad-7288-4800cca2d783}" 2024-06-11T14:40:19.435Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: pipelining replication: peer="{Voter server-f76edade-e8b5-63fe-fe3f-ca601fc384e4 f76edade-e8b5-63fe-fe3f-ca601fc384e4}" 2024-06-11T14:40:20.440Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:19.435230954 +0000 UTC m=+26.018309906 (1 nodes), 586.287126ms from start of poll, 586.960301ms from cluster start. Timeout at 2024-06-11 14:40:20.440478721 +0000 UTC m=+27.023557633, 1.005247727s after stability 2024-06-11T14:40:20.440Z [INFO] creating new snapshot: path=/tmp/raft3887510886/snapshots/2-13-1718116820440.tmp 2024-06-11T14:40:20.441Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: copied to local snapshot: bytes=71 2024-06-11T14:40:20.442Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: snapshot restore progress: id=2-13-1718116820440 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:20.442Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: removing all old logs from log store 2024-06-11T14:40:20.442Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: compacting logs: from=1 to=2 2024-06-11T14:40:20.442Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: restored user snapshot: index=13 2024-06-11T14:40:20.442Z [ERROR] server-e64079de-ba6b-f0e0-a862-737b6c593cba: failed to get log: index=2 error="log not found" 2024-06-11T14:40:20.442Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: aborting pipeline replication: peer="{Voter server-60c93417-6c40-b4ad-7288-4800cca2d783 60c93417-6c40-b4ad-7288-4800cca2d783}" 2024-06-11T14:40:20.442Z [ERROR] server-e64079de-ba6b-f0e0-a862-737b6c593cba: failed to get log: index=2 error="log not found" 2024-06-11T14:40:20.442Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: aborting pipeline replication: peer="{Voter server-f76edade-e8b5-63fe-fe3f-ca601fc384e4 f76edade-e8b5-63fe-fe3f-ca601fc384e4}" 2024-06-11T14:40:20.449Z [ERROR] server-e64079de-ba6b-f0e0-a862-737b6c593cba: failed to get log: index=2 error="log not found" 2024-06-11T14:40:20.449Z [ERROR] server-e64079de-ba6b-f0e0-a862-737b6c593cba: failed to get log: index=2 error="log not found" 2024-06-11T14:40:20.450Z [INFO] creating new snapshot: path=/tmp/raft4292265409/snapshots/2-13-1718116820450.tmp 2024-06-11T14:40:20.450Z [INFO] creating new snapshot: path=/tmp/raft1857923010/snapshots/2-13-1718116820450.tmp 2024-06-11T14:40:20.450Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:20.450Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:20.451Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: copied to local snapshot: bytes=71 2024-06-11T14:40:20.451Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: copied to local snapshot: bytes=71 2024-06-11T14:40:20.451Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: snapshot restore progress: id=2-13-1718116820450 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:20.451Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: removing all old logs from log store 2024-06-11T14:40:20.451Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: compacting logs: from=1 to=2 2024-06-11T14:40:20.451Z [INFO] server-60c93417-6c40-b4ad-7288-4800cca2d783: Installed remote snapshot 2024-06-11T14:40:20.452Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: pipelining replication: peer="{Voter server-60c93417-6c40-b4ad-7288-4800cca2d783 60c93417-6c40-b4ad-7288-4800cca2d783}" 2024-06-11T14:40:20.452Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:20.452Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: snapshot restore progress: id=2-13-1718116820450 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:20.452Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: removing all old logs from log store 2024-06-11T14:40:20.452Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: compacting logs: from=1 to=2 2024-06-11T14:40:20.452Z [INFO] server-f76edade-e8b5-63fe-fe3f-ca601fc384e4: Installed remote snapshot 2024-06-11T14:40:20.452Z [INFO] server-e64079de-ba6b-f0e0-a862-737b6c593cba: pipelining replication: peer="{Voter server-f76edade-e8b5-63fe-fe3f-ca601fc384e4 f76edade-e8b5-63fe-fe3f-ca601fc384e4}" 2024-06-11T14:40:21.457Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:20.452221978 +0000 UTC m=+27.035300930 (2 nodes), 1.44µs from start of poll, 1.603951325s from cluster start. Timeout at 2024-06-11 14:40:21.457502344 +0000 UTC m=+28.040581296, 1.005280366s after stability 2024-06-11T14:40:21.471Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: aborting pipeline replication: peer="{Voter server-4d7237a9-73d1-99de-09de-84096dfefb70 4d7237a9-73d1-99de-09de-84096dfefb70}" 2024-06-11T14:40:21.471Z [INFO] server-62fcfd35-f89b-87e0-7ae9-4159f00b6b89: aborting pipeline replication: peer="{Voter server-cce5f04b-68a6-32c8-1a6c-7d37061cd0c3 cce5f04b-68a6-32c8-1a6c-7d37061cd0c3}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2024-06-11T14:40:21.476Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a36a58-13a0-bf16-e11d-c74586b9f75e Address:33a36a58-13a0-bf16-e11d-c74586b9f75e} {Suffrage:Voter ID:server-a6ade7e8-6077-515b-042c-e919b44737fb Address:a6ade7e8-6077-515b-042c-e919b44737fb} {Suffrage:Voter ID:server-1f16f900-98c0-2d51-10ba-9fadf606888c Address:1f16f900-98c0-2d51-10ba-9fadf606888c}]" 2024-06-11T14:40:21.476Z [INFO] server-a6ade7e8-6077-515b-042c-e919b44737fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a36a58-13a0-bf16-e11d-c74586b9f75e Address:33a36a58-13a0-bf16-e11d-c74586b9f75e} {Suffrage:Voter ID:server-a6ade7e8-6077-515b-042c-e919b44737fb Address:a6ade7e8-6077-515b-042c-e919b44737fb} {Suffrage:Voter ID:server-1f16f900-98c0-2d51-10ba-9fadf606888c Address:1f16f900-98c0-2d51-10ba-9fadf606888c}]" 2024-06-11T14:40:21.477Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: entering follower state: follower="Node at 33a36a58-13a0-bf16-e11d-c74586b9f75e [Follower]" leader-address= leader-id= 2024-06-11T14:40:21.477Z [INFO] server-1f16f900-98c0-2d51-10ba-9fadf606888c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a36a58-13a0-bf16-e11d-c74586b9f75e Address:33a36a58-13a0-bf16-e11d-c74586b9f75e} {Suffrage:Voter ID:server-a6ade7e8-6077-515b-042c-e919b44737fb Address:a6ade7e8-6077-515b-042c-e919b44737fb} {Suffrage:Voter ID:server-1f16f900-98c0-2d51-10ba-9fadf606888c Address:1f16f900-98c0-2d51-10ba-9fadf606888c}]" 2024-06-11T14:40:21.477Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:21.477Z [INFO] server-a6ade7e8-6077-515b-042c-e919b44737fb: entering follower state: follower="Node at a6ade7e8-6077-515b-042c-e919b44737fb [Follower]" leader-address= leader-id= 2024-06-11T14:40:21.477Z [INFO] server-1f16f900-98c0-2d51-10ba-9fadf606888c: entering follower state: follower="Node at 1f16f900-98c0-2d51-10ba-9fadf606888c [Follower]" leader-address= leader-id= 2024-06-11T14:40:22.125Z [WARN] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:22.125Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: entering candidate state: node="Node at 33a36a58-13a0-bf16-e11d-c74586b9f75e [Candidate]" term=2 2024-06-11T14:40:22.125Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: election won: term=2 tally=2 2024-06-11T14:40:22.125Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: entering leader state: leader="Node at 33a36a58-13a0-bf16-e11d-c74586b9f75e [Leader]" 2024-06-11T14:40:22.125Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: added peer, starting replication: peer=server-a6ade7e8-6077-515b-042c-e919b44737fb 2024-06-11T14:40:22.125Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: added peer, starting replication: peer=server-1f16f900-98c0-2d51-10ba-9fadf606888c 2024-06-11T14:40:22.125Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: pipelining replication: peer="{Voter server-1f16f900-98c0-2d51-10ba-9fadf606888c 1f16f900-98c0-2d51-10ba-9fadf606888c}" 2024-06-11T14:40:22.126Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: pipelining replication: peer="{Voter server-a6ade7e8-6077-515b-042c-e919b44737fb a6ade7e8-6077-515b-042c-e919b44737fb}" 2024-06-11T14:40:23.131Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:22.126248254 +0000 UTC m=+28.709327206 (1 nodes), 648.441642ms from start of poll, 650.484729ms from cluster start. Timeout at 2024-06-11 14:40:23.131660816 +0000 UTC m=+29.714739728, 1.005412522s after stability 2024-06-11T14:40:23.133Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: starting snapshot up to: index=12 2024-06-11T14:40:23.133Z [INFO] creating new snapshot: path=/tmp/raft1540024813/snapshots/2-12-1718116823133.tmp 2024-06-11T14:40:23.134Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: snapshot complete up to: index=12 2024-06-11T14:40:23.136Z [INFO] creating new snapshot: path=/tmp/raft1540024813/snapshots/2-23-1718116823136.tmp 2024-06-11T14:40:23.137Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: copied to local snapshot: bytes=71 2024-06-11T14:40:23.137Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: snapshot restore progress: id=2-23-1718116823136 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:23.137Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: restored user snapshot: index=23 2024-06-11T14:40:23.137Z [ERROR] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: failed to get log: index=23 error="log not found" 2024-06-11T14:40:23.137Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: aborting pipeline replication: peer="{Voter server-1f16f900-98c0-2d51-10ba-9fadf606888c 1f16f900-98c0-2d51-10ba-9fadf606888c}" 2024-06-11T14:40:23.137Z [ERROR] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: failed to get log: index=23 error="log not found" 2024-06-11T14:40:23.138Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: aborting pipeline replication: peer="{Voter server-a6ade7e8-6077-515b-042c-e919b44737fb a6ade7e8-6077-515b-042c-e919b44737fb}" 2024-06-11T14:40:23.145Z [ERROR] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: failed to get log: index=23 error="log not found" 2024-06-11T14:40:23.145Z [INFO] creating new snapshot: path=/tmp/raft141688729/snapshots/2-23-1718116823145.tmp 2024-06-11T14:40:23.146Z [INFO] server-a6ade7e8-6077-515b-042c-e919b44737fb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:23.146Z [INFO] server-a6ade7e8-6077-515b-042c-e919b44737fb: copied to local snapshot: bytes=71 2024-06-11T14:40:23.147Z [ERROR] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: failed to get log: index=23 error="log not found" 2024-06-11T14:40:23.147Z [INFO] creating new snapshot: path=/tmp/raft3805924788/snapshots/2-23-1718116823147.tmp 2024-06-11T14:40:23.148Z [INFO] server-1f16f900-98c0-2d51-10ba-9fadf606888c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:23.148Z [INFO] server-1f16f900-98c0-2d51-10ba-9fadf606888c: copied to local snapshot: bytes=71 2024-06-11T14:40:23.149Z [INFO] server-1f16f900-98c0-2d51-10ba-9fadf606888c: snapshot restore progress: id=2-23-1718116823147 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:23.149Z [INFO] server-1f16f900-98c0-2d51-10ba-9fadf606888c: Installed remote snapshot 2024-06-11T14:40:23.149Z [WARN] server-1f16f900-98c0-2d51-10ba-9fadf606888c: skipping application of old log: index=22 2024-06-11T14:40:23.149Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: pipelining replication: peer="{Voter server-1f16f900-98c0-2d51-10ba-9fadf606888c 1f16f900-98c0-2d51-10ba-9fadf606888c}" 2024-06-11T14:40:23.149Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:23.150Z [INFO] server-a6ade7e8-6077-515b-042c-e919b44737fb: snapshot restore progress: id=2-23-1718116823145 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:23.150Z [INFO] server-a6ade7e8-6077-515b-042c-e919b44737fb: Installed remote snapshot 2024-06-11T14:40:23.150Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: pipelining replication: peer="{Voter server-a6ade7e8-6077-515b-042c-e919b44737fb a6ade7e8-6077-515b-042c-e919b44737fb}" 2024-06-11T14:40:24.155Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:23.149867159 +0000 UTC m=+29.732946111 (2 nodes), 1.56µs from start of poll, 1.674103634s from cluster start. Timeout at 2024-06-11 14:40:24.155194524 +0000 UTC m=+30.738273476, 1.005327365s after stability 2024-06-11T14:40:24.166Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: aborting pipeline replication: peer="{Voter server-1f16f900-98c0-2d51-10ba-9fadf606888c 1f16f900-98c0-2d51-10ba-9fadf606888c}" 2024-06-11T14:40:24.167Z [INFO] server-33a36a58-13a0-bf16-e11d-c74586b9f75e: aborting pipeline replication: peer="{Voter server-a6ade7e8-6077-515b-042c-e919b44737fb a6ade7e8-6077-515b-042c-e919b44737fb}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2024-06-11T14:40:24.169Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1aa5b2d4-eb58-2fb4-9609-c870461e3170 Address:1aa5b2d4-eb58-2fb4-9609-c870461e3170} {Suffrage:Voter ID:server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e Address:6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e} {Suffrage:Voter ID:server-86d0d4de-eea9-ba78-da64-c0139531c59e Address:86d0d4de-eea9-ba78-da64-c0139531c59e}]" 2024-06-11T14:40:24.170Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: entering follower state: follower="Node at 1aa5b2d4-eb58-2fb4-9609-c870461e3170 [Follower]" leader-address= leader-id= 2024-06-11T14:40:24.170Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1aa5b2d4-eb58-2fb4-9609-c870461e3170 Address:1aa5b2d4-eb58-2fb4-9609-c870461e3170} {Suffrage:Voter ID:server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e Address:6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e} {Suffrage:Voter ID:server-86d0d4de-eea9-ba78-da64-c0139531c59e Address:86d0d4de-eea9-ba78-da64-c0139531c59e}]" 2024-06-11T14:40:24.170Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1aa5b2d4-eb58-2fb4-9609-c870461e3170 Address:1aa5b2d4-eb58-2fb4-9609-c870461e3170} {Suffrage:Voter ID:server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e Address:6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e} {Suffrage:Voter ID:server-86d0d4de-eea9-ba78-da64-c0139531c59e Address:86d0d4de-eea9-ba78-da64-c0139531c59e}]" 2024-06-11T14:40:24.170Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:24.170Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: entering follower state: follower="Node at 6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e [Follower]" leader-address= leader-id= 2024-06-11T14:40:24.170Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: entering follower state: follower="Node at 86d0d4de-eea9-ba78-da64-c0139531c59e [Follower]" leader-address= leader-id= 2024-06-11T14:40:24.880Z [WARN] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:24.880Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: entering candidate state: node="Node at 1aa5b2d4-eb58-2fb4-9609-c870461e3170 [Candidate]" term=2 2024-06-11T14:40:24.880Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: election won: term=2 tally=2 2024-06-11T14:40:24.880Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: entering leader state: leader="Node at 1aa5b2d4-eb58-2fb4-9609-c870461e3170 [Leader]" 2024-06-11T14:40:24.880Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: added peer, starting replication: peer=server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e 2024-06-11T14:40:24.881Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: added peer, starting replication: peer=server-86d0d4de-eea9-ba78-da64-c0139531c59e 2024-06-11T14:40:24.881Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: pipelining replication: peer="{Voter server-86d0d4de-eea9-ba78-da64-c0139531c59e 86d0d4de-eea9-ba78-da64-c0139531c59e}" 2024-06-11T14:40:24.881Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: pipelining replication: peer="{Voter server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e 6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e}" 2024-06-11T14:40:25.886Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:24.881513436 +0000 UTC m=+31.464592348 (1 nodes), 710.611959ms from start of poll, 711.786796ms from cluster start. Timeout at 2024-06-11 14:40:25.886776244 +0000 UTC m=+32.469855156, 1.005262808s after stability 2024-06-11T14:40:25.888Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: starting snapshot up to: index=12 2024-06-11T14:40:25.888Z [INFO] creating new snapshot: path=/tmp/raft2514855835/snapshots/2-12-1718116825888.tmp 2024-06-11T14:40:25.889Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: snapshot complete up to: index=12 2024-06-11T14:40:25.891Z [INFO] creating new snapshot: path=/tmp/raft2514855835/snapshots/2-23-1718116825891.tmp 2024-06-11T14:40:25.892Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: copied to local snapshot: bytes=71 2024-06-11T14:40:25.892Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: snapshot restore progress: id=2-23-1718116825891 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:25.892Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: removing all old logs from log store 2024-06-11T14:40:25.892Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: compacting logs: from=1 to=22 2024-06-11T14:40:25.892Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: restored user snapshot: index=23 2024-06-11T14:40:25.892Z [ERROR] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: failed to get log: index=22 error="log not found" 2024-06-11T14:40:25.892Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: aborting pipeline replication: peer="{Voter server-86d0d4de-eea9-ba78-da64-c0139531c59e 86d0d4de-eea9-ba78-da64-c0139531c59e}" 2024-06-11T14:40:25.892Z [ERROR] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: failed to get log: index=22 error="log not found" 2024-06-11T14:40:25.892Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: aborting pipeline replication: peer="{Voter server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e 6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e}" 2024-06-11T14:40:25.899Z [ERROR] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: failed to get log: index=22 error="log not found" 2024-06-11T14:40:25.899Z [INFO] creating new snapshot: path=/tmp/raft550428308/snapshots/2-23-1718116825899.tmp 2024-06-11T14:40:25.900Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:25.900Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: copied to local snapshot: bytes=71 2024-06-11T14:40:25.901Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: snapshot restore progress: id=2-23-1718116825899 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:25.901Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: removing all old logs from log store 2024-06-11T14:40:25.901Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: compacting logs: from=1 to=22 2024-06-11T14:40:25.901Z [INFO] server-86d0d4de-eea9-ba78-da64-c0139531c59e: Installed remote snapshot 2024-06-11T14:40:25.901Z [ERROR] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: failed to get log: index=22 error="log not found" 2024-06-11T14:40:25.902Z [INFO] creating new snapshot: path=/tmp/raft358195773/snapshots/2-23-1718116825902.tmp 2024-06-11T14:40:25.902Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:25.903Z [WARN] server-86d0d4de-eea9-ba78-da64-c0139531c59e: skipping application of old log: index=22 2024-06-11T14:40:25.903Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: pipelining replication: peer="{Voter server-86d0d4de-eea9-ba78-da64-c0139531c59e 86d0d4de-eea9-ba78-da64-c0139531c59e}" 2024-06-11T14:40:25.903Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: copied to local snapshot: bytes=71 2024-06-11T14:40:25.904Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:25.904Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: snapshot restore progress: id=2-23-1718116825902 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:25.904Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: removing all old logs from log store 2024-06-11T14:40:25.904Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: compacting logs: from=1 to=22 2024-06-11T14:40:25.904Z [INFO] server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e: Installed remote snapshot 2024-06-11T14:40:25.904Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: pipelining replication: peer="{Voter server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e 6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e}" 2024-06-11T14:40:26.909Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:25.904037141 +0000 UTC m=+32.487116093 (2 nodes), 1.72µs from start of poll, 1.734310541s from cluster start. Timeout at 2024-06-11 14:40:26.909486542 +0000 UTC m=+33.492565494, 1.005449401s after stability 2024-06-11T14:40:26.922Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: aborting pipeline replication: peer="{Voter server-86d0d4de-eea9-ba78-da64-c0139531c59e 86d0d4de-eea9-ba78-da64-c0139531c59e}" 2024-06-11T14:40:26.922Z [INFO] server-1aa5b2d4-eb58-2fb4-9609-c870461e3170: aborting pipeline replication: peer="{Voter server-6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e 6ec4cb20-1c3d-39cc-24d2-9f8f798b8d0e}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2024-06-11T14:40:26.925Z [INFO] server-328717c6-6ac5-2ce9-fe28-b3274727dedc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-328717c6-6ac5-2ce9-fe28-b3274727dedc Address:328717c6-6ac5-2ce9-fe28-b3274727dedc} {Suffrage:Voter ID:server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385 Address:75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385} {Suffrage:Voter ID:server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704 Address:e7c424a1-4a31-7db8-1f60-dc7f6e9fc704}]" 2024-06-11T14:40:26.925Z [INFO] server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-328717c6-6ac5-2ce9-fe28-b3274727dedc Address:328717c6-6ac5-2ce9-fe28-b3274727dedc} {Suffrage:Voter ID:server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385 Address:75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385} {Suffrage:Voter ID:server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704 Address:e7c424a1-4a31-7db8-1f60-dc7f6e9fc704}]" 2024-06-11T14:40:26.925Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-328717c6-6ac5-2ce9-fe28-b3274727dedc Address:328717c6-6ac5-2ce9-fe28-b3274727dedc} {Suffrage:Voter ID:server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385 Address:75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385} {Suffrage:Voter ID:server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704 Address:e7c424a1-4a31-7db8-1f60-dc7f6e9fc704}]" 2024-06-11T14:40:26.925Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:26.925Z [INFO] server-328717c6-6ac5-2ce9-fe28-b3274727dedc: entering follower state: follower="Node at 328717c6-6ac5-2ce9-fe28-b3274727dedc [Follower]" leader-address= leader-id= 2024-06-11T14:40:26.925Z [INFO] server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385: entering follower state: follower="Node at 75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385 [Follower]" leader-address= leader-id= 2024-06-11T14:40:26.925Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: entering follower state: follower="Node at e7c424a1-4a31-7db8-1f60-dc7f6e9fc704 [Follower]" leader-address= leader-id= 2024-06-11T14:40:27.486Z [WARN] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:27.486Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: entering candidate state: node="Node at e7c424a1-4a31-7db8-1f60-dc7f6e9fc704 [Candidate]" term=2 2024-06-11T14:40:27.486Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: election won: term=2 tally=2 2024-06-11T14:40:27.486Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: entering leader state: leader="Node at e7c424a1-4a31-7db8-1f60-dc7f6e9fc704 [Leader]" 2024-06-11T14:40:27.486Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: added peer, starting replication: peer=server-328717c6-6ac5-2ce9-fe28-b3274727dedc 2024-06-11T14:40:27.486Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: added peer, starting replication: peer=server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385 2024-06-11T14:40:27.487Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: pipelining replication: peer="{Voter server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385 75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385}" 2024-06-11T14:40:27.487Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: pipelining replication: peer="{Voter server-328717c6-6ac5-2ce9-fe28-b3274727dedc 328717c6-6ac5-2ce9-fe28-b3274727dedc}" 2024-06-11T14:40:28.493Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:27.487574722 +0000 UTC m=+34.070653674 (1 nodes), 561.822207ms from start of poll, 562.527982ms from cluster start. Timeout at 2024-06-11 14:40:28.493624261 +0000 UTC m=+35.076703213, 1.006049539s after stability 2024-06-11T14:40:28.494Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: starting snapshot up to: index=12 2024-06-11T14:40:28.494Z [INFO] creating new snapshot: path=/tmp/raft3342769941/snapshots/2-12-1718116828494.tmp 2024-06-11T14:40:28.495Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: snapshot complete up to: index=12 2024-06-11T14:40:28.498Z [INFO] server-8e0f887e-4481-3367-9760-10e69666caec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e0f887e-4481-3367-9760-10e69666caec Address:8e0f887e-4481-3367-9760-10e69666caec} {Suffrage:Voter ID:server-995a1355-4994-6ee4-7810-4faa973e49a9 Address:995a1355-4994-6ee4-7810-4faa973e49a9} {Suffrage:Voter ID:server-99a6cb27-f0d2-879a-2882-a8482a7ad456 Address:99a6cb27-f0d2-879a-2882-a8482a7ad456}]" 2024-06-11T14:40:28.499Z [INFO] server-8e0f887e-4481-3367-9760-10e69666caec: entering follower state: follower="Node at 8e0f887e-4481-3367-9760-10e69666caec [Follower]" leader-address= leader-id= 2024-06-11T14:40:28.499Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e0f887e-4481-3367-9760-10e69666caec Address:8e0f887e-4481-3367-9760-10e69666caec} {Suffrage:Voter ID:server-995a1355-4994-6ee4-7810-4faa973e49a9 Address:995a1355-4994-6ee4-7810-4faa973e49a9} {Suffrage:Voter ID:server-99a6cb27-f0d2-879a-2882-a8482a7ad456 Address:99a6cb27-f0d2-879a-2882-a8482a7ad456}]" 2024-06-11T14:40:28.499Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: entering follower state: follower="Node at 995a1355-4994-6ee4-7810-4faa973e49a9 [Follower]" leader-address= leader-id= 2024-06-11T14:40:28.499Z [INFO] server-99a6cb27-f0d2-879a-2882-a8482a7ad456: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e0f887e-4481-3367-9760-10e69666caec Address:8e0f887e-4481-3367-9760-10e69666caec} {Suffrage:Voter ID:server-995a1355-4994-6ee4-7810-4faa973e49a9 Address:995a1355-4994-6ee4-7810-4faa973e49a9} {Suffrage:Voter ID:server-99a6cb27-f0d2-879a-2882-a8482a7ad456 Address:99a6cb27-f0d2-879a-2882-a8482a7ad456}]" 2024-06-11T14:40:28.499Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:28.499Z [INFO] server-99a6cb27-f0d2-879a-2882-a8482a7ad456: entering follower state: follower="Node at 99a6cb27-f0d2-879a-2882-a8482a7ad456 [Follower]" leader-address= leader-id= 2024-06-11T14:40:29.379Z [WARN] server-995a1355-4994-6ee4-7810-4faa973e49a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:29.379Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: entering candidate state: node="Node at 995a1355-4994-6ee4-7810-4faa973e49a9 [Candidate]" term=2 2024-06-11T14:40:29.379Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: election won: term=2 tally=2 2024-06-11T14:40:29.379Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: entering leader state: leader="Node at 995a1355-4994-6ee4-7810-4faa973e49a9 [Leader]" 2024-06-11T14:40:29.379Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: added peer, starting replication: peer=server-8e0f887e-4481-3367-9760-10e69666caec 2024-06-11T14:40:29.379Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: added peer, starting replication: peer=server-99a6cb27-f0d2-879a-2882-a8482a7ad456 2024-06-11T14:40:29.379Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: pipelining replication: peer="{Voter server-99a6cb27-f0d2-879a-2882-a8482a7ad456 99a6cb27-f0d2-879a-2882-a8482a7ad456}" 2024-06-11T14:40:29.379Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: pipelining replication: peer="{Voter server-8e0f887e-4481-3367-9760-10e69666caec 8e0f887e-4481-3367-9760-10e69666caec}" 2024-06-11T14:40:30.385Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:29.379882043 +0000 UTC m=+35.962960995 (1 nodes), 880.47555ms from start of poll, 881.133366ms from cluster start. Timeout at 2024-06-11 14:40:30.385619993 +0000 UTC m=+36.968698945, 1.00573795s after stability 2024-06-11T14:40:30.386Z [INFO] creating new snapshot: path=/tmp/raft4230799436/snapshots/2-14-1718116830386.tmp 2024-06-11T14:40:30.387Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: copied to local snapshot: bytes=71 2024-06-11T14:40:30.387Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: snapshot restore progress: id=2-14-1718116830386 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:30.387Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: restored user snapshot: index=14 2024-06-11T14:40:30.387Z [ERROR] server-995a1355-4994-6ee4-7810-4faa973e49a9: failed to get log: index=3 error="log not found" 2024-06-11T14:40:30.387Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: aborting pipeline replication: peer="{Voter server-99a6cb27-f0d2-879a-2882-a8482a7ad456 99a6cb27-f0d2-879a-2882-a8482a7ad456}" 2024-06-11T14:40:30.387Z [ERROR] server-995a1355-4994-6ee4-7810-4faa973e49a9: failed to get log: index=3 error="log not found" 2024-06-11T14:40:30.387Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: aborting pipeline replication: peer="{Voter server-8e0f887e-4481-3367-9760-10e69666caec 8e0f887e-4481-3367-9760-10e69666caec}" 2024-06-11T14:40:30.393Z [ERROR] server-995a1355-4994-6ee4-7810-4faa973e49a9: failed to get log: index=3 error="log not found" 2024-06-11T14:40:30.393Z [INFO] creating new snapshot: path=/tmp/raft2672287613/snapshots/2-14-1718116830393.tmp 2024-06-11T14:40:30.394Z [INFO] server-8e0f887e-4481-3367-9760-10e69666caec: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:30.395Z [INFO] server-8e0f887e-4481-3367-9760-10e69666caec: copied to local snapshot: bytes=71 2024-06-11T14:40:30.395Z [INFO] server-8e0f887e-4481-3367-9760-10e69666caec: snapshot restore progress: id=2-14-1718116830393 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:30.395Z [INFO] server-8e0f887e-4481-3367-9760-10e69666caec: Installed remote snapshot 2024-06-11T14:40:30.395Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: pipelining replication: peer="{Voter server-8e0f887e-4481-3367-9760-10e69666caec 8e0f887e-4481-3367-9760-10e69666caec}" 2024-06-11T14:40:30.396Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:30.397Z [ERROR] server-995a1355-4994-6ee4-7810-4faa973e49a9: failed to get log: index=3 error="log not found" 2024-06-11T14:40:30.398Z [INFO] creating new snapshot: path=/tmp/raft404308063/snapshots/2-14-1718116830398.tmp 2024-06-11T14:40:30.399Z [INFO] server-99a6cb27-f0d2-879a-2882-a8482a7ad456: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:30.399Z [INFO] server-99a6cb27-f0d2-879a-2882-a8482a7ad456: copied to local snapshot: bytes=71 2024-06-11T14:40:30.400Z [INFO] server-99a6cb27-f0d2-879a-2882-a8482a7ad456: snapshot restore progress: id=2-14-1718116830398 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:30.400Z [INFO] server-99a6cb27-f0d2-879a-2882-a8482a7ad456: Installed remote snapshot 2024-06-11T14:40:30.400Z [INFO] server-995a1355-4994-6ee4-7810-4faa973e49a9: pipelining replication: peer="{Voter server-99a6cb27-f0d2-879a-2882-a8482a7ad456 99a6cb27-f0d2-879a-2882-a8482a7ad456}" 2024-06-11T14:40:31.402Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:30.396059656 +0000 UTC m=+36.979138568 (2 nodes), 1.56µs from start of poll, 1.897310939s from cluster start. Timeout at 2024-06-11 14:40:31.402438223 +0000 UTC m=+37.985517175, 1.006378607s after stability 2024-06-11T14:40:31.415Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: aborting pipeline replication: peer="{Voter server-328717c6-6ac5-2ce9-fe28-b3274727dedc 328717c6-6ac5-2ce9-fe28-b3274727dedc}" 2024-06-11T14:40:31.415Z [INFO] server-e7c424a1-4a31-7db8-1f60-dc7f6e9fc704: aborting pipeline replication: peer="{Voter server-75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385 75f2dc90-ce90-fdf2-e0fe-9f5e70fd2385}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2024-06-11T14:40:31.417Z [INFO] server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286 Address:6fbbdeec-9302-c7f6-1d94-ccf15e8bb286} {Suffrage:Voter ID:server-d93a9cf7-39f4-a641-defa-7f74620db930 Address:d93a9cf7-39f4-a641-defa-7f74620db930} {Suffrage:Voter ID:server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4 Address:594a2f4d-4b88-1e5e-ae12-9a74beead9f4}]" 2024-06-11T14:40:31.417Z [INFO] server-d93a9cf7-39f4-a641-defa-7f74620db930: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286 Address:6fbbdeec-9302-c7f6-1d94-ccf15e8bb286} {Suffrage:Voter ID:server-d93a9cf7-39f4-a641-defa-7f74620db930 Address:d93a9cf7-39f4-a641-defa-7f74620db930} {Suffrage:Voter ID:server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4 Address:594a2f4d-4b88-1e5e-ae12-9a74beead9f4}]" 2024-06-11T14:40:31.417Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286 Address:6fbbdeec-9302-c7f6-1d94-ccf15e8bb286} {Suffrage:Voter ID:server-d93a9cf7-39f4-a641-defa-7f74620db930 Address:d93a9cf7-39f4-a641-defa-7f74620db930} {Suffrage:Voter ID:server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4 Address:594a2f4d-4b88-1e5e-ae12-9a74beead9f4}]" 2024-06-11T14:40:31.418Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:31.418Z [INFO] server-d93a9cf7-39f4-a641-defa-7f74620db930: entering follower state: follower="Node at d93a9cf7-39f4-a641-defa-7f74620db930 [Follower]" leader-address= leader-id= 2024-06-11T14:40:31.418Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: entering follower state: follower="Node at 594a2f4d-4b88-1e5e-ae12-9a74beead9f4 [Follower]" leader-address= leader-id= 2024-06-11T14:40:31.418Z [INFO] server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286: entering follower state: follower="Node at 6fbbdeec-9302-c7f6-1d94-ccf15e8bb286 [Follower]" leader-address= leader-id= 2024-06-11T14:40:32.332Z [WARN] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:32.332Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: entering candidate state: node="Node at 594a2f4d-4b88-1e5e-ae12-9a74beead9f4 [Candidate]" term=2 2024-06-11T14:40:32.333Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: election won: term=2 tally=2 2024-06-11T14:40:32.333Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: entering leader state: leader="Node at 594a2f4d-4b88-1e5e-ae12-9a74beead9f4 [Leader]" 2024-06-11T14:40:32.333Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: added peer, starting replication: peer=server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286 2024-06-11T14:40:32.333Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: added peer, starting replication: peer=server-d93a9cf7-39f4-a641-defa-7f74620db930 2024-06-11T14:40:32.333Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: pipelining replication: peer="{Voter server-d93a9cf7-39f4-a641-defa-7f74620db930 d93a9cf7-39f4-a641-defa-7f74620db930}" 2024-06-11T14:40:32.333Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: pipelining replication: peer="{Voter server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286 6fbbdeec-9302-c7f6-1d94-ccf15e8bb286}" 2024-06-11T14:40:33.339Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:32.333839416 +0000 UTC m=+38.916918328 (1 nodes), 915.761557ms from start of poll, 916.453532ms from cluster start. Timeout at 2024-06-11 14:40:33.339066064 +0000 UTC m=+39.922145016, 1.005226688s after stability 2024-06-11T14:40:33.340Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: starting snapshot up to: index=12 2024-06-11T14:40:33.340Z [INFO] creating new snapshot: path=/tmp/raft3321900177/snapshots/2-12-1718116833340.tmp 2024-06-11T14:40:33.347Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: snapshot complete up to: index=12 2024-06-11T14:40:33.350Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-244a9c79-6fa6-a50b-8539-a0c7393c68fb Address:244a9c79-6fa6-a50b-8539-a0c7393c68fb} {Suffrage:Voter ID:server-093dbb6e-8ba3-5512-867d-8be1b1d706f7 Address:093dbb6e-8ba3-5512-867d-8be1b1d706f7} {Suffrage:Voter ID:server-781aaee7-3d24-608d-7201-eb0070157c22 Address:781aaee7-3d24-608d-7201-eb0070157c22}]" 2024-06-11T14:40:33.350Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: entering follower state: follower="Node at 244a9c79-6fa6-a50b-8539-a0c7393c68fb [Follower]" leader-address= leader-id= 2024-06-11T14:40:33.350Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-244a9c79-6fa6-a50b-8539-a0c7393c68fb Address:244a9c79-6fa6-a50b-8539-a0c7393c68fb} {Suffrage:Voter ID:server-093dbb6e-8ba3-5512-867d-8be1b1d706f7 Address:093dbb6e-8ba3-5512-867d-8be1b1d706f7} {Suffrage:Voter ID:server-781aaee7-3d24-608d-7201-eb0070157c22 Address:781aaee7-3d24-608d-7201-eb0070157c22}]" 2024-06-11T14:40:33.350Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-244a9c79-6fa6-a50b-8539-a0c7393c68fb Address:244a9c79-6fa6-a50b-8539-a0c7393c68fb} {Suffrage:Voter ID:server-093dbb6e-8ba3-5512-867d-8be1b1d706f7 Address:093dbb6e-8ba3-5512-867d-8be1b1d706f7} {Suffrage:Voter ID:server-781aaee7-3d24-608d-7201-eb0070157c22 Address:781aaee7-3d24-608d-7201-eb0070157c22}]" 2024-06-11T14:40:33.350Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:33.350Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: entering follower state: follower="Node at 093dbb6e-8ba3-5512-867d-8be1b1d706f7 [Follower]" leader-address= leader-id= 2024-06-11T14:40:33.350Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: entering follower state: follower="Node at 781aaee7-3d24-608d-7201-eb0070157c22 [Follower]" leader-address= leader-id= 2024-06-11T14:40:33.939Z [WARN] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:33.939Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: entering candidate state: node="Node at 093dbb6e-8ba3-5512-867d-8be1b1d706f7 [Candidate]" term=2 2024-06-11T14:40:33.939Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: election won: term=2 tally=2 2024-06-11T14:40:33.939Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: entering leader state: leader="Node at 093dbb6e-8ba3-5512-867d-8be1b1d706f7 [Leader]" 2024-06-11T14:40:33.940Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: added peer, starting replication: peer=server-244a9c79-6fa6-a50b-8539-a0c7393c68fb 2024-06-11T14:40:33.940Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: added peer, starting replication: peer=server-781aaee7-3d24-608d-7201-eb0070157c22 2024-06-11T14:40:33.940Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: pipelining replication: peer="{Voter server-781aaee7-3d24-608d-7201-eb0070157c22 781aaee7-3d24-608d-7201-eb0070157c22}" 2024-06-11T14:40:33.940Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: pipelining replication: peer="{Voter server-244a9c79-6fa6-a50b-8539-a0c7393c68fb 244a9c79-6fa6-a50b-8539-a0c7393c68fb}" 2024-06-11T14:40:34.945Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:33.940002541 +0000 UTC m=+40.523081453 (1 nodes), 589.138542ms from start of poll, 590.134746ms from cluster start. Timeout at 2024-06-11 14:40:34.945135873 +0000 UTC m=+41.528214825, 1.005133372s after stability 2024-06-11T14:40:34.945Z [INFO] creating new snapshot: path=/tmp/raft76425497/snapshots/2-14-1718116834945.tmp 2024-06-11T14:40:34.946Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: copied to local snapshot: bytes=71 2024-06-11T14:40:34.947Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: snapshot restore progress: id=2-14-1718116834945 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:34.947Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: removing all old logs from log store 2024-06-11T14:40:34.947Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: compacting logs: from=1 to=2 2024-06-11T14:40:34.947Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: restored user snapshot: index=14 2024-06-11T14:40:34.948Z [ERROR] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: failed to get log: index=2 error="log not found" 2024-06-11T14:40:34.948Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: aborting pipeline replication: peer="{Voter server-244a9c79-6fa6-a50b-8539-a0c7393c68fb 244a9c79-6fa6-a50b-8539-a0c7393c68fb}" 2024-06-11T14:40:34.948Z [ERROR] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: failed to get log: index=2 error="log not found" 2024-06-11T14:40:34.948Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: aborting pipeline replication: peer="{Voter server-781aaee7-3d24-608d-7201-eb0070157c22 781aaee7-3d24-608d-7201-eb0070157c22}" 2024-06-11T14:40:34.955Z [ERROR] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: failed to get log: index=2 error="log not found" 2024-06-11T14:40:34.956Z [INFO] creating new snapshot: path=/tmp/raft2946935339/snapshots/2-14-1718116834956.tmp 2024-06-11T14:40:34.956Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:34.957Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: copied to local snapshot: bytes=71 2024-06-11T14:40:34.957Z [ERROR] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: failed to get log: index=2 error="log not found" 2024-06-11T14:40:34.957Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: snapshot restore progress: id=2-14-1718116834956 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:34.957Z [INFO] creating new snapshot: path=/tmp/raft1722186494/snapshots/2-14-1718116834957.tmp 2024-06-11T14:40:34.957Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: removing all old logs from log store 2024-06-11T14:40:34.957Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: compacting logs: from=1 to=2 2024-06-11T14:40:34.957Z [INFO] server-244a9c79-6fa6-a50b-8539-a0c7393c68fb: Installed remote snapshot 2024-06-11T14:40:34.958Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: pipelining replication: peer="{Voter server-244a9c79-6fa6-a50b-8539-a0c7393c68fb 244a9c79-6fa6-a50b-8539-a0c7393c68fb}" 2024-06-11T14:40:34.958Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:34.958Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:34.959Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: copied to local snapshot: bytes=71 2024-06-11T14:40:34.959Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: snapshot restore progress: id=2-14-1718116834957 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:34.959Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: removing all old logs from log store 2024-06-11T14:40:34.959Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: compacting logs: from=1 to=2 2024-06-11T14:40:34.959Z [INFO] server-781aaee7-3d24-608d-7201-eb0070157c22: Installed remote snapshot 2024-06-11T14:40:34.959Z [INFO] server-093dbb6e-8ba3-5512-867d-8be1b1d706f7: pipelining replication: peer="{Voter server-781aaee7-3d24-608d-7201-eb0070157c22 781aaee7-3d24-608d-7201-eb0070157c22}" 2024-06-11T14:40:35.964Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:34.958366276 +0000 UTC m=+41.541445188 (2 nodes), 2.08µs from start of poll, 1.608498481s from cluster start. Timeout at 2024-06-11 14:40:35.964055028 +0000 UTC m=+42.547133980, 1.005688792s after stability 2024-06-11T14:40:35.976Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: aborting pipeline replication: peer="{Voter server-d93a9cf7-39f4-a641-defa-7f74620db930 d93a9cf7-39f4-a641-defa-7f74620db930}" 2024-06-11T14:40:35.976Z [INFO] server-594a2f4d-4b88-1e5e-ae12-9a74beead9f4: aborting pipeline replication: peer="{Voter server-6fbbdeec-9302-c7f6-1d94-ccf15e8bb286 6fbbdeec-9302-c7f6-1d94-ccf15e8bb286}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2024-06-11T14:40:35.979Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c Address:78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c} {Suffrage:Voter ID:server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 Address:b6d1e88d-efb9-ab00-2c63-f03a8d6c7172} {Suffrage:Voter ID:server-84087ef5-fd01-ae1b-a347-abda111ac919 Address:84087ef5-fd01-ae1b-a347-abda111ac919}]" 2024-06-11T14:40:35.979Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: entering follower state: follower="Node at 78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c [Follower]" leader-address= leader-id= 2024-06-11T14:40:35.979Z [INFO] server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c Address:78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c} {Suffrage:Voter ID:server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 Address:b6d1e88d-efb9-ab00-2c63-f03a8d6c7172} {Suffrage:Voter ID:server-84087ef5-fd01-ae1b-a347-abda111ac919 Address:84087ef5-fd01-ae1b-a347-abda111ac919}]" 2024-06-11T14:40:35.979Z [INFO] server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172: entering follower state: follower="Node at b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 [Follower]" leader-address= leader-id= 2024-06-11T14:40:35.979Z [INFO] server-84087ef5-fd01-ae1b-a347-abda111ac919: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c Address:78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c} {Suffrage:Voter ID:server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 Address:b6d1e88d-efb9-ab00-2c63-f03a8d6c7172} {Suffrage:Voter ID:server-84087ef5-fd01-ae1b-a347-abda111ac919 Address:84087ef5-fd01-ae1b-a347-abda111ac919}]" 2024-06-11T14:40:35.979Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:35.979Z [INFO] server-84087ef5-fd01-ae1b-a347-abda111ac919: entering follower state: follower="Node at 84087ef5-fd01-ae1b-a347-abda111ac919 [Follower]" leader-address= leader-id= 2024-06-11T14:40:36.521Z [WARN] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:36.521Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: entering candidate state: node="Node at 78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c [Candidate]" term=2 2024-06-11T14:40:36.521Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: election won: term=2 tally=2 2024-06-11T14:40:36.521Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: entering leader state: leader="Node at 78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c [Leader]" 2024-06-11T14:40:36.521Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: added peer, starting replication: peer=server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 2024-06-11T14:40:36.521Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: added peer, starting replication: peer=server-84087ef5-fd01-ae1b-a347-abda111ac919 2024-06-11T14:40:36.522Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: pipelining replication: peer="{Voter server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 b6d1e88d-efb9-ab00-2c63-f03a8d6c7172}" 2024-06-11T14:40:36.522Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: pipelining replication: peer="{Voter server-84087ef5-fd01-ae1b-a347-abda111ac919 84087ef5-fd01-ae1b-a347-abda111ac919}" 2024-06-11T14:40:37.527Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:36.521942498 +0000 UTC m=+43.105021450 (1 nodes), 542.329392ms from start of poll, 542.987488ms from cluster start. Timeout at 2024-06-11 14:40:37.527040311 +0000 UTC m=+44.110119263, 1.005097813s after stability 2024-06-11T14:40:37.528Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: starting snapshot up to: index=12 2024-06-11T14:40:37.528Z [INFO] creating new snapshot: path=/tmp/raft2627778456/snapshots/2-12-1718116837528.tmp 2024-06-11T14:40:37.529Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: snapshot complete up to: index=12 2024-06-11T14:40:37.530Z [INFO] creating new snapshot: path=/tmp/raft2627778456/snapshots/2-23-1718116837530.tmp 2024-06-11T14:40:37.535Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: copied to local snapshot: bytes=71 2024-06-11T14:40:37.536Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: snapshot restore progress: id=2-23-1718116837530 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:37.536Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: restored user snapshot: index=23 2024-06-11T14:40:37.536Z [ERROR] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: failed to get log: index=23 error="log not found" 2024-06-11T14:40:37.536Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: aborting pipeline replication: peer="{Voter server-84087ef5-fd01-ae1b-a347-abda111ac919 84087ef5-fd01-ae1b-a347-abda111ac919}" 2024-06-11T14:40:37.536Z [ERROR] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: failed to get log: index=23 error="log not found" 2024-06-11T14:40:37.536Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: aborting pipeline replication: peer="{Voter server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 b6d1e88d-efb9-ab00-2c63-f03a8d6c7172}" 2024-06-11T14:40:37.544Z [ERROR] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: failed to get log: index=23 error="log not found" 2024-06-11T14:40:37.545Z [ERROR] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: failed to get log: index=23 error="log not found" 2024-06-11T14:40:37.545Z [INFO] creating new snapshot: path=/tmp/raft2352507705/snapshots/2-23-1718116837545.tmp 2024-06-11T14:40:37.545Z [INFO] creating new snapshot: path=/tmp/raft2259681825/snapshots/2-23-1718116837545.tmp 2024-06-11T14:40:37.546Z [INFO] server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:37.546Z [INFO] server-84087ef5-fd01-ae1b-a347-abda111ac919: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:37.547Z [INFO] server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172: copied to local snapshot: bytes=71 2024-06-11T14:40:37.547Z [INFO] server-84087ef5-fd01-ae1b-a347-abda111ac919: copied to local snapshot: bytes=71 2024-06-11T14:40:37.547Z [INFO] server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172: snapshot restore progress: id=2-23-1718116837545 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:37.547Z [INFO] server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172: Installed remote snapshot 2024-06-11T14:40:37.547Z [INFO] server-84087ef5-fd01-ae1b-a347-abda111ac919: snapshot restore progress: id=2-23-1718116837545 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:37.547Z [WARN] server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172: skipping application of old log: index=22 2024-06-11T14:40:37.547Z [INFO] server-84087ef5-fd01-ae1b-a347-abda111ac919: Installed remote snapshot 2024-06-11T14:40:37.547Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: pipelining replication: peer="{Voter server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 b6d1e88d-efb9-ab00-2c63-f03a8d6c7172}" 2024-06-11T14:40:37.547Z [WARN] server-84087ef5-fd01-ae1b-a347-abda111ac919: skipping application of old log: index=22 2024-06-11T14:40:37.548Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:37.548Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: pipelining replication: peer="{Voter server-84087ef5-fd01-ae1b-a347-abda111ac919 84087ef5-fd01-ae1b-a347-abda111ac919}" 2024-06-11T14:40:38.553Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:37.548048553 +0000 UTC m=+44.131127505 (2 nodes), 1.52µs from start of poll, 1.569093543s from cluster start. Timeout at 2024-06-11 14:40:38.553479634 +0000 UTC m=+45.136558586, 1.005431081s after stability 2024-06-11T14:40:38.566Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: aborting pipeline replication: peer="{Voter server-84087ef5-fd01-ae1b-a347-abda111ac919 84087ef5-fd01-ae1b-a347-abda111ac919}" 2024-06-11T14:40:38.566Z [INFO] server-78ce97b5-0a28-12e1-f6a6-ea9fa0d2d63c: aborting pipeline replication: peer="{Voter server-b6d1e88d-efb9-ab00-2c63-f03a8d6c7172 b6d1e88d-efb9-ab00-2c63-f03a8d6c7172}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2024-06-11T14:40:38.569Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-875cb992-8a5a-1225-b88a-5fe8223dff98 Address:875cb992-8a5a-1225-b88a-5fe8223dff98} {Suffrage:Voter ID:server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 Address:c3ac039b-7426-ff2e-183a-6d5c6cfb25d8} {Suffrage:Voter ID:server-a2e65242-f35a-e273-61c1-06bf8eb31b61 Address:a2e65242-f35a-e273-61c1-06bf8eb31b61}]" 2024-06-11T14:40:38.569Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-875cb992-8a5a-1225-b88a-5fe8223dff98 Address:875cb992-8a5a-1225-b88a-5fe8223dff98} {Suffrage:Voter ID:server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 Address:c3ac039b-7426-ff2e-183a-6d5c6cfb25d8} {Suffrage:Voter ID:server-a2e65242-f35a-e273-61c1-06bf8eb31b61 Address:a2e65242-f35a-e273-61c1-06bf8eb31b61}]" 2024-06-11T14:40:38.569Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-875cb992-8a5a-1225-b88a-5fe8223dff98 Address:875cb992-8a5a-1225-b88a-5fe8223dff98} {Suffrage:Voter ID:server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 Address:c3ac039b-7426-ff2e-183a-6d5c6cfb25d8} {Suffrage:Voter ID:server-a2e65242-f35a-e273-61c1-06bf8eb31b61 Address:a2e65242-f35a-e273-61c1-06bf8eb31b61}]" 2024-06-11T14:40:38.569Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:38.569Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: entering follower state: follower="Node at 875cb992-8a5a-1225-b88a-5fe8223dff98 [Follower]" leader-address= leader-id= 2024-06-11T14:40:38.569Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: entering follower state: follower="Node at c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 [Follower]" leader-address= leader-id= 2024-06-11T14:40:38.569Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: entering follower state: follower="Node at a2e65242-f35a-e273-61c1-06bf8eb31b61 [Follower]" leader-address= leader-id= 2024-06-11T14:40:39.099Z [WARN] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:39.099Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: entering candidate state: node="Node at a2e65242-f35a-e273-61c1-06bf8eb31b61 [Candidate]" term=2 2024-06-11T14:40:39.099Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: election won: term=2 tally=2 2024-06-11T14:40:39.099Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: entering leader state: leader="Node at a2e65242-f35a-e273-61c1-06bf8eb31b61 [Leader]" 2024-06-11T14:40:39.099Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: added peer, starting replication: peer=server-875cb992-8a5a-1225-b88a-5fe8223dff98 2024-06-11T14:40:39.099Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: added peer, starting replication: peer=server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 2024-06-11T14:40:39.099Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: pipelining replication: peer="{Voter server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 c3ac039b-7426-ff2e-183a-6d5c6cfb25d8}" 2024-06-11T14:40:39.099Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: pipelining replication: peer="{Voter server-875cb992-8a5a-1225-b88a-5fe8223dff98 875cb992-8a5a-1225-b88a-5fe8223dff98}" 2024-06-11T14:40:40.105Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:39.099708925 +0000 UTC m=+45.682787877 (1 nodes), 530.006436ms from start of poll, 530.644373ms from cluster start. Timeout at 2024-06-11 14:40:40.104886935 +0000 UTC m=+46.687965847, 1.00517797s after stability 2024-06-11T14:40:40.107Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: starting snapshot up to: index=12 2024-06-11T14:40:40.108Z [INFO] creating new snapshot: path=/tmp/raft490376019/snapshots/2-12-1718116840107.tmp 2024-06-11T14:40:40.109Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: snapshot complete up to: index=12 2024-06-11T14:40:40.110Z [INFO] creating new snapshot: path=/tmp/raft490376019/snapshots/2-23-1718116840110.tmp 2024-06-11T14:40:40.111Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: copied to local snapshot: bytes=71 2024-06-11T14:40:40.112Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: snapshot restore progress: id=2-23-1718116840110 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:40.112Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: removing all old logs from log store 2024-06-11T14:40:40.112Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: compacting logs: from=1 to=22 2024-06-11T14:40:40.112Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: restored user snapshot: index=23 2024-06-11T14:40:40.112Z [ERROR] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: failed to get log: index=22 error="log not found" 2024-06-11T14:40:40.112Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: aborting pipeline replication: peer="{Voter server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 c3ac039b-7426-ff2e-183a-6d5c6cfb25d8}" 2024-06-11T14:40:40.112Z [ERROR] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: failed to get log: index=22 error="log not found" 2024-06-11T14:40:40.112Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: aborting pipeline replication: peer="{Voter server-875cb992-8a5a-1225-b88a-5fe8223dff98 875cb992-8a5a-1225-b88a-5fe8223dff98}" 2024-06-11T14:40:40.119Z [ERROR] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: failed to get log: index=22 error="log not found" 2024-06-11T14:40:40.120Z [INFO] creating new snapshot: path=/tmp/raft4143704224/snapshots/2-23-1718116840120.tmp 2024-06-11T14:40:40.121Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:40.121Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: copied to local snapshot: bytes=71 2024-06-11T14:40:40.122Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: snapshot restore progress: id=2-23-1718116840120 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:40.122Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: removing all old logs from log store 2024-06-11T14:40:40.122Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: compacting logs: from=1 to=22 2024-06-11T14:40:40.122Z [INFO] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: Installed remote snapshot 2024-06-11T14:40:40.122Z [ERROR] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: failed to get log: index=22 error="log not found" 2024-06-11T14:40:40.122Z [WARN] server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8: skipping application of old log: index=22 2024-06-11T14:40:40.122Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: pipelining replication: peer="{Voter server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 c3ac039b-7426-ff2e-183a-6d5c6cfb25d8}" 2024-06-11T14:40:40.122Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:40.123Z [INFO] creating new snapshot: path=/tmp/raft3871267956/snapshots/2-23-1718116840123.tmp 2024-06-11T14:40:40.123Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:40.124Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: copied to local snapshot: bytes=71 2024-06-11T14:40:40.124Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: snapshot restore progress: id=2-23-1718116840123 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:40.124Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: removing all old logs from log store 2024-06-11T14:40:40.124Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: compacting logs: from=1 to=22 2024-06-11T14:40:40.124Z [INFO] server-875cb992-8a5a-1225-b88a-5fe8223dff98: Installed remote snapshot 2024-06-11T14:40:40.125Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: pipelining replication: peer="{Voter server-875cb992-8a5a-1225-b88a-5fe8223dff98 875cb992-8a5a-1225-b88a-5fe8223dff98}" 2024-06-11T14:40:41.128Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:40.122830168 +0000 UTC m=+46.705909120 (2 nodes), 1.52µs from start of poll, 1.553765616s from cluster start. Timeout at 2024-06-11 14:40:41.128346926 +0000 UTC m=+47.711425878, 1.005516758s after stability 2024-06-11T14:40:41.139Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: aborting pipeline replication: peer="{Voter server-c3ac039b-7426-ff2e-183a-6d5c6cfb25d8 c3ac039b-7426-ff2e-183a-6d5c6cfb25d8}" 2024-06-11T14:40:41.139Z [INFO] server-a2e65242-f35a-e273-61c1-06bf8eb31b61: aborting pipeline replication: peer="{Voter server-875cb992-8a5a-1225-b88a-5fe8223dff98 875cb992-8a5a-1225-b88a-5fe8223dff98}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2024-06-11T14:40:41.143Z [INFO] server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e Address:0ef7380e-55d1-540a-5e5a-d844b8c3ed0e} {Suffrage:Voter ID:server-e8171963-29a9-7fc0-f0f7-d85eb917f32d Address:e8171963-29a9-7fc0-f0f7-d85eb917f32d} {Suffrage:Voter ID:server-083744d4-3910-878a-9837-66fb4a512f7d Address:083744d4-3910-878a-9837-66fb4a512f7d}]" 2024-06-11T14:40:41.143Z [INFO] server-e8171963-29a9-7fc0-f0f7-d85eb917f32d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e Address:0ef7380e-55d1-540a-5e5a-d844b8c3ed0e} {Suffrage:Voter ID:server-e8171963-29a9-7fc0-f0f7-d85eb917f32d Address:e8171963-29a9-7fc0-f0f7-d85eb917f32d} {Suffrage:Voter ID:server-083744d4-3910-878a-9837-66fb4a512f7d Address:083744d4-3910-878a-9837-66fb4a512f7d}]" 2024-06-11T14:40:41.143Z [INFO] server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e: entering follower state: follower="Node at 0ef7380e-55d1-540a-5e5a-d844b8c3ed0e [Follower]" leader-address= leader-id= 2024-06-11T14:40:41.143Z [INFO] server-e8171963-29a9-7fc0-f0f7-d85eb917f32d: entering follower state: follower="Node at e8171963-29a9-7fc0-f0f7-d85eb917f32d [Follower]" leader-address= leader-id= 2024-06-11T14:40:41.143Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e Address:0ef7380e-55d1-540a-5e5a-d844b8c3ed0e} {Suffrage:Voter ID:server-e8171963-29a9-7fc0-f0f7-d85eb917f32d Address:e8171963-29a9-7fc0-f0f7-d85eb917f32d} {Suffrage:Voter ID:server-083744d4-3910-878a-9837-66fb4a512f7d Address:083744d4-3910-878a-9837-66fb4a512f7d}]" 2024-06-11T14:40:41.143Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:41.143Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: entering follower state: follower="Node at 083744d4-3910-878a-9837-66fb4a512f7d [Follower]" leader-address= leader-id= 2024-06-11T14:40:41.688Z [WARN] server-083744d4-3910-878a-9837-66fb4a512f7d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:41.688Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: entering candidate state: node="Node at 083744d4-3910-878a-9837-66fb4a512f7d [Candidate]" term=2 2024-06-11T14:40:41.688Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: election won: term=2 tally=2 2024-06-11T14:40:41.688Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: entering leader state: leader="Node at 083744d4-3910-878a-9837-66fb4a512f7d [Leader]" 2024-06-11T14:40:41.688Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: added peer, starting replication: peer=server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e 2024-06-11T14:40:41.688Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: added peer, starting replication: peer=server-e8171963-29a9-7fc0-f0f7-d85eb917f32d 2024-06-11T14:40:41.688Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: pipelining replication: peer="{Voter server-e8171963-29a9-7fc0-f0f7-d85eb917f32d e8171963-29a9-7fc0-f0f7-d85eb917f32d}" 2024-06-11T14:40:41.688Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: pipelining replication: peer="{Voter server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e 0ef7380e-55d1-540a-5e5a-d844b8c3ed0e}" 2024-06-11T14:40:42.694Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:41.688777584 +0000 UTC m=+48.271856536 (1 nodes), 545.079612ms from start of poll, 545.767907ms from cluster start. Timeout at 2024-06-11 14:40:42.694782005 +0000 UTC m=+49.277860957, 1.006004421s after stability 2024-06-11T14:40:42.696Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: starting snapshot up to: index=12 2024-06-11T14:40:42.696Z [INFO] creating new snapshot: path=/tmp/raft66759769/snapshots/2-12-1718116842696.tmp 2024-06-11T14:40:42.697Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: snapshot complete up to: index=12 2024-06-11T14:40:42.700Z [INFO] server-881eb42d-772e-fef0-2c80-29724a843d6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-881eb42d-772e-fef0-2c80-29724a843d6d Address:881eb42d-772e-fef0-2c80-29724a843d6d} {Suffrage:Voter ID:server-03229033-3920-c312-f574-3499e68b4d32 Address:03229033-3920-c312-f574-3499e68b4d32} {Suffrage:Voter ID:server-505da08a-20e5-b23b-ada1-4b130adea240 Address:505da08a-20e5-b23b-ada1-4b130adea240}]" 2024-06-11T14:40:42.700Z [INFO] server-881eb42d-772e-fef0-2c80-29724a843d6d: entering follower state: follower="Node at 881eb42d-772e-fef0-2c80-29724a843d6d [Follower]" leader-address= leader-id= 2024-06-11T14:40:42.700Z [INFO] server-03229033-3920-c312-f574-3499e68b4d32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-881eb42d-772e-fef0-2c80-29724a843d6d Address:881eb42d-772e-fef0-2c80-29724a843d6d} {Suffrage:Voter ID:server-03229033-3920-c312-f574-3499e68b4d32 Address:03229033-3920-c312-f574-3499e68b4d32} {Suffrage:Voter ID:server-505da08a-20e5-b23b-ada1-4b130adea240 Address:505da08a-20e5-b23b-ada1-4b130adea240}]" 2024-06-11T14:40:42.700Z [INFO] server-03229033-3920-c312-f574-3499e68b4d32: entering follower state: follower="Node at 03229033-3920-c312-f574-3499e68b4d32 [Follower]" leader-address= leader-id= 2024-06-11T14:40:42.700Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-881eb42d-772e-fef0-2c80-29724a843d6d Address:881eb42d-772e-fef0-2c80-29724a843d6d} {Suffrage:Voter ID:server-03229033-3920-c312-f574-3499e68b4d32 Address:03229033-3920-c312-f574-3499e68b4d32} {Suffrage:Voter ID:server-505da08a-20e5-b23b-ada1-4b130adea240 Address:505da08a-20e5-b23b-ada1-4b130adea240}]" 2024-06-11T14:40:42.700Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:42.700Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: entering follower state: follower="Node at 505da08a-20e5-b23b-ada1-4b130adea240 [Follower]" leader-address= leader-id= 2024-06-11T14:40:43.286Z [WARN] server-505da08a-20e5-b23b-ada1-4b130adea240: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:43.286Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: entering candidate state: node="Node at 505da08a-20e5-b23b-ada1-4b130adea240 [Candidate]" term=2 2024-06-11T14:40:43.286Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: election won: term=2 tally=2 2024-06-11T14:40:43.286Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: entering leader state: leader="Node at 505da08a-20e5-b23b-ada1-4b130adea240 [Leader]" 2024-06-11T14:40:43.286Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: added peer, starting replication: peer=server-881eb42d-772e-fef0-2c80-29724a843d6d 2024-06-11T14:40:43.286Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: added peer, starting replication: peer=server-03229033-3920-c312-f574-3499e68b4d32 2024-06-11T14:40:43.286Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: pipelining replication: peer="{Voter server-03229033-3920-c312-f574-3499e68b4d32 03229033-3920-c312-f574-3499e68b4d32}" 2024-06-11T14:40:43.287Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: pipelining replication: peer="{Voter server-881eb42d-772e-fef0-2c80-29724a843d6d 881eb42d-772e-fef0-2c80-29724a843d6d}" 2024-06-11T14:40:44.293Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:43.28714847 +0000 UTC m=+49.870227422 (1 nodes), 586.583634ms from start of poll, 587.291968ms from cluster start. Timeout at 2024-06-11 14:40:44.293046095 +0000 UTC m=+50.876125007, 1.005897585s after stability 2024-06-11T14:40:44.293Z [INFO] creating new snapshot: path=/tmp/raft89209673/snapshots/2-15-1718116844293.tmp 2024-06-11T14:40:44.295Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: copied to local snapshot: bytes=71 2024-06-11T14:40:44.295Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: snapshot restore progress: id=2-15-1718116844293 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:44.295Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: restored user snapshot: index=15 2024-06-11T14:40:44.295Z [ERROR] server-505da08a-20e5-b23b-ada1-4b130adea240: failed to get log: index=3 error="log not found" 2024-06-11T14:40:44.295Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: aborting pipeline replication: peer="{Voter server-03229033-3920-c312-f574-3499e68b4d32 03229033-3920-c312-f574-3499e68b4d32}" 2024-06-11T14:40:44.295Z [ERROR] server-505da08a-20e5-b23b-ada1-4b130adea240: failed to get log: index=3 error="log not found" 2024-06-11T14:40:44.296Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: aborting pipeline replication: peer="{Voter server-881eb42d-772e-fef0-2c80-29724a843d6d 881eb42d-772e-fef0-2c80-29724a843d6d}" 2024-06-11T14:40:44.301Z [ERROR] server-505da08a-20e5-b23b-ada1-4b130adea240: failed to get log: index=3 error="log not found" 2024-06-11T14:40:44.302Z [INFO] creating new snapshot: path=/tmp/raft2765779169/snapshots/2-15-1718116844302.tmp 2024-06-11T14:40:44.303Z [INFO] server-881eb42d-772e-fef0-2c80-29724a843d6d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:44.304Z [INFO] server-881eb42d-772e-fef0-2c80-29724a843d6d: copied to local snapshot: bytes=71 2024-06-11T14:40:44.304Z [INFO] server-881eb42d-772e-fef0-2c80-29724a843d6d: snapshot restore progress: id=2-15-1718116844302 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:44.304Z [INFO] server-881eb42d-772e-fef0-2c80-29724a843d6d: Installed remote snapshot 2024-06-11T14:40:44.304Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: pipelining replication: peer="{Voter server-881eb42d-772e-fef0-2c80-29724a843d6d 881eb42d-772e-fef0-2c80-29724a843d6d}" 2024-06-11T14:40:44.304Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:44.305Z [ERROR] server-505da08a-20e5-b23b-ada1-4b130adea240: failed to get log: index=3 error="log not found" 2024-06-11T14:40:44.305Z [INFO] creating new snapshot: path=/tmp/raft1349562943/snapshots/2-15-1718116844305.tmp 2024-06-11T14:40:44.306Z [INFO] server-03229033-3920-c312-f574-3499e68b4d32: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:44.307Z [INFO] server-03229033-3920-c312-f574-3499e68b4d32: copied to local snapshot: bytes=71 2024-06-11T14:40:44.307Z [INFO] server-03229033-3920-c312-f574-3499e68b4d32: snapshot restore progress: id=2-15-1718116844305 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:44.307Z [INFO] server-03229033-3920-c312-f574-3499e68b4d32: Installed remote snapshot 2024-06-11T14:40:44.307Z [INFO] server-505da08a-20e5-b23b-ada1-4b130adea240: pipelining replication: peer="{Voter server-03229033-3920-c312-f574-3499e68b4d32 03229033-3920-c312-f574-3499e68b4d32}" 2024-06-11T14:40:45.310Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:44.304969905 +0000 UTC m=+50.888048857 (2 nodes), 1.6µs from start of poll, 1.605113403s from cluster start. Timeout at 2024-06-11 14:40:45.310091557 +0000 UTC m=+51.893170469, 1.005121612s after stability 2024-06-11T14:40:45.322Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: aborting pipeline replication: peer="{Voter server-e8171963-29a9-7fc0-f0f7-d85eb917f32d e8171963-29a9-7fc0-f0f7-d85eb917f32d}" 2024-06-11T14:40:45.322Z [INFO] server-083744d4-3910-878a-9837-66fb4a512f7d: aborting pipeline replication: peer="{Voter server-0ef7380e-55d1-540a-5e5a-d844b8c3ed0e 0ef7380e-55d1-540a-5e5a-d844b8c3ed0e}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2024-06-11T14:40:45.328Z [INFO] server-23b7678e-168d-1138-e350-37204e15f8f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23b7678e-168d-1138-e350-37204e15f8f2 Address:23b7678e-168d-1138-e350-37204e15f8f2} {Suffrage:Voter ID:server-afa586ed-2954-abdb-58ef-c69ce44c5d0e Address:afa586ed-2954-abdb-58ef-c69ce44c5d0e} {Suffrage:Voter ID:server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc Address:fb65dd45-ee23-a04b-8e34-7e26d01c68dc}]" 2024-06-11T14:40:45.328Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23b7678e-168d-1138-e350-37204e15f8f2 Address:23b7678e-168d-1138-e350-37204e15f8f2} {Suffrage:Voter ID:server-afa586ed-2954-abdb-58ef-c69ce44c5d0e Address:afa586ed-2954-abdb-58ef-c69ce44c5d0e} {Suffrage:Voter ID:server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc Address:fb65dd45-ee23-a04b-8e34-7e26d01c68dc}]" 2024-06-11T14:40:45.328Z [INFO] server-23b7678e-168d-1138-e350-37204e15f8f2: entering follower state: follower="Node at 23b7678e-168d-1138-e350-37204e15f8f2 [Follower]" leader-address= leader-id= 2024-06-11T14:40:45.328Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: entering follower state: follower="Node at afa586ed-2954-abdb-58ef-c69ce44c5d0e [Follower]" leader-address= leader-id= 2024-06-11T14:40:45.328Z [INFO] server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23b7678e-168d-1138-e350-37204e15f8f2 Address:23b7678e-168d-1138-e350-37204e15f8f2} {Suffrage:Voter ID:server-afa586ed-2954-abdb-58ef-c69ce44c5d0e Address:afa586ed-2954-abdb-58ef-c69ce44c5d0e} {Suffrage:Voter ID:server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc Address:fb65dd45-ee23-a04b-8e34-7e26d01c68dc}]" 2024-06-11T14:40:45.328Z [INFO] server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc: entering follower state: follower="Node at fb65dd45-ee23-a04b-8e34-7e26d01c68dc [Follower]" leader-address= leader-id= 2024-06-11T14:40:45.328Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:46.134Z [WARN] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:46.134Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: entering candidate state: node="Node at afa586ed-2954-abdb-58ef-c69ce44c5d0e [Candidate]" term=2 2024-06-11T14:40:46.134Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: election won: term=2 tally=2 2024-06-11T14:40:46.134Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: entering leader state: leader="Node at afa586ed-2954-abdb-58ef-c69ce44c5d0e [Leader]" 2024-06-11T14:40:46.135Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: added peer, starting replication: peer=server-23b7678e-168d-1138-e350-37204e15f8f2 2024-06-11T14:40:46.135Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: added peer, starting replication: peer=server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc 2024-06-11T14:40:46.135Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: pipelining replication: peer="{Voter server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc fb65dd45-ee23-a04b-8e34-7e26d01c68dc}" 2024-06-11T14:40:46.135Z [WARN] server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc: rejecting vote request since our last term is greater: candidate=afa586ed-2954-abdb-58ef-c69ce44c5d0e last-term=2 last-candidate-term=1 2024-06-11T14:40:46.135Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: pipelining replication: peer="{Voter server-23b7678e-168d-1138-e350-37204e15f8f2 23b7678e-168d-1138-e350-37204e15f8f2}" 2024-06-11T14:40:47.140Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:46.135465495 +0000 UTC m=+52.718544447 (1 nodes), 806.446301ms from start of poll, 807.263471ms from cluster start. Timeout at 2024-06-11 14:40:47.140759022 +0000 UTC m=+53.723837934, 1.005293487s after stability 2024-06-11T14:40:47.142Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: starting snapshot up to: index=12 2024-06-11T14:40:47.142Z [INFO] creating new snapshot: path=/tmp/raft1593112946/snapshots/2-12-1718116847142.tmp 2024-06-11T14:40:47.143Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: snapshot complete up to: index=12 2024-06-11T14:40:47.145Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8072516-6ede-8538-61ba-5546f6ea5ec0 Address:d8072516-6ede-8538-61ba-5546f6ea5ec0} {Suffrage:Voter ID:server-7e82277d-e1b0-a916-f5a3-fe7393a4d126 Address:7e82277d-e1b0-a916-f5a3-fe7393a4d126} {Suffrage:Voter ID:server-70bb8dd8-fc71-384e-71ac-3a8f42f32525 Address:70bb8dd8-fc71-384e-71ac-3a8f42f32525}]" 2024-06-11T14:40:47.145Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: entering follower state: follower="Node at d8072516-6ede-8538-61ba-5546f6ea5ec0 [Follower]" leader-address= leader-id= 2024-06-11T14:40:47.145Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8072516-6ede-8538-61ba-5546f6ea5ec0 Address:d8072516-6ede-8538-61ba-5546f6ea5ec0} {Suffrage:Voter ID:server-7e82277d-e1b0-a916-f5a3-fe7393a4d126 Address:7e82277d-e1b0-a916-f5a3-fe7393a4d126} {Suffrage:Voter ID:server-70bb8dd8-fc71-384e-71ac-3a8f42f32525 Address:70bb8dd8-fc71-384e-71ac-3a8f42f32525}]" 2024-06-11T14:40:47.145Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: entering follower state: follower="Node at 7e82277d-e1b0-a916-f5a3-fe7393a4d126 [Follower]" leader-address= leader-id= 2024-06-11T14:40:47.146Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8072516-6ede-8538-61ba-5546f6ea5ec0 Address:d8072516-6ede-8538-61ba-5546f6ea5ec0} {Suffrage:Voter ID:server-7e82277d-e1b0-a916-f5a3-fe7393a4d126 Address:7e82277d-e1b0-a916-f5a3-fe7393a4d126} {Suffrage:Voter ID:server-70bb8dd8-fc71-384e-71ac-3a8f42f32525 Address:70bb8dd8-fc71-384e-71ac-3a8f42f32525}]" 2024-06-11T14:40:47.146Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:47.146Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: entering follower state: follower="Node at 70bb8dd8-fc71-384e-71ac-3a8f42f32525 [Follower]" leader-address= leader-id= 2024-06-11T14:40:47.771Z [WARN] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:47.771Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: entering candidate state: node="Node at 7e82277d-e1b0-a916-f5a3-fe7393a4d126 [Candidate]" term=2 2024-06-11T14:40:47.771Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: election won: term=2 tally=2 2024-06-11T14:40:47.771Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: entering leader state: leader="Node at 7e82277d-e1b0-a916-f5a3-fe7393a4d126 [Leader]" 2024-06-11T14:40:47.771Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: added peer, starting replication: peer=server-d8072516-6ede-8538-61ba-5546f6ea5ec0 2024-06-11T14:40:47.771Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: added peer, starting replication: peer=server-70bb8dd8-fc71-384e-71ac-3a8f42f32525 2024-06-11T14:40:47.771Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: pipelining replication: peer="{Voter server-d8072516-6ede-8538-61ba-5546f6ea5ec0 d8072516-6ede-8538-61ba-5546f6ea5ec0}" 2024-06-11T14:40:47.771Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: pipelining replication: peer="{Voter server-70bb8dd8-fc71-384e-71ac-3a8f42f32525 70bb8dd8-fc71-384e-71ac-3a8f42f32525}" 2024-06-11T14:40:48.777Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:47.771957366 +0000 UTC m=+54.355036318 (1 nodes), 625.674344ms from start of poll, 626.555432ms from cluster start. Timeout at 2024-06-11 14:40:48.777705236 +0000 UTC m=+55.360784188, 1.00574787s after stability 2024-06-11T14:40:48.778Z [INFO] creating new snapshot: path=/tmp/raft388407353/snapshots/2-15-1718116848778.tmp 2024-06-11T14:40:48.779Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: copied to local snapshot: bytes=71 2024-06-11T14:40:48.779Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: snapshot restore progress: id=2-15-1718116848778 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:48.779Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: removing all old logs from log store 2024-06-11T14:40:48.779Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: compacting logs: from=1 to=2 2024-06-11T14:40:48.779Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: restored user snapshot: index=15 2024-06-11T14:40:48.780Z [ERROR] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: failed to get log: index=2 error="log not found" 2024-06-11T14:40:48.780Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: aborting pipeline replication: peer="{Voter server-70bb8dd8-fc71-384e-71ac-3a8f42f32525 70bb8dd8-fc71-384e-71ac-3a8f42f32525}" 2024-06-11T14:40:48.780Z [ERROR] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: failed to get log: index=2 error="log not found" 2024-06-11T14:40:48.780Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: aborting pipeline replication: peer="{Voter server-d8072516-6ede-8538-61ba-5546f6ea5ec0 d8072516-6ede-8538-61ba-5546f6ea5ec0}" 2024-06-11T14:40:48.787Z [ERROR] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: failed to get log: index=2 error="log not found" 2024-06-11T14:40:48.787Z [ERROR] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: failed to get log: index=2 error="log not found" 2024-06-11T14:40:48.787Z [INFO] creating new snapshot: path=/tmp/raft1939587601/snapshots/2-15-1718116848787.tmp 2024-06-11T14:40:48.787Z [INFO] creating new snapshot: path=/tmp/raft683516414/snapshots/2-15-1718116848787.tmp 2024-06-11T14:40:48.788Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:48.789Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: copied to local snapshot: bytes=71 2024-06-11T14:40:48.790Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: snapshot restore progress: id=2-15-1718116848787 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:48.791Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: removing all old logs from log store 2024-06-11T14:40:48.791Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: compacting logs: from=1 to=2 2024-06-11T14:40:48.791Z [INFO] server-d8072516-6ede-8538-61ba-5546f6ea5ec0: Installed remote snapshot 2024-06-11T14:40:48.792Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: pipelining replication: peer="{Voter server-d8072516-6ede-8538-61ba-5546f6ea5ec0 d8072516-6ede-8538-61ba-5546f6ea5ec0}" 2024-06-11T14:40:48.794Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:48.795Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: copied to local snapshot: bytes=71 2024-06-11T14:40:48.795Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:48.795Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: snapshot restore progress: id=2-15-1718116848787 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:48.796Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: removing all old logs from log store 2024-06-11T14:40:48.796Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: compacting logs: from=1 to=2 2024-06-11T14:40:48.796Z [INFO] server-70bb8dd8-fc71-384e-71ac-3a8f42f32525: Installed remote snapshot 2024-06-11T14:40:48.796Z [INFO] server-7e82277d-e1b0-a916-f5a3-fe7393a4d126: pipelining replication: peer="{Voter server-70bb8dd8-fc71-384e-71ac-3a8f42f32525 70bb8dd8-fc71-384e-71ac-3a8f42f32525}" 2024-06-11T14:40:49.801Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:48.795920019 +0000 UTC m=+55.378998931 (2 nodes), 1.56µs from start of poll, 1.650518045s from cluster start. Timeout at 2024-06-11 14:40:49.801036952 +0000 UTC m=+56.384115904, 1.005116973s after stability 2024-06-11T14:40:49.813Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: aborting pipeline replication: peer="{Voter server-fb65dd45-ee23-a04b-8e34-7e26d01c68dc fb65dd45-ee23-a04b-8e34-7e26d01c68dc}" 2024-06-11T14:40:49.813Z [INFO] server-afa586ed-2954-abdb-58ef-c69ce44c5d0e: aborting pipeline replication: peer="{Voter server-23b7678e-168d-1138-e350-37204e15f8f2 23b7678e-168d-1138-e350-37204e15f8f2}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2024-06-11T14:40:49.816Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce Address:b5117ecf-a15d-4437-b0b1-8f9f3299ebce} {Suffrage:Voter ID:server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e Address:6f8d059c-8933-eaaf-13e9-46a9fb8b276e} {Suffrage:Voter ID:server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 Address:fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7}]" 2024-06-11T14:40:49.816Z [INFO] server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce Address:b5117ecf-a15d-4437-b0b1-8f9f3299ebce} {Suffrage:Voter ID:server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e Address:6f8d059c-8933-eaaf-13e9-46a9fb8b276e} {Suffrage:Voter ID:server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 Address:fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7}]" 2024-06-11T14:40:49.816Z [INFO] server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce Address:b5117ecf-a15d-4437-b0b1-8f9f3299ebce} {Suffrage:Voter ID:server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e Address:6f8d059c-8933-eaaf-13e9-46a9fb8b276e} {Suffrage:Voter ID:server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 Address:fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7}]" 2024-06-11T14:40:49.816Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:49.816Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: entering follower state: follower="Node at b5117ecf-a15d-4437-b0b1-8f9f3299ebce [Follower]" leader-address= leader-id= 2024-06-11T14:40:49.816Z [INFO] server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e: entering follower state: follower="Node at 6f8d059c-8933-eaaf-13e9-46a9fb8b276e [Follower]" leader-address= leader-id= 2024-06-11T14:40:49.816Z [INFO] server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7: entering follower state: follower="Node at fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 [Follower]" leader-address= leader-id= 2024-06-11T14:40:50.593Z [WARN] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:50.593Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: entering candidate state: node="Node at b5117ecf-a15d-4437-b0b1-8f9f3299ebce [Candidate]" term=2 2024-06-11T14:40:50.593Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: election won: term=2 tally=2 2024-06-11T14:40:50.593Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: entering leader state: leader="Node at b5117ecf-a15d-4437-b0b1-8f9f3299ebce [Leader]" 2024-06-11T14:40:50.593Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: added peer, starting replication: peer=server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e 2024-06-11T14:40:50.594Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: added peer, starting replication: peer=server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 2024-06-11T14:40:50.594Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: pipelining replication: peer="{Voter server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7}" 2024-06-11T14:40:50.594Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: pipelining replication: peer="{Voter server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e 6f8d059c-8933-eaaf-13e9-46a9fb8b276e}" 2024-06-11T14:40:51.599Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:50.593895103 +0000 UTC m=+57.176974055 (1 nodes), 777.123719ms from start of poll, 777.879052ms from cluster start. Timeout at 2024-06-11 14:40:51.599031315 +0000 UTC m=+58.182110267, 1.005136212s after stability 2024-06-11T14:40:51.600Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: starting snapshot up to: index=12 2024-06-11T14:40:51.600Z [INFO] creating new snapshot: path=/tmp/raft4094355951/snapshots/2-12-1718116851600.tmp 2024-06-11T14:40:51.601Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: snapshot complete up to: index=12 2024-06-11T14:40:51.603Z [INFO] creating new snapshot: path=/tmp/raft4094355951/snapshots/2-113-1718116851603.tmp 2024-06-11T14:40:51.604Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: copied to local snapshot: bytes=71 2024-06-11T14:40:51.605Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: snapshot restore progress: id=2-113-1718116851603 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:51.605Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: restored user snapshot: index=113 2024-06-11T14:40:51.605Z [ERROR] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: failed to get log: index=23 error="log not found" 2024-06-11T14:40:51.605Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: aborting pipeline replication: peer="{Voter server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7}" 2024-06-11T14:40:51.605Z [ERROR] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: failed to get log: index=23 error="log not found" 2024-06-11T14:40:51.605Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: aborting pipeline replication: peer="{Voter server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e 6f8d059c-8933-eaaf-13e9-46a9fb8b276e}" 2024-06-11T14:40:51.611Z [ERROR] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: failed to get log: index=23 error="log not found" 2024-06-11T14:40:51.612Z [INFO] creating new snapshot: path=/tmp/raft3053224227/snapshots/2-113-1718116851612.tmp 2024-06-11T14:40:51.612Z [ERROR] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: failed to get log: index=23 error="log not found" 2024-06-11T14:40:51.612Z [INFO] server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:51.613Z [INFO] creating new snapshot: path=/tmp/raft3440610652/snapshots/2-113-1718116851613.tmp 2024-06-11T14:40:51.613Z [INFO] server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e: copied to local snapshot: bytes=71 2024-06-11T14:40:51.613Z [INFO] server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:51.613Z [INFO] server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e: snapshot restore progress: id=2-113-1718116851612 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:51.613Z [INFO] server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e: Installed remote snapshot 2024-06-11T14:40:51.614Z [WARN] server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e: skipping application of old log: index=22 2024-06-11T14:40:51.614Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: pipelining replication: peer="{Voter server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e 6f8d059c-8933-eaaf-13e9-46a9fb8b276e}" 2024-06-11T14:40:51.614Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:51.614Z [INFO] server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7: copied to local snapshot: bytes=71 2024-06-11T14:40:51.614Z [INFO] server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7: snapshot restore progress: id=2-113-1718116851613 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:51.615Z [INFO] server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7: Installed remote snapshot 2024-06-11T14:40:51.615Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: pipelining replication: peer="{Voter server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7}" 2024-06-11T14:40:52.619Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:51.614458599 +0000 UTC m=+58.197537551 (2 nodes), 1.56µs from start of poll, 1.798442548s from cluster start. Timeout at 2024-06-11 14:40:52.619913839 +0000 UTC m=+59.202992751, 1.0054552s after stability 2024-06-11T14:40:52.632Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: aborting pipeline replication: peer="{Voter server-fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7 fee7bdf0-ec71-0c3e-9b87-79ec23fad0c7}" 2024-06-11T14:40:52.632Z [INFO] server-b5117ecf-a15d-4437-b0b1-8f9f3299ebce: aborting pipeline replication: peer="{Voter server-6f8d059c-8933-eaaf-13e9-46a9fb8b276e 6f8d059c-8933-eaaf-13e9-46a9fb8b276e}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2024-06-11T14:40:52.635Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44575c8c-e245-c504-af6e-18fb34246872 Address:44575c8c-e245-c504-af6e-18fb34246872} {Suffrage:Voter ID:server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c Address:4eeb0dab-917f-3bf1-4b68-f6cbe27b515c} {Suffrage:Voter ID:server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 Address:17fad173-25b2-608c-0b08-6ac4c0c8ac79}]" 2024-06-11T14:40:52.636Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44575c8c-e245-c504-af6e-18fb34246872 Address:44575c8c-e245-c504-af6e-18fb34246872} {Suffrage:Voter ID:server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c Address:4eeb0dab-917f-3bf1-4b68-f6cbe27b515c} {Suffrage:Voter ID:server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 Address:17fad173-25b2-608c-0b08-6ac4c0c8ac79}]" 2024-06-11T14:40:52.636Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44575c8c-e245-c504-af6e-18fb34246872 Address:44575c8c-e245-c504-af6e-18fb34246872} {Suffrage:Voter ID:server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c Address:4eeb0dab-917f-3bf1-4b68-f6cbe27b515c} {Suffrage:Voter ID:server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 Address:17fad173-25b2-608c-0b08-6ac4c0c8ac79}]" 2024-06-11T14:40:52.636Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:52.636Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: entering follower state: follower="Node at 44575c8c-e245-c504-af6e-18fb34246872 [Follower]" leader-address= leader-id= 2024-06-11T14:40:52.636Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: entering follower state: follower="Node at 4eeb0dab-917f-3bf1-4b68-f6cbe27b515c [Follower]" leader-address= leader-id= 2024-06-11T14:40:52.636Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: entering follower state: follower="Node at 17fad173-25b2-608c-0b08-6ac4c0c8ac79 [Follower]" leader-address= leader-id= 2024-06-11T14:40:53.344Z [WARN] server-44575c8c-e245-c504-af6e-18fb34246872: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:53.345Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: entering candidate state: node="Node at 44575c8c-e245-c504-af6e-18fb34246872 [Candidate]" term=2 2024-06-11T14:40:53.345Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: election won: term=2 tally=2 2024-06-11T14:40:53.345Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: entering leader state: leader="Node at 44575c8c-e245-c504-af6e-18fb34246872 [Leader]" 2024-06-11T14:40:53.345Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: added peer, starting replication: peer=server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c 2024-06-11T14:40:53.345Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: added peer, starting replication: peer=server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 2024-06-11T14:40:53.345Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: pipelining replication: peer="{Voter server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 17fad173-25b2-608c-0b08-6ac4c0c8ac79}" 2024-06-11T14:40:53.345Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: pipelining replication: peer="{Voter server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c 4eeb0dab-917f-3bf1-4b68-f6cbe27b515c}" 2024-06-11T14:40:54.351Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:53.345953282 +0000 UTC m=+59.929032234 (1 nodes), 709.419483ms from start of poll, 710.198655ms from cluster start. Timeout at 2024-06-11 14:40:54.351495079 +0000 UTC m=+60.934573991, 1.005541757s after stability 2024-06-11T14:40:54.353Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: starting snapshot up to: index=12 2024-06-11T14:40:54.353Z [INFO] creating new snapshot: path=/tmp/raft330152379/snapshots/2-12-1718116854353.tmp 2024-06-11T14:40:54.360Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: snapshot complete up to: index=12 2024-06-11T14:40:54.362Z [INFO] creating new snapshot: path=/tmp/raft330152379/snapshots/2-113-1718116854362.tmp 2024-06-11T14:40:54.364Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: copied to local snapshot: bytes=71 2024-06-11T14:40:54.364Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: snapshot restore progress: id=2-113-1718116854362 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:54.364Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: removing all old logs from log store 2024-06-11T14:40:54.364Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: compacting logs: from=1 to=22 2024-06-11T14:40:54.364Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: restored user snapshot: index=113 2024-06-11T14:40:54.364Z [ERROR] server-44575c8c-e245-c504-af6e-18fb34246872: failed to get log: index=22 error="log not found" 2024-06-11T14:40:54.364Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: aborting pipeline replication: peer="{Voter server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 17fad173-25b2-608c-0b08-6ac4c0c8ac79}" 2024-06-11T14:40:54.364Z [ERROR] server-44575c8c-e245-c504-af6e-18fb34246872: failed to get log: index=22 error="log not found" 2024-06-11T14:40:54.364Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: aborting pipeline replication: peer="{Voter server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c 4eeb0dab-917f-3bf1-4b68-f6cbe27b515c}" 2024-06-11T14:40:54.371Z [ERROR] server-44575c8c-e245-c504-af6e-18fb34246872: failed to get log: index=22 error="log not found" 2024-06-11T14:40:54.372Z [INFO] creating new snapshot: path=/tmp/raft941716656/snapshots/2-113-1718116854372.tmp 2024-06-11T14:40:54.373Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:54.374Z [ERROR] server-44575c8c-e245-c504-af6e-18fb34246872: failed to get log: index=22 error="log not found" 2024-06-11T14:40:54.375Z [INFO] creating new snapshot: path=/tmp/raft2961470041/snapshots/2-113-1718116854375.tmp 2024-06-11T14:40:54.375Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: copied to local snapshot: bytes=71 2024-06-11T14:40:54.375Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:54.375Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: snapshot restore progress: id=2-113-1718116854372 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:54.375Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: removing all old logs from log store 2024-06-11T14:40:54.375Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: compacting logs: from=1 to=22 2024-06-11T14:40:54.375Z [INFO] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: Installed remote snapshot 2024-06-11T14:40:54.375Z [WARN] server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c: skipping application of old log: index=22 2024-06-11T14:40:54.375Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: pipelining replication: peer="{Voter server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c 4eeb0dab-917f-3bf1-4b68-f6cbe27b515c}" 2024-06-11T14:40:54.376Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:54.376Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: copied to local snapshot: bytes=71 2024-06-11T14:40:54.376Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: snapshot restore progress: id=2-113-1718116854375 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:54.376Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: removing all old logs from log store 2024-06-11T14:40:54.376Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: compacting logs: from=1 to=22 2024-06-11T14:40:54.376Z [INFO] server-17fad173-25b2-608c-0b08-6ac4c0c8ac79: Installed remote snapshot 2024-06-11T14:40:54.377Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: pipelining replication: peer="{Voter server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 17fad173-25b2-608c-0b08-6ac4c0c8ac79}" 2024-06-11T14:40:55.381Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:54.376145989 +0000 UTC m=+60.959224901 (2 nodes), 1.599µs from start of poll, 1.740391322s from cluster start. Timeout at 2024-06-11 14:40:55.381671627 +0000 UTC m=+61.964750539, 1.005525638s after stability 2024-06-11T14:40:55.393Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: aborting pipeline replication: peer="{Voter server-17fad173-25b2-608c-0b08-6ac4c0c8ac79 17fad173-25b2-608c-0b08-6ac4c0c8ac79}" 2024-06-11T14:40:55.394Z [INFO] server-44575c8c-e245-c504-af6e-18fb34246872: aborting pipeline replication: peer="{Voter server-4eeb0dab-917f-3bf1-4b68-f6cbe27b515c 4eeb0dab-917f-3bf1-4b68-f6cbe27b515c}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2024-06-11T14:40:55.397Z [INFO] server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5 Address:4787d3a0-0025-6128-ae9b-36d59b8bb5a5} {Suffrage:Voter ID:server-34f42883-1c94-5de5-e800-1033a09ddb0b Address:34f42883-1c94-5de5-e800-1033a09ddb0b} {Suffrage:Voter ID:server-4219bdb6-8789-c785-329c-5c259ea665b1 Address:4219bdb6-8789-c785-329c-5c259ea665b1}]" 2024-06-11T14:40:55.397Z [INFO] server-34f42883-1c94-5de5-e800-1033a09ddb0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5 Address:4787d3a0-0025-6128-ae9b-36d59b8bb5a5} {Suffrage:Voter ID:server-34f42883-1c94-5de5-e800-1033a09ddb0b Address:34f42883-1c94-5de5-e800-1033a09ddb0b} {Suffrage:Voter ID:server-4219bdb6-8789-c785-329c-5c259ea665b1 Address:4219bdb6-8789-c785-329c-5c259ea665b1}]" 2024-06-11T14:40:55.397Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5 Address:4787d3a0-0025-6128-ae9b-36d59b8bb5a5} {Suffrage:Voter ID:server-34f42883-1c94-5de5-e800-1033a09ddb0b Address:34f42883-1c94-5de5-e800-1033a09ddb0b} {Suffrage:Voter ID:server-4219bdb6-8789-c785-329c-5c259ea665b1 Address:4219bdb6-8789-c785-329c-5c259ea665b1}]" 2024-06-11T14:40:55.397Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:55.398Z [INFO] server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5: entering follower state: follower="Node at 4787d3a0-0025-6128-ae9b-36d59b8bb5a5 [Follower]" leader-address= leader-id= 2024-06-11T14:40:55.398Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: entering follower state: follower="Node at 4219bdb6-8789-c785-329c-5c259ea665b1 [Follower]" leader-address= leader-id= 2024-06-11T14:40:55.398Z [INFO] server-34f42883-1c94-5de5-e800-1033a09ddb0b: entering follower state: follower="Node at 34f42883-1c94-5de5-e800-1033a09ddb0b [Follower]" leader-address= leader-id= 2024-06-11T14:40:55.966Z [WARN] server-4219bdb6-8789-c785-329c-5c259ea665b1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:55.966Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: entering candidate state: node="Node at 4219bdb6-8789-c785-329c-5c259ea665b1 [Candidate]" term=2 2024-06-11T14:40:55.966Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: election won: term=2 tally=2 2024-06-11T14:40:55.966Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: entering leader state: leader="Node at 4219bdb6-8789-c785-329c-5c259ea665b1 [Leader]" 2024-06-11T14:40:55.966Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: added peer, starting replication: peer=server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5 2024-06-11T14:40:55.966Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: added peer, starting replication: peer=server-34f42883-1c94-5de5-e800-1033a09ddb0b 2024-06-11T14:40:55.967Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: pipelining replication: peer="{Voter server-34f42883-1c94-5de5-e800-1033a09ddb0b 34f42883-1c94-5de5-e800-1033a09ddb0b}" 2024-06-11T14:40:55.967Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: pipelining replication: peer="{Voter server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5 4787d3a0-0025-6128-ae9b-36d59b8bb5a5}" 2024-06-11T14:40:56.973Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:55.967616085 +0000 UTC m=+62.550694997 (1 nodes), 569.695484ms from start of poll, 570.498095ms from cluster start. Timeout at 2024-06-11 14:40:56.973120644 +0000 UTC m=+63.556199556, 1.005504559s after stability 2024-06-11T14:40:56.974Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: starting snapshot up to: index=12 2024-06-11T14:40:56.974Z [INFO] creating new snapshot: path=/tmp/raft4045796373/snapshots/2-12-1718116856974.tmp 2024-06-11T14:40:56.976Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: snapshot complete up to: index=12 2024-06-11T14:40:56.979Z [INFO] server-43ac552d-5810-d958-bb57-7f0ee27af43d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43ac552d-5810-d958-bb57-7f0ee27af43d Address:43ac552d-5810-d958-bb57-7f0ee27af43d} {Suffrage:Voter ID:server-cb305366-5568-9ec5-a06a-f3029431874d Address:cb305366-5568-9ec5-a06a-f3029431874d} {Suffrage:Voter ID:server-97b7391e-b76d-3d41-c955-27bbe03fb666 Address:97b7391e-b76d-3d41-c955-27bbe03fb666}]" 2024-06-11T14:40:56.979Z [INFO] server-cb305366-5568-9ec5-a06a-f3029431874d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43ac552d-5810-d958-bb57-7f0ee27af43d Address:43ac552d-5810-d958-bb57-7f0ee27af43d} {Suffrage:Voter ID:server-cb305366-5568-9ec5-a06a-f3029431874d Address:cb305366-5568-9ec5-a06a-f3029431874d} {Suffrage:Voter ID:server-97b7391e-b76d-3d41-c955-27bbe03fb666 Address:97b7391e-b76d-3d41-c955-27bbe03fb666}]" 2024-06-11T14:40:56.980Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43ac552d-5810-d958-bb57-7f0ee27af43d Address:43ac552d-5810-d958-bb57-7f0ee27af43d} {Suffrage:Voter ID:server-cb305366-5568-9ec5-a06a-f3029431874d Address:cb305366-5568-9ec5-a06a-f3029431874d} {Suffrage:Voter ID:server-97b7391e-b76d-3d41-c955-27bbe03fb666 Address:97b7391e-b76d-3d41-c955-27bbe03fb666}]" 2024-06-11T14:40:56.980Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:56.980Z [INFO] server-43ac552d-5810-d958-bb57-7f0ee27af43d: entering follower state: follower="Node at 43ac552d-5810-d958-bb57-7f0ee27af43d [Follower]" leader-address= leader-id= 2024-06-11T14:40:56.980Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: entering follower state: follower="Node at 97b7391e-b76d-3d41-c955-27bbe03fb666 [Follower]" leader-address= leader-id= 2024-06-11T14:40:56.980Z [INFO] server-cb305366-5568-9ec5-a06a-f3029431874d: entering follower state: follower="Node at cb305366-5568-9ec5-a06a-f3029431874d [Follower]" leader-address= leader-id= 2024-06-11T14:40:57.737Z [WARN] server-97b7391e-b76d-3d41-c955-27bbe03fb666: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:40:57.737Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: entering candidate state: node="Node at 97b7391e-b76d-3d41-c955-27bbe03fb666 [Candidate]" term=2 2024-06-11T14:40:57.737Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: election won: term=2 tally=2 2024-06-11T14:40:57.737Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: entering leader state: leader="Node at 97b7391e-b76d-3d41-c955-27bbe03fb666 [Leader]" 2024-06-11T14:40:57.737Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: added peer, starting replication: peer=server-43ac552d-5810-d958-bb57-7f0ee27af43d 2024-06-11T14:40:57.737Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: added peer, starting replication: peer=server-cb305366-5568-9ec5-a06a-f3029431874d 2024-06-11T14:40:57.737Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: pipelining replication: peer="{Voter server-cb305366-5568-9ec5-a06a-f3029431874d cb305366-5568-9ec5-a06a-f3029431874d}" 2024-06-11T14:40:57.738Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: pipelining replication: peer="{Voter server-43ac552d-5810-d958-bb57-7f0ee27af43d 43ac552d-5810-d958-bb57-7f0ee27af43d}" 2024-06-11T14:40:58.744Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:40:57.738116161 +0000 UTC m=+64.321195113 (1 nodes), 757.956771ms from start of poll, 758.572069ms from cluster start. Timeout at 2024-06-11 14:40:58.744033504 +0000 UTC m=+65.327112456, 1.005917343s after stability 2024-06-11T14:40:58.744Z [INFO] creating new snapshot: path=/tmp/raft3465012389/snapshots/2-113-1718116858744.tmp 2024-06-11T14:40:58.745Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: copied to local snapshot: bytes=71 2024-06-11T14:40:58.745Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: snapshot restore progress: id=2-113-1718116858744 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:58.746Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: restored user snapshot: index=113 2024-06-11T14:40:58.746Z [ERROR] server-97b7391e-b76d-3d41-c955-27bbe03fb666: failed to get log: index=3 error="log not found" 2024-06-11T14:40:58.746Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: aborting pipeline replication: peer="{Voter server-cb305366-5568-9ec5-a06a-f3029431874d cb305366-5568-9ec5-a06a-f3029431874d}" 2024-06-11T14:40:58.746Z [ERROR] server-97b7391e-b76d-3d41-c955-27bbe03fb666: failed to get log: index=3 error="log not found" 2024-06-11T14:40:58.746Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: aborting pipeline replication: peer="{Voter server-43ac552d-5810-d958-bb57-7f0ee27af43d 43ac552d-5810-d958-bb57-7f0ee27af43d}" 2024-06-11T14:40:58.755Z [ERROR] server-97b7391e-b76d-3d41-c955-27bbe03fb666: failed to get log: index=3 error="log not found" 2024-06-11T14:40:58.756Z [INFO] creating new snapshot: path=/tmp/raft45038791/snapshots/2-113-1718116858756.tmp 2024-06-11T14:40:58.756Z [INFO] server-cb305366-5568-9ec5-a06a-f3029431874d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:58.757Z [INFO] server-cb305366-5568-9ec5-a06a-f3029431874d: copied to local snapshot: bytes=71 2024-06-11T14:40:58.757Z [ERROR] server-97b7391e-b76d-3d41-c955-27bbe03fb666: failed to get log: index=3 error="log not found" 2024-06-11T14:40:58.758Z [INFO] server-cb305366-5568-9ec5-a06a-f3029431874d: snapshot restore progress: id=2-113-1718116858756 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:58.758Z [INFO] server-cb305366-5568-9ec5-a06a-f3029431874d: Installed remote snapshot 2024-06-11T14:40:58.758Z [INFO] creating new snapshot: path=/tmp/raft757683762/snapshots/2-113-1718116858758.tmp 2024-06-11T14:40:58.759Z [INFO] server-43ac552d-5810-d958-bb57-7f0ee27af43d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:58.759Z [INFO] server-43ac552d-5810-d958-bb57-7f0ee27af43d: copied to local snapshot: bytes=71 2024-06-11T14:40:58.760Z [INFO] server-43ac552d-5810-d958-bb57-7f0ee27af43d: snapshot restore progress: id=2-113-1718116858758 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:40:58.760Z [INFO] server-43ac552d-5810-d958-bb57-7f0ee27af43d: Installed remote snapshot 2024-06-11T14:40:58.766Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: pipelining replication: peer="{Voter server-cb305366-5568-9ec5-a06a-f3029431874d cb305366-5568-9ec5-a06a-f3029431874d}" 2024-06-11T14:40:58.766Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:40:58.770Z [INFO] server-97b7391e-b76d-3d41-c955-27bbe03fb666: pipelining replication: peer="{Voter server-43ac552d-5810-d958-bb57-7f0ee27af43d 43ac552d-5810-d958-bb57-7f0ee27af43d}" 2024-06-11T14:40:59.772Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:40:58.766934998 +0000 UTC m=+65.350013950 (2 nodes), 2.04µs from start of poll, 1.787390906s from cluster start. Timeout at 2024-06-11 14:40:59.772407238 +0000 UTC m=+66.355486150, 1.0054722s after stability 2024-06-11T14:40:59.784Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: aborting pipeline replication: peer="{Voter server-34f42883-1c94-5de5-e800-1033a09ddb0b 34f42883-1c94-5de5-e800-1033a09ddb0b}" 2024-06-11T14:40:59.785Z [INFO] server-4219bdb6-8789-c785-329c-5c259ea665b1: aborting pipeline replication: peer="{Voter server-4787d3a0-0025-6128-ae9b-36d59b8bb5a5 4787d3a0-0025-6128-ae9b-36d59b8bb5a5}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2024-06-11T14:40:59.787Z [INFO] server-909af506-6816-f400-320a-6771261e6dbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-909af506-6816-f400-320a-6771261e6dbb Address:909af506-6816-f400-320a-6771261e6dbb} {Suffrage:Voter ID:server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c Address:ae6f5f28-c917-766b-b3fb-4a1cfb9d278c} {Suffrage:Voter ID:server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4 Address:f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4}]" 2024-06-11T14:40:59.787Z [INFO] server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-909af506-6816-f400-320a-6771261e6dbb Address:909af506-6816-f400-320a-6771261e6dbb} {Suffrage:Voter ID:server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c Address:ae6f5f28-c917-766b-b3fb-4a1cfb9d278c} {Suffrage:Voter ID:server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4 Address:f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4}]" 2024-06-11T14:40:59.787Z [INFO] server-909af506-6816-f400-320a-6771261e6dbb: entering follower state: follower="Node at 909af506-6816-f400-320a-6771261e6dbb [Follower]" leader-address= leader-id= 2024-06-11T14:40:59.788Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-909af506-6816-f400-320a-6771261e6dbb Address:909af506-6816-f400-320a-6771261e6dbb} {Suffrage:Voter ID:server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c Address:ae6f5f28-c917-766b-b3fb-4a1cfb9d278c} {Suffrage:Voter ID:server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4 Address:f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4}]" 2024-06-11T14:40:59.788Z [INFO] server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c: entering follower state: follower="Node at ae6f5f28-c917-766b-b3fb-4a1cfb9d278c [Follower]" leader-address= leader-id= 2024-06-11T14:40:59.788Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:40:59.788Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: entering follower state: follower="Node at f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4 [Follower]" leader-address= leader-id= 2024-06-11T14:41:00.373Z [WARN] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:00.373Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: entering candidate state: node="Node at f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4 [Candidate]" term=2 2024-06-11T14:41:00.373Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: election won: term=2 tally=2 2024-06-11T14:41:00.373Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: entering leader state: leader="Node at f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4 [Leader]" 2024-06-11T14:41:00.373Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: added peer, starting replication: peer=server-909af506-6816-f400-320a-6771261e6dbb 2024-06-11T14:41:00.373Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: added peer, starting replication: peer=server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c 2024-06-11T14:41:00.373Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: pipelining replication: peer="{Voter server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c ae6f5f28-c917-766b-b3fb-4a1cfb9d278c}" 2024-06-11T14:41:00.373Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: pipelining replication: peer="{Voter server-909af506-6816-f400-320a-6771261e6dbb 909af506-6816-f400-320a-6771261e6dbb}" 2024-06-11T14:41:01.379Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:00.374098967 +0000 UTC m=+66.957177919 (1 nodes), 585.995495ms from start of poll, 586.591754ms from cluster start. Timeout at 2024-06-11 14:41:01.379312497 +0000 UTC m=+67.962391409, 1.00521349s after stability 2024-06-11T14:41:01.380Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: starting snapshot up to: index=12 2024-06-11T14:41:01.380Z [INFO] creating new snapshot: path=/tmp/raft3805050798/snapshots/2-12-1718116861380.tmp 2024-06-11T14:41:01.381Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: snapshot complete up to: index=12 2024-06-11T14:41:01.384Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f066873-1c8d-3a73-659e-ae41c4163f20 Address:8f066873-1c8d-3a73-659e-ae41c4163f20} {Suffrage:Voter ID:server-841c4003-2f37-3ae1-1521-7bfab5dc025f Address:841c4003-2f37-3ae1-1521-7bfab5dc025f} {Suffrage:Voter ID:server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd Address:7e16fce4-5f41-8c40-c67d-a7fc08c479fd}]" 2024-06-11T14:41:01.384Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f066873-1c8d-3a73-659e-ae41c4163f20 Address:8f066873-1c8d-3a73-659e-ae41c4163f20} {Suffrage:Voter ID:server-841c4003-2f37-3ae1-1521-7bfab5dc025f Address:841c4003-2f37-3ae1-1521-7bfab5dc025f} {Suffrage:Voter ID:server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd Address:7e16fce4-5f41-8c40-c67d-a7fc08c479fd}]" 2024-06-11T14:41:01.384Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: entering follower state: follower="Node at 8f066873-1c8d-3a73-659e-ae41c4163f20 [Follower]" leader-address= leader-id= 2024-06-11T14:41:01.384Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: entering follower state: follower="Node at 841c4003-2f37-3ae1-1521-7bfab5dc025f [Follower]" leader-address= leader-id= 2024-06-11T14:41:01.385Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f066873-1c8d-3a73-659e-ae41c4163f20 Address:8f066873-1c8d-3a73-659e-ae41c4163f20} {Suffrage:Voter ID:server-841c4003-2f37-3ae1-1521-7bfab5dc025f Address:841c4003-2f37-3ae1-1521-7bfab5dc025f} {Suffrage:Voter ID:server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd Address:7e16fce4-5f41-8c40-c67d-a7fc08c479fd}]" 2024-06-11T14:41:01.385Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:01.385Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: entering follower state: follower="Node at 7e16fce4-5f41-8c40-c67d-a7fc08c479fd [Follower]" leader-address= leader-id= 2024-06-11T14:41:01.951Z [WARN] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:01.951Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: entering candidate state: node="Node at 841c4003-2f37-3ae1-1521-7bfab5dc025f [Candidate]" term=2 2024-06-11T14:41:01.951Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: election won: term=2 tally=2 2024-06-11T14:41:01.951Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: entering leader state: leader="Node at 841c4003-2f37-3ae1-1521-7bfab5dc025f [Leader]" 2024-06-11T14:41:01.951Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: added peer, starting replication: peer=server-8f066873-1c8d-3a73-659e-ae41c4163f20 2024-06-11T14:41:01.951Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: added peer, starting replication: peer=server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd 2024-06-11T14:41:01.951Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: pipelining replication: peer="{Voter server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd 7e16fce4-5f41-8c40-c67d-a7fc08c479fd}" 2024-06-11T14:41:01.952Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: pipelining replication: peer="{Voter server-8f066873-1c8d-3a73-659e-ae41c4163f20 8f066873-1c8d-3a73-659e-ae41c4163f20}" 2024-06-11T14:41:02.958Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:01.952242784 +0000 UTC m=+68.535321736 (1 nodes), 567.135217ms from start of poll, 567.775473ms from cluster start. Timeout at 2024-06-11 14:41:02.958060931 +0000 UTC m=+69.541139883, 1.005818147s after stability 2024-06-11T14:41:02.958Z [INFO] creating new snapshot: path=/tmp/raft2592708250/snapshots/2-113-1718116862958.tmp 2024-06-11T14:41:02.967Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: copied to local snapshot: bytes=71 2024-06-11T14:41:02.967Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: snapshot restore progress: id=2-113-1718116862958 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:02.967Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: removing all old logs from log store 2024-06-11T14:41:02.967Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: compacting logs: from=1 to=2 2024-06-11T14:41:02.967Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: restored user snapshot: index=113 2024-06-11T14:41:02.967Z [ERROR] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: failed to get log: index=2 error="log not found" 2024-06-11T14:41:02.967Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: aborting pipeline replication: peer="{Voter server-8f066873-1c8d-3a73-659e-ae41c4163f20 8f066873-1c8d-3a73-659e-ae41c4163f20}" 2024-06-11T14:41:02.967Z [ERROR] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: failed to get log: index=2 error="log not found" 2024-06-11T14:41:02.967Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: aborting pipeline replication: peer="{Voter server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd 7e16fce4-5f41-8c40-c67d-a7fc08c479fd}" 2024-06-11T14:41:02.977Z [ERROR] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: failed to get log: index=2 error="log not found" 2024-06-11T14:41:02.977Z [ERROR] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: failed to get log: index=2 error="log not found" 2024-06-11T14:41:02.977Z [INFO] creating new snapshot: path=/tmp/raft711220052/snapshots/2-113-1718116862977.tmp 2024-06-11T14:41:02.977Z [INFO] creating new snapshot: path=/tmp/raft595238825/snapshots/2-113-1718116862977.tmp 2024-06-11T14:41:02.978Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:02.978Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:02.979Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: copied to local snapshot: bytes=71 2024-06-11T14:41:02.979Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: copied to local snapshot: bytes=71 2024-06-11T14:41:02.979Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: snapshot restore progress: id=2-113-1718116862977 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:02.979Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: removing all old logs from log store 2024-06-11T14:41:02.979Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: compacting logs: from=1 to=2 2024-06-11T14:41:02.979Z [INFO] server-8f066873-1c8d-3a73-659e-ae41c4163f20: Installed remote snapshot 2024-06-11T14:41:02.979Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: snapshot restore progress: id=2-113-1718116862977 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:02.979Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: removing all old logs from log store 2024-06-11T14:41:02.979Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: compacting logs: from=1 to=2 2024-06-11T14:41:02.979Z [INFO] server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd: Installed remote snapshot 2024-06-11T14:41:02.980Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: pipelining replication: peer="{Voter server-7e16fce4-5f41-8c40-c67d-a7fc08c479fd 7e16fce4-5f41-8c40-c67d-a7fc08c479fd}" 2024-06-11T14:41:02.980Z [INFO] server-841c4003-2f37-3ae1-1521-7bfab5dc025f: pipelining replication: peer="{Voter server-8f066873-1c8d-3a73-659e-ae41c4163f20 8f066873-1c8d-3a73-659e-ae41c4163f20}" 2024-06-11T14:41:02.980Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:03.985Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:02.980171373 +0000 UTC m=+69.563250285 (2 nodes), 1.52µs from start of poll, 1.595704022s from cluster start. Timeout at 2024-06-11 14:41:03.98518455 +0000 UTC m=+70.568263462, 1.005013177s after stability 2024-06-11T14:41:03.997Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: aborting pipeline replication: peer="{Voter server-909af506-6816-f400-320a-6771261e6dbb 909af506-6816-f400-320a-6771261e6dbb}" 2024-06-11T14:41:03.997Z [INFO] server-f20ba2cc-5fd3-c42e-bec7-b1f153ef6dc4: aborting pipeline replication: peer="{Voter server-ae6f5f28-c917-766b-b3fb-4a1cfb9d278c ae6f5f28-c917-766b-b3fb-4a1cfb9d278c}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2024-06-11T14:41:04.000Z [INFO] server-2ec15c2b-bc25-d170-f57e-09609873a3a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ec15c2b-bc25-d170-f57e-09609873a3a6 Address:2ec15c2b-bc25-d170-f57e-09609873a3a6} {Suffrage:Voter ID:server-1689d0f1-2b36-e7e2-228d-b8418db2976b Address:1689d0f1-2b36-e7e2-228d-b8418db2976b} {Suffrage:Voter ID:server-587f37d8-4190-ed18-ccff-81b6539c4e5c Address:587f37d8-4190-ed18-ccff-81b6539c4e5c}]" 2024-06-11T14:41:04.000Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ec15c2b-bc25-d170-f57e-09609873a3a6 Address:2ec15c2b-bc25-d170-f57e-09609873a3a6} {Suffrage:Voter ID:server-1689d0f1-2b36-e7e2-228d-b8418db2976b Address:1689d0f1-2b36-e7e2-228d-b8418db2976b} {Suffrage:Voter ID:server-587f37d8-4190-ed18-ccff-81b6539c4e5c Address:587f37d8-4190-ed18-ccff-81b6539c4e5c}]" 2024-06-11T14:41:04.000Z [INFO] server-587f37d8-4190-ed18-ccff-81b6539c4e5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ec15c2b-bc25-d170-f57e-09609873a3a6 Address:2ec15c2b-bc25-d170-f57e-09609873a3a6} {Suffrage:Voter ID:server-1689d0f1-2b36-e7e2-228d-b8418db2976b Address:1689d0f1-2b36-e7e2-228d-b8418db2976b} {Suffrage:Voter ID:server-587f37d8-4190-ed18-ccff-81b6539c4e5c Address:587f37d8-4190-ed18-ccff-81b6539c4e5c}]" 2024-06-11T14:41:04.000Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:04.000Z [INFO] server-2ec15c2b-bc25-d170-f57e-09609873a3a6: entering follower state: follower="Node at 2ec15c2b-bc25-d170-f57e-09609873a3a6 [Follower]" leader-address= leader-id= 2024-06-11T14:41:04.000Z [INFO] server-587f37d8-4190-ed18-ccff-81b6539c4e5c: entering follower state: follower="Node at 587f37d8-4190-ed18-ccff-81b6539c4e5c [Follower]" leader-address= leader-id= 2024-06-11T14:41:04.000Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: entering follower state: follower="Node at 1689d0f1-2b36-e7e2-228d-b8418db2976b [Follower]" leader-address= leader-id= 2024-06-11T14:41:04.628Z [WARN] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:04.628Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: entering candidate state: node="Node at 1689d0f1-2b36-e7e2-228d-b8418db2976b [Candidate]" term=2 2024-06-11T14:41:04.628Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: election won: term=2 tally=2 2024-06-11T14:41:04.628Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: entering leader state: leader="Node at 1689d0f1-2b36-e7e2-228d-b8418db2976b [Leader]" 2024-06-11T14:41:04.629Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: added peer, starting replication: peer=server-2ec15c2b-bc25-d170-f57e-09609873a3a6 2024-06-11T14:41:04.629Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: added peer, starting replication: peer=server-587f37d8-4190-ed18-ccff-81b6539c4e5c 2024-06-11T14:41:04.629Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: pipelining replication: peer="{Voter server-587f37d8-4190-ed18-ccff-81b6539c4e5c 587f37d8-4190-ed18-ccff-81b6539c4e5c}" 2024-06-11T14:41:04.629Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: pipelining replication: peer="{Voter server-2ec15c2b-bc25-d170-f57e-09609873a3a6 2ec15c2b-bc25-d170-f57e-09609873a3a6}" 2024-06-11T14:41:05.634Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:04.629031838 +0000 UTC m=+71.212110790 (1 nodes), 628.357647ms from start of poll, 628.948106ms from cluster start. Timeout at 2024-06-11 14:41:05.634389602 +0000 UTC m=+72.217468514, 1.005357724s after stability 2024-06-11T14:41:05.635Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: starting snapshot up to: index=12 2024-06-11T14:41:05.635Z [INFO] creating new snapshot: path=/tmp/raft346722078/snapshots/2-12-1718116865635.tmp 2024-06-11T14:41:05.636Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: snapshot complete up to: index=12 2024-06-11T14:41:05.638Z [INFO] creating new snapshot: path=/tmp/raft346722078/snapshots/2-1013-1718116865638.tmp 2024-06-11T14:41:05.640Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: copied to local snapshot: bytes=71 2024-06-11T14:41:05.646Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: snapshot restore progress: id=2-1013-1718116865638 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:05.646Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: restored user snapshot: index=1013 2024-06-11T14:41:05.646Z [ERROR] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: failed to get log: index=23 error="log not found" 2024-06-11T14:41:05.646Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: aborting pipeline replication: peer="{Voter server-587f37d8-4190-ed18-ccff-81b6539c4e5c 587f37d8-4190-ed18-ccff-81b6539c4e5c}" 2024-06-11T14:41:05.646Z [ERROR] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: failed to get log: index=23 error="log not found" 2024-06-11T14:41:05.646Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: aborting pipeline replication: peer="{Voter server-2ec15c2b-bc25-d170-f57e-09609873a3a6 2ec15c2b-bc25-d170-f57e-09609873a3a6}" 2024-06-11T14:41:05.652Z [ERROR] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: failed to get log: index=23 error="log not found" 2024-06-11T14:41:05.653Z [INFO] creating new snapshot: path=/tmp/raft4607523/snapshots/2-1013-1718116865653.tmp 2024-06-11T14:41:05.654Z [INFO] server-587f37d8-4190-ed18-ccff-81b6539c4e5c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:05.654Z [INFO] server-587f37d8-4190-ed18-ccff-81b6539c4e5c: copied to local snapshot: bytes=71 2024-06-11T14:41:05.654Z [ERROR] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: failed to get log: index=23 error="log not found" 2024-06-11T14:41:05.655Z [INFO] server-587f37d8-4190-ed18-ccff-81b6539c4e5c: snapshot restore progress: id=2-1013-1718116865653 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:05.655Z [INFO] server-587f37d8-4190-ed18-ccff-81b6539c4e5c: Installed remote snapshot 2024-06-11T14:41:05.656Z [WARN] server-587f37d8-4190-ed18-ccff-81b6539c4e5c: skipping application of old log: index=22 2024-06-11T14:41:05.656Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: pipelining replication: peer="{Voter server-587f37d8-4190-ed18-ccff-81b6539c4e5c 587f37d8-4190-ed18-ccff-81b6539c4e5c}" 2024-06-11T14:41:05.656Z [INFO] creating new snapshot: path=/tmp/raft3256178186/snapshots/2-1013-1718116865656.tmp 2024-06-11T14:41:05.656Z [INFO] server-2ec15c2b-bc25-d170-f57e-09609873a3a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:05.657Z [INFO] server-2ec15c2b-bc25-d170-f57e-09609873a3a6: copied to local snapshot: bytes=71 2024-06-11T14:41:05.657Z [INFO] server-2ec15c2b-bc25-d170-f57e-09609873a3a6: snapshot restore progress: id=2-1013-1718116865656 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:05.657Z [INFO] server-2ec15c2b-bc25-d170-f57e-09609873a3a6: Installed remote snapshot 2024-06-11T14:41:05.658Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: pipelining replication: peer="{Voter server-2ec15c2b-bc25-d170-f57e-09609873a3a6 2ec15c2b-bc25-d170-f57e-09609873a3a6}" 2024-06-11T14:41:05.658Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:06.664Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:05.658971995 +0000 UTC m=+72.242050947 (2 nodes), 1.8µs from start of poll, 1.658888263s from cluster start. Timeout at 2024-06-11 14:41:06.664469434 +0000 UTC m=+73.247548346, 1.005497399s after stability 2024-06-11T14:41:06.678Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: aborting pipeline replication: peer="{Voter server-587f37d8-4190-ed18-ccff-81b6539c4e5c 587f37d8-4190-ed18-ccff-81b6539c4e5c}" 2024-06-11T14:41:06.678Z [INFO] server-1689d0f1-2b36-e7e2-228d-b8418db2976b: aborting pipeline replication: peer="{Voter server-2ec15c2b-bc25-d170-f57e-09609873a3a6 2ec15c2b-bc25-d170-f57e-09609873a3a6}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2024-06-11T14:41:06.681Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31a9022a-e608-3abc-9df8-79b328600214 Address:31a9022a-e608-3abc-9df8-79b328600214} {Suffrage:Voter ID:server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47 Address:0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47} {Suffrage:Voter ID:server-69b2bb28-828a-849f-54e5-dea5ac8b6560 Address:69b2bb28-828a-849f-54e5-dea5ac8b6560}]" 2024-06-11T14:41:06.681Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31a9022a-e608-3abc-9df8-79b328600214 Address:31a9022a-e608-3abc-9df8-79b328600214} {Suffrage:Voter ID:server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47 Address:0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47} {Suffrage:Voter ID:server-69b2bb28-828a-849f-54e5-dea5ac8b6560 Address:69b2bb28-828a-849f-54e5-dea5ac8b6560}]" 2024-06-11T14:41:06.682Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31a9022a-e608-3abc-9df8-79b328600214 Address:31a9022a-e608-3abc-9df8-79b328600214} {Suffrage:Voter ID:server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47 Address:0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47} {Suffrage:Voter ID:server-69b2bb28-828a-849f-54e5-dea5ac8b6560 Address:69b2bb28-828a-849f-54e5-dea5ac8b6560}]" 2024-06-11T14:41:06.682Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:06.682Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: entering follower state: follower="Node at 69b2bb28-828a-849f-54e5-dea5ac8b6560 [Follower]" leader-address= leader-id= 2024-06-11T14:41:06.682Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: entering follower state: follower="Node at 0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47 [Follower]" leader-address= leader-id= 2024-06-11T14:41:06.682Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: entering follower state: follower="Node at 31a9022a-e608-3abc-9df8-79b328600214 [Follower]" leader-address= leader-id= 2024-06-11T14:41:07.387Z [WARN] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:07.387Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: entering candidate state: node="Node at 0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47 [Candidate]" term=2 2024-06-11T14:41:07.387Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: election won: term=2 tally=2 2024-06-11T14:41:07.387Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: entering leader state: leader="Node at 0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47 [Leader]" 2024-06-11T14:41:07.387Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: added peer, starting replication: peer=server-31a9022a-e608-3abc-9df8-79b328600214 2024-06-11T14:41:07.388Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: added peer, starting replication: peer=server-69b2bb28-828a-849f-54e5-dea5ac8b6560 2024-06-11T14:41:07.388Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: pipelining replication: peer="{Voter server-69b2bb28-828a-849f-54e5-dea5ac8b6560 69b2bb28-828a-849f-54e5-dea5ac8b6560}" 2024-06-11T14:41:07.388Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: pipelining replication: peer="{Voter server-31a9022a-e608-3abc-9df8-79b328600214 31a9022a-e608-3abc-9df8-79b328600214}" 2024-06-11T14:41:08.394Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:07.388525188 +0000 UTC m=+73.971604140 (1 nodes), 706.409631ms from start of poll, 707.088606ms from cluster start. Timeout at 2024-06-11 14:41:08.394458132 +0000 UTC m=+74.977537084, 1.005932944s after stability 2024-06-11T14:41:08.396Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: starting snapshot up to: index=12 2024-06-11T14:41:08.396Z [INFO] creating new snapshot: path=/tmp/raft3501835152/snapshots/2-12-1718116868396.tmp 2024-06-11T14:41:08.397Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: snapshot complete up to: index=12 2024-06-11T14:41:08.399Z [INFO] creating new snapshot: path=/tmp/raft3501835152/snapshots/2-1013-1718116868399.tmp 2024-06-11T14:41:08.400Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: copied to local snapshot: bytes=71 2024-06-11T14:41:08.401Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: snapshot restore progress: id=2-1013-1718116868399 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:08.401Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: removing all old logs from log store 2024-06-11T14:41:08.401Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: compacting logs: from=1 to=22 2024-06-11T14:41:08.401Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: restored user snapshot: index=1013 2024-06-11T14:41:08.401Z [ERROR] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: failed to get log: index=22 error="log not found" 2024-06-11T14:41:08.401Z [ERROR] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: failed to get log: index=22 error="log not found" 2024-06-11T14:41:08.401Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: aborting pipeline replication: peer="{Voter server-31a9022a-e608-3abc-9df8-79b328600214 31a9022a-e608-3abc-9df8-79b328600214}" 2024-06-11T14:41:08.401Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: aborting pipeline replication: peer="{Voter server-69b2bb28-828a-849f-54e5-dea5ac8b6560 69b2bb28-828a-849f-54e5-dea5ac8b6560}" 2024-06-11T14:41:08.408Z [ERROR] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: failed to get log: index=22 error="log not found" 2024-06-11T14:41:08.409Z [INFO] creating new snapshot: path=/tmp/raft2782631057/snapshots/2-1013-1718116868409.tmp 2024-06-11T14:41:08.409Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:08.410Z [ERROR] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: failed to get log: index=16 error="log not found" 2024-06-11T14:41:08.410Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: copied to local snapshot: bytes=71 2024-06-11T14:41:08.410Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: snapshot restore progress: id=2-1013-1718116868409 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:08.411Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: removing all old logs from log store 2024-06-11T14:41:08.411Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: compacting logs: from=1 to=22 2024-06-11T14:41:08.411Z [INFO] creating new snapshot: path=/tmp/raft813347176/snapshots/2-1013-1718116868411.tmp 2024-06-11T14:41:08.411Z [INFO] server-31a9022a-e608-3abc-9df8-79b328600214: Installed remote snapshot 2024-06-11T14:41:08.411Z [WARN] server-31a9022a-e608-3abc-9df8-79b328600214: skipping application of old log: index=22 2024-06-11T14:41:08.411Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: pipelining replication: peer="{Voter server-31a9022a-e608-3abc-9df8-79b328600214 31a9022a-e608-3abc-9df8-79b328600214}" 2024-06-11T14:41:08.411Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:08.411Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:08.412Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: copied to local snapshot: bytes=71 2024-06-11T14:41:08.412Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: snapshot restore progress: id=2-1013-1718116868411 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:08.412Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: removing all old logs from log store 2024-06-11T14:41:08.412Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: compacting logs: from=1 to=22 2024-06-11T14:41:08.412Z [INFO] server-69b2bb28-828a-849f-54e5-dea5ac8b6560: Installed remote snapshot 2024-06-11T14:41:08.412Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: pipelining replication: peer="{Voter server-69b2bb28-828a-849f-54e5-dea5ac8b6560 69b2bb28-828a-849f-54e5-dea5ac8b6560}" 2024-06-11T14:41:09.416Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:08.411462358 +0000 UTC m=+74.994541310 (2 nodes), 1.52µs from start of poll, 1.730025776s from cluster start. Timeout at 2024-06-11 14:41:09.416651248 +0000 UTC m=+75.999730200, 1.00518889s after stability 2024-06-11T14:41:09.429Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: aborting pipeline replication: peer="{Voter server-69b2bb28-828a-849f-54e5-dea5ac8b6560 69b2bb28-828a-849f-54e5-dea5ac8b6560}" 2024-06-11T14:41:09.429Z [INFO] server-0e54a4f3-ca37-c24e-2cbe-36a6f2d85c47: aborting pipeline replication: peer="{Voter server-31a9022a-e608-3abc-9df8-79b328600214 31a9022a-e608-3abc-9df8-79b328600214}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2024-06-11T14:41:09.432Z [INFO] server-46288b31-3d8c-824f-acb7-161eb6c62fe7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46288b31-3d8c-824f-acb7-161eb6c62fe7 Address:46288b31-3d8c-824f-acb7-161eb6c62fe7} {Suffrage:Voter ID:server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8 Address:0a9f4f59-3588-ca19-2178-2383cf2f7fa8} {Suffrage:Voter ID:server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4 Address:59680fea-eadc-8a6d-bb99-a59a7e4e1aa4}]" 2024-06-11T14:41:09.432Z [INFO] server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46288b31-3d8c-824f-acb7-161eb6c62fe7 Address:46288b31-3d8c-824f-acb7-161eb6c62fe7} {Suffrage:Voter ID:server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8 Address:0a9f4f59-3588-ca19-2178-2383cf2f7fa8} {Suffrage:Voter ID:server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4 Address:59680fea-eadc-8a6d-bb99-a59a7e4e1aa4}]" 2024-06-11T14:41:09.432Z [INFO] server-46288b31-3d8c-824f-acb7-161eb6c62fe7: entering follower state: follower="Node at 46288b31-3d8c-824f-acb7-161eb6c62fe7 [Follower]" leader-address= leader-id= 2024-06-11T14:41:09.433Z [INFO] server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8: entering follower state: follower="Node at 0a9f4f59-3588-ca19-2178-2383cf2f7fa8 [Follower]" leader-address= leader-id= 2024-06-11T14:41:09.433Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46288b31-3d8c-824f-acb7-161eb6c62fe7 Address:46288b31-3d8c-824f-acb7-161eb6c62fe7} {Suffrage:Voter ID:server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8 Address:0a9f4f59-3588-ca19-2178-2383cf2f7fa8} {Suffrage:Voter ID:server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4 Address:59680fea-eadc-8a6d-bb99-a59a7e4e1aa4}]" 2024-06-11T14:41:09.433Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:09.433Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: entering follower state: follower="Node at 59680fea-eadc-8a6d-bb99-a59a7e4e1aa4 [Follower]" leader-address= leader-id= 2024-06-11T14:41:09.996Z [WARN] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:09.996Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: entering candidate state: node="Node at 59680fea-eadc-8a6d-bb99-a59a7e4e1aa4 [Candidate]" term=2 2024-06-11T14:41:09.996Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: election won: term=2 tally=2 2024-06-11T14:41:09.996Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: entering leader state: leader="Node at 59680fea-eadc-8a6d-bb99-a59a7e4e1aa4 [Leader]" 2024-06-11T14:41:09.996Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: added peer, starting replication: peer=server-46288b31-3d8c-824f-acb7-161eb6c62fe7 2024-06-11T14:41:09.996Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: added peer, starting replication: peer=server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8 2024-06-11T14:41:09.996Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: pipelining replication: peer="{Voter server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8 0a9f4f59-3588-ca19-2178-2383cf2f7fa8}" 2024-06-11T14:41:09.997Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: pipelining replication: peer="{Voter server-46288b31-3d8c-824f-acb7-161eb6c62fe7 46288b31-3d8c-824f-acb7-161eb6c62fe7}" 2024-06-11T14:41:11.003Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:09.997174461 +0000 UTC m=+76.580253413 (1 nodes), 563.96593ms from start of poll, 564.581828ms from cluster start. Timeout at 2024-06-11 14:41:11.002899892 +0000 UTC m=+77.585978844, 1.005725431s after stability 2024-06-11T14:41:11.004Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: starting snapshot up to: index=12 2024-06-11T14:41:11.004Z [INFO] creating new snapshot: path=/tmp/raft2065851689/snapshots/2-12-1718116871004.tmp 2024-06-11T14:41:11.005Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: snapshot complete up to: index=12 2024-06-11T14:41:11.008Z [INFO] server-655fd8fa-6a12-9624-371b-6b17c6455909: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-655fd8fa-6a12-9624-371b-6b17c6455909 Address:655fd8fa-6a12-9624-371b-6b17c6455909} {Suffrage:Voter ID:server-0a636497-62f0-b64d-ae84-80abe36f6305 Address:0a636497-62f0-b64d-ae84-80abe36f6305} {Suffrage:Voter ID:server-97510baa-fda0-b417-3376-0d6a9bad45c4 Address:97510baa-fda0-b417-3376-0d6a9bad45c4}]" 2024-06-11T14:41:11.008Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-655fd8fa-6a12-9624-371b-6b17c6455909 Address:655fd8fa-6a12-9624-371b-6b17c6455909} {Suffrage:Voter ID:server-0a636497-62f0-b64d-ae84-80abe36f6305 Address:0a636497-62f0-b64d-ae84-80abe36f6305} {Suffrage:Voter ID:server-97510baa-fda0-b417-3376-0d6a9bad45c4 Address:97510baa-fda0-b417-3376-0d6a9bad45c4}]" 2024-06-11T14:41:11.008Z [INFO] server-97510baa-fda0-b417-3376-0d6a9bad45c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-655fd8fa-6a12-9624-371b-6b17c6455909 Address:655fd8fa-6a12-9624-371b-6b17c6455909} {Suffrage:Voter ID:server-0a636497-62f0-b64d-ae84-80abe36f6305 Address:0a636497-62f0-b64d-ae84-80abe36f6305} {Suffrage:Voter ID:server-97510baa-fda0-b417-3376-0d6a9bad45c4 Address:97510baa-fda0-b417-3376-0d6a9bad45c4}]" 2024-06-11T14:41:11.008Z [INFO] server-655fd8fa-6a12-9624-371b-6b17c6455909: entering follower state: follower="Node at 655fd8fa-6a12-9624-371b-6b17c6455909 [Follower]" leader-address= leader-id= 2024-06-11T14:41:11.008Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:11.008Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: entering follower state: follower="Node at 0a636497-62f0-b64d-ae84-80abe36f6305 [Follower]" leader-address= leader-id= 2024-06-11T14:41:11.008Z [INFO] server-97510baa-fda0-b417-3376-0d6a9bad45c4: entering follower state: follower="Node at 97510baa-fda0-b417-3376-0d6a9bad45c4 [Follower]" leader-address= leader-id= 2024-06-11T14:41:11.533Z [WARN] server-0a636497-62f0-b64d-ae84-80abe36f6305: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:11.533Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: entering candidate state: node="Node at 0a636497-62f0-b64d-ae84-80abe36f6305 [Candidate]" term=2 2024-06-11T14:41:11.533Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: election won: term=2 tally=2 2024-06-11T14:41:11.533Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: entering leader state: leader="Node at 0a636497-62f0-b64d-ae84-80abe36f6305 [Leader]" 2024-06-11T14:41:11.533Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: added peer, starting replication: peer=server-655fd8fa-6a12-9624-371b-6b17c6455909 2024-06-11T14:41:11.533Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: added peer, starting replication: peer=server-97510baa-fda0-b417-3376-0d6a9bad45c4 2024-06-11T14:41:11.534Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: pipelining replication: peer="{Voter server-655fd8fa-6a12-9624-371b-6b17c6455909 655fd8fa-6a12-9624-371b-6b17c6455909}" 2024-06-11T14:41:11.534Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: pipelining replication: peer="{Voter server-97510baa-fda0-b417-3376-0d6a9bad45c4 97510baa-fda0-b417-3376-0d6a9bad45c4}" 2024-06-11T14:41:12.539Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:11.533832855 +0000 UTC m=+78.116911767 (1 nodes), 525.327685ms from start of poll, 525.995581ms from cluster start. Timeout at 2024-06-11 14:41:12.539332134 +0000 UTC m=+79.122411086, 1.005499319s after stability 2024-06-11T14:41:12.539Z [INFO] creating new snapshot: path=/tmp/raft103232614/snapshots/2-1013-1718116872539.tmp 2024-06-11T14:41:12.540Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: copied to local snapshot: bytes=71 2024-06-11T14:41:12.541Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: snapshot restore progress: id=2-1013-1718116872539 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:12.541Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: restored user snapshot: index=1013 2024-06-11T14:41:12.541Z [ERROR] server-0a636497-62f0-b64d-ae84-80abe36f6305: failed to get log: index=3 error="log not found" 2024-06-11T14:41:12.541Z [ERROR] server-0a636497-62f0-b64d-ae84-80abe36f6305: failed to get log: index=3 error="log not found" 2024-06-11T14:41:12.541Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: aborting pipeline replication: peer="{Voter server-655fd8fa-6a12-9624-371b-6b17c6455909 655fd8fa-6a12-9624-371b-6b17c6455909}" 2024-06-11T14:41:12.541Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: aborting pipeline replication: peer="{Voter server-97510baa-fda0-b417-3376-0d6a9bad45c4 97510baa-fda0-b417-3376-0d6a9bad45c4}" 2024-06-11T14:41:12.547Z [ERROR] server-0a636497-62f0-b64d-ae84-80abe36f6305: failed to get log: index=3 error="log not found" 2024-06-11T14:41:12.547Z [INFO] creating new snapshot: path=/tmp/raft3988144814/snapshots/2-1013-1718116872547.tmp 2024-06-11T14:41:12.548Z [INFO] server-655fd8fa-6a12-9624-371b-6b17c6455909: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:12.548Z [INFO] server-655fd8fa-6a12-9624-371b-6b17c6455909: copied to local snapshot: bytes=71 2024-06-11T14:41:12.549Z [INFO] server-655fd8fa-6a12-9624-371b-6b17c6455909: snapshot restore progress: id=2-1013-1718116872547 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:12.549Z [INFO] server-655fd8fa-6a12-9624-371b-6b17c6455909: Installed remote snapshot 2024-06-11T14:41:12.549Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: pipelining replication: peer="{Voter server-655fd8fa-6a12-9624-371b-6b17c6455909 655fd8fa-6a12-9624-371b-6b17c6455909}" 2024-06-11T14:41:12.549Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:12.552Z [ERROR] server-0a636497-62f0-b64d-ae84-80abe36f6305: failed to get log: index=3 error="log not found" 2024-06-11T14:41:12.552Z [INFO] creating new snapshot: path=/tmp/raft3276039723/snapshots/2-1013-1718116872552.tmp 2024-06-11T14:41:12.553Z [INFO] server-97510baa-fda0-b417-3376-0d6a9bad45c4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:12.554Z [INFO] server-97510baa-fda0-b417-3376-0d6a9bad45c4: copied to local snapshot: bytes=71 2024-06-11T14:41:12.554Z [INFO] server-97510baa-fda0-b417-3376-0d6a9bad45c4: snapshot restore progress: id=2-1013-1718116872552 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:12.554Z [INFO] server-97510baa-fda0-b417-3376-0d6a9bad45c4: Installed remote snapshot 2024-06-11T14:41:12.555Z [INFO] server-0a636497-62f0-b64d-ae84-80abe36f6305: pipelining replication: peer="{Voter server-97510baa-fda0-b417-3376-0d6a9bad45c4 97510baa-fda0-b417-3376-0d6a9bad45c4}" 2024-06-11T14:41:13.555Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:12.549895593 +0000 UTC m=+79.132974545 (2 nodes), 1.68µs from start of poll, 1.542058359s from cluster start. Timeout at 2024-06-11 14:41:13.555819776 +0000 UTC m=+80.138898728, 1.005924183s after stability 2024-06-11T14:41:13.573Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: aborting pipeline replication: peer="{Voter server-0a9f4f59-3588-ca19-2178-2383cf2f7fa8 0a9f4f59-3588-ca19-2178-2383cf2f7fa8}" 2024-06-11T14:41:13.573Z [INFO] server-59680fea-eadc-8a6d-bb99-a59a7e4e1aa4: aborting pipeline replication: peer="{Voter server-46288b31-3d8c-824f-acb7-161eb6c62fe7 46288b31-3d8c-824f-acb7-161eb6c62fe7}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2024-06-11T14:41:13.576Z [INFO] server-59cbcaaf-efdc-4779-9aaf-07855b85ed98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59cbcaaf-efdc-4779-9aaf-07855b85ed98 Address:59cbcaaf-efdc-4779-9aaf-07855b85ed98} {Suffrage:Voter ID:server-af08fb93-2f35-4377-2236-8b60761c9aa7 Address:af08fb93-2f35-4377-2236-8b60761c9aa7} {Suffrage:Voter ID:server-f0ad6f73-4564-a91f-4813-0bb6322051e1 Address:f0ad6f73-4564-a91f-4813-0bb6322051e1}]" 2024-06-11T14:41:13.576Z [INFO] server-af08fb93-2f35-4377-2236-8b60761c9aa7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59cbcaaf-efdc-4779-9aaf-07855b85ed98 Address:59cbcaaf-efdc-4779-9aaf-07855b85ed98} {Suffrage:Voter ID:server-af08fb93-2f35-4377-2236-8b60761c9aa7 Address:af08fb93-2f35-4377-2236-8b60761c9aa7} {Suffrage:Voter ID:server-f0ad6f73-4564-a91f-4813-0bb6322051e1 Address:f0ad6f73-4564-a91f-4813-0bb6322051e1}]" 2024-06-11T14:41:13.576Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59cbcaaf-efdc-4779-9aaf-07855b85ed98 Address:59cbcaaf-efdc-4779-9aaf-07855b85ed98} {Suffrage:Voter ID:server-af08fb93-2f35-4377-2236-8b60761c9aa7 Address:af08fb93-2f35-4377-2236-8b60761c9aa7} {Suffrage:Voter ID:server-f0ad6f73-4564-a91f-4813-0bb6322051e1 Address:f0ad6f73-4564-a91f-4813-0bb6322051e1}]" 2024-06-11T14:41:13.576Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:13.576Z [INFO] server-59cbcaaf-efdc-4779-9aaf-07855b85ed98: entering follower state: follower="Node at 59cbcaaf-efdc-4779-9aaf-07855b85ed98 [Follower]" leader-address= leader-id= 2024-06-11T14:41:13.576Z [INFO] server-af08fb93-2f35-4377-2236-8b60761c9aa7: entering follower state: follower="Node at af08fb93-2f35-4377-2236-8b60761c9aa7 [Follower]" leader-address= leader-id= 2024-06-11T14:41:13.576Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: entering follower state: follower="Node at f0ad6f73-4564-a91f-4813-0bb6322051e1 [Follower]" leader-address= leader-id= 2024-06-11T14:41:14.291Z [WARN] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:14.291Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: entering candidate state: node="Node at f0ad6f73-4564-a91f-4813-0bb6322051e1 [Candidate]" term=2 2024-06-11T14:41:14.291Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: election won: term=2 tally=2 2024-06-11T14:41:14.291Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: entering leader state: leader="Node at f0ad6f73-4564-a91f-4813-0bb6322051e1 [Leader]" 2024-06-11T14:41:14.291Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: added peer, starting replication: peer=server-59cbcaaf-efdc-4779-9aaf-07855b85ed98 2024-06-11T14:41:14.291Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: added peer, starting replication: peer=server-af08fb93-2f35-4377-2236-8b60761c9aa7 2024-06-11T14:41:14.291Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: pipelining replication: peer="{Voter server-59cbcaaf-efdc-4779-9aaf-07855b85ed98 59cbcaaf-efdc-4779-9aaf-07855b85ed98}" 2024-06-11T14:41:14.291Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: pipelining replication: peer="{Voter server-af08fb93-2f35-4377-2236-8b60761c9aa7 af08fb93-2f35-4377-2236-8b60761c9aa7}" 2024-06-11T14:41:15.297Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:14.29208785 +0000 UTC m=+80.875166762 (1 nodes), 715.524462ms from start of poll, 716.129281ms from cluster start. Timeout at 2024-06-11 14:41:15.297859399 +0000 UTC m=+81.880938351, 1.005771589s after stability 2024-06-11T14:41:15.299Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: starting snapshot up to: index=12 2024-06-11T14:41:15.299Z [INFO] creating new snapshot: path=/tmp/raft1663030435/snapshots/2-12-1718116875299.tmp 2024-06-11T14:41:15.300Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: snapshot complete up to: index=12 2024-06-11T14:41:15.302Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d Address:c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d} {Suffrage:Voter ID:server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 Address:f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46} {Suffrage:Voter ID:server-8aa15fce-4803-09e7-6665-a8d3d09f5b63 Address:8aa15fce-4803-09e7-6665-a8d3d09f5b63}]" 2024-06-11T14:41:15.302Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d Address:c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d} {Suffrage:Voter ID:server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 Address:f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46} {Suffrage:Voter ID:server-8aa15fce-4803-09e7-6665-a8d3d09f5b63 Address:8aa15fce-4803-09e7-6665-a8d3d09f5b63}]" 2024-06-11T14:41:15.302Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: entering follower state: follower="Node at c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d [Follower]" leader-address= leader-id= 2024-06-11T14:41:15.303Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: entering follower state: follower="Node at f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 [Follower]" leader-address= leader-id= 2024-06-11T14:41:15.303Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d Address:c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d} {Suffrage:Voter ID:server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 Address:f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46} {Suffrage:Voter ID:server-8aa15fce-4803-09e7-6665-a8d3d09f5b63 Address:8aa15fce-4803-09e7-6665-a8d3d09f5b63}]" 2024-06-11T14:41:15.303Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:15.303Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: entering follower state: follower="Node at 8aa15fce-4803-09e7-6665-a8d3d09f5b63 [Follower]" leader-address= leader-id= 2024-06-11T14:41:16.069Z [WARN] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:16.069Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: entering candidate state: node="Node at 8aa15fce-4803-09e7-6665-a8d3d09f5b63 [Candidate]" term=2 2024-06-11T14:41:16.069Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: election won: term=2 tally=2 2024-06-11T14:41:16.069Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: entering leader state: leader="Node at 8aa15fce-4803-09e7-6665-a8d3d09f5b63 [Leader]" 2024-06-11T14:41:16.069Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: added peer, starting replication: peer=server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d 2024-06-11T14:41:16.069Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: added peer, starting replication: peer=server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 2024-06-11T14:41:16.069Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: pipelining replication: peer="{Voter server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46}" 2024-06-11T14:41:16.069Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: pipelining replication: peer="{Voter server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d}" 2024-06-11T14:41:17.074Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:16.069482797 +0000 UTC m=+82.652561709 (1 nodes), 766.292111ms from start of poll, 766.965446ms from cluster start. Timeout at 2024-06-11 14:41:17.074494893 +0000 UTC m=+83.657573845, 1.005012136s after stability 2024-06-11T14:41:17.074Z [INFO] creating new snapshot: path=/tmp/raft3524994394/snapshots/2-1013-1718116877074.tmp 2024-06-11T14:41:17.076Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: copied to local snapshot: bytes=71 2024-06-11T14:41:17.078Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: snapshot restore progress: id=2-1013-1718116877074 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:17.078Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: removing all old logs from log store 2024-06-11T14:41:17.078Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: compacting logs: from=1 to=2 2024-06-11T14:41:17.078Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: restored user snapshot: index=1013 2024-06-11T14:41:17.078Z [ERROR] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: failed to get log: index=2 error="log not found" 2024-06-11T14:41:17.079Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: aborting pipeline replication: peer="{Voter server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46}" 2024-06-11T14:41:17.080Z [ERROR] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: failed to get log: index=2 error="log not found" 2024-06-11T14:41:17.080Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: aborting pipeline replication: peer="{Voter server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d}" 2024-06-11T14:41:17.088Z [ERROR] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: failed to get log: index=2 error="log not found" 2024-06-11T14:41:17.088Z [ERROR] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: failed to get log: index=2 error="log not found" 2024-06-11T14:41:17.089Z [INFO] creating new snapshot: path=/tmp/raft26103707/snapshots/2-1013-1718116877089.tmp 2024-06-11T14:41:17.089Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:17.090Z [INFO] creating new snapshot: path=/tmp/raft977497356/snapshots/2-1013-1718116877090.tmp 2024-06-11T14:41:17.090Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: copied to local snapshot: bytes=71 2024-06-11T14:41:17.090Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:17.091Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: snapshot restore progress: id=2-1013-1718116877089 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:17.091Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: removing all old logs from log store 2024-06-11T14:41:17.091Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: compacting logs: from=1 to=2 2024-06-11T14:41:17.091Z [INFO] server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d: Installed remote snapshot 2024-06-11T14:41:17.091Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: pipelining replication: peer="{Voter server-c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d c60ebcb9-4f36-4622-3dcd-6c9a7cdbf05d}" 2024-06-11T14:41:17.091Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:17.091Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: copied to local snapshot: bytes=71 2024-06-11T14:41:17.092Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: snapshot restore progress: id=2-1013-1718116877090 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:17.092Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: removing all old logs from log store 2024-06-11T14:41:17.092Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: compacting logs: from=1 to=2 2024-06-11T14:41:17.092Z [INFO] server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46: Installed remote snapshot 2024-06-11T14:41:17.092Z [INFO] server-8aa15fce-4803-09e7-6665-a8d3d09f5b63: pipelining replication: peer="{Voter server-f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46 f6deeb8b-1d27-f96e-2b70-9ca2e6bb2e46}" 2024-06-11T14:41:18.096Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:17.091587876 +0000 UTC m=+83.674666788 (2 nodes), 1.56µs from start of poll, 1.789070525s from cluster start. Timeout at 2024-06-11 14:41:18.096606893 +0000 UTC m=+84.679685845, 1.005019057s after stability 2024-06-11T14:41:18.110Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: aborting pipeline replication: peer="{Voter server-af08fb93-2f35-4377-2236-8b60761c9aa7 af08fb93-2f35-4377-2236-8b60761c9aa7}" 2024-06-11T14:41:18.110Z [INFO] server-f0ad6f73-4564-a91f-4813-0bb6322051e1: aborting pipeline replication: peer="{Voter server-59cbcaaf-efdc-4779-9aaf-07855b85ed98 59cbcaaf-efdc-4779-9aaf-07855b85ed98}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2024-06-11T14:41:18.113Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2d95a5-ed94-3427-eb25-a3c8473852c7 Address:6b2d95a5-ed94-3427-eb25-a3c8473852c7} {Suffrage:Voter ID:server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 Address:a375d026-f0a1-d641-c9cd-0b08fa0613a2} {Suffrage:Voter ID:server-f6b48a49-f87a-2756-36c7-49ffaf91028d Address:f6b48a49-f87a-2756-36c7-49ffaf91028d}]" 2024-06-11T14:41:18.113Z [INFO] server-a375d026-f0a1-d641-c9cd-0b08fa0613a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2d95a5-ed94-3427-eb25-a3c8473852c7 Address:6b2d95a5-ed94-3427-eb25-a3c8473852c7} {Suffrage:Voter ID:server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 Address:a375d026-f0a1-d641-c9cd-0b08fa0613a2} {Suffrage:Voter ID:server-f6b48a49-f87a-2756-36c7-49ffaf91028d Address:f6b48a49-f87a-2756-36c7-49ffaf91028d}]" 2024-06-11T14:41:18.113Z [INFO] server-f6b48a49-f87a-2756-36c7-49ffaf91028d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2d95a5-ed94-3427-eb25-a3c8473852c7 Address:6b2d95a5-ed94-3427-eb25-a3c8473852c7} {Suffrage:Voter ID:server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 Address:a375d026-f0a1-d641-c9cd-0b08fa0613a2} {Suffrage:Voter ID:server-f6b48a49-f87a-2756-36c7-49ffaf91028d Address:f6b48a49-f87a-2756-36c7-49ffaf91028d}]" 2024-06-11T14:41:18.113Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:18.113Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: entering follower state: follower="Node at 6b2d95a5-ed94-3427-eb25-a3c8473852c7 [Follower]" leader-address= leader-id= 2024-06-11T14:41:18.113Z [INFO] server-a375d026-f0a1-d641-c9cd-0b08fa0613a2: entering follower state: follower="Node at a375d026-f0a1-d641-c9cd-0b08fa0613a2 [Follower]" leader-address= leader-id= 2024-06-11T14:41:18.113Z [INFO] server-f6b48a49-f87a-2756-36c7-49ffaf91028d: entering follower state: follower="Node at f6b48a49-f87a-2756-36c7-49ffaf91028d [Follower]" leader-address= leader-id= 2024-06-11T14:41:18.648Z [WARN] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:18.648Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: entering candidate state: node="Node at 6b2d95a5-ed94-3427-eb25-a3c8473852c7 [Candidate]" term=2 2024-06-11T14:41:18.648Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: election won: term=2 tally=2 2024-06-11T14:41:18.648Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: entering leader state: leader="Node at 6b2d95a5-ed94-3427-eb25-a3c8473852c7 [Leader]" 2024-06-11T14:41:18.648Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: added peer, starting replication: peer=server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 2024-06-11T14:41:18.648Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: added peer, starting replication: peer=server-f6b48a49-f87a-2756-36c7-49ffaf91028d 2024-06-11T14:41:18.649Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: pipelining replication: peer="{Voter server-f6b48a49-f87a-2756-36c7-49ffaf91028d f6b48a49-f87a-2756-36c7-49ffaf91028d}" 2024-06-11T14:41:18.649Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: pipelining replication: peer="{Voter server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 a375d026-f0a1-d641-c9cd-0b08fa0613a2}" 2024-06-11T14:41:19.655Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:18.649665737 +0000 UTC m=+85.232744649 (1 nodes), 536.150894ms from start of poll, 536.796151ms from cluster start. Timeout at 2024-06-11 14:41:19.655373768 +0000 UTC m=+86.238452680, 1.005708031s after stability 2024-06-11T14:41:19.661Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: starting snapshot up to: index=12 2024-06-11T14:41:19.661Z [INFO] creating new snapshot: path=/tmp/raft296166262/snapshots/2-12-1718116879661.tmp 2024-06-11T14:41:19.663Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: snapshot complete up to: index=12 2024-06-11T14:41:19.664Z [INFO] creating new snapshot: path=/tmp/raft296166262/snapshots/2-10013-1718116879664.tmp 2024-06-11T14:41:19.665Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: copied to local snapshot: bytes=71 2024-06-11T14:41:19.666Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: snapshot restore progress: id=2-10013-1718116879664 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:19.666Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: restored user snapshot: index=10013 2024-06-11T14:41:19.666Z [ERROR] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: failed to get log: index=23 error="log not found" 2024-06-11T14:41:19.666Z [ERROR] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: failed to get log: index=23 error="log not found" 2024-06-11T14:41:19.666Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: aborting pipeline replication: peer="{Voter server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 a375d026-f0a1-d641-c9cd-0b08fa0613a2}" 2024-06-11T14:41:19.666Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: aborting pipeline replication: peer="{Voter server-f6b48a49-f87a-2756-36c7-49ffaf91028d f6b48a49-f87a-2756-36c7-49ffaf91028d}" 2024-06-11T14:41:19.672Z [ERROR] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: failed to get log: index=23 error="log not found" 2024-06-11T14:41:19.673Z [INFO] creating new snapshot: path=/tmp/raft2453446214/snapshots/2-10013-1718116879673.tmp 2024-06-11T14:41:19.673Z [ERROR] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: failed to get log: index=23 error="log not found" 2024-06-11T14:41:19.673Z [INFO] server-a375d026-f0a1-d641-c9cd-0b08fa0613a2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:19.674Z [INFO] server-a375d026-f0a1-d641-c9cd-0b08fa0613a2: copied to local snapshot: bytes=71 2024-06-11T14:41:19.674Z [INFO] creating new snapshot: path=/tmp/raft4033953407/snapshots/2-10013-1718116879674.tmp 2024-06-11T14:41:19.674Z [INFO] server-a375d026-f0a1-d641-c9cd-0b08fa0613a2: snapshot restore progress: id=2-10013-1718116879673 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:19.674Z [INFO] server-a375d026-f0a1-d641-c9cd-0b08fa0613a2: Installed remote snapshot 2024-06-11T14:41:19.675Z [WARN] server-a375d026-f0a1-d641-c9cd-0b08fa0613a2: skipping application of old log: index=22 2024-06-11T14:41:19.675Z [INFO] server-f6b48a49-f87a-2756-36c7-49ffaf91028d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:19.675Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: pipelining replication: peer="{Voter server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 a375d026-f0a1-d641-c9cd-0b08fa0613a2}" 2024-06-11T14:41:19.675Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:19.675Z [INFO] server-f6b48a49-f87a-2756-36c7-49ffaf91028d: copied to local snapshot: bytes=71 2024-06-11T14:41:19.675Z [INFO] server-f6b48a49-f87a-2756-36c7-49ffaf91028d: snapshot restore progress: id=2-10013-1718116879674 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:19.676Z [INFO] server-f6b48a49-f87a-2756-36c7-49ffaf91028d: Installed remote snapshot 2024-06-11T14:41:19.676Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: pipelining replication: peer="{Voter server-f6b48a49-f87a-2756-36c7-49ffaf91028d f6b48a49-f87a-2756-36c7-49ffaf91028d}" 2024-06-11T14:41:20.680Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:19.675250451 +0000 UTC m=+86.258329363 (2 nodes), 1.44µs from start of poll, 1.562380865s from cluster start. Timeout at 2024-06-11 14:41:20.680713731 +0000 UTC m=+87.263792683, 1.00546332s after stability 2024-06-11T14:41:20.694Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: aborting pipeline replication: peer="{Voter server-f6b48a49-f87a-2756-36c7-49ffaf91028d f6b48a49-f87a-2756-36c7-49ffaf91028d}" 2024-06-11T14:41:20.694Z [INFO] server-6b2d95a5-ed94-3427-eb25-a3c8473852c7: aborting pipeline replication: peer="{Voter server-a375d026-f0a1-d641-c9cd-0b08fa0613a2 a375d026-f0a1-d641-c9cd-0b08fa0613a2}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2024-06-11T14:41:20.698Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 Address:8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3} {Suffrage:Voter ID:server-bd959d28-94fe-815e-4b00-7b3987e232ae Address:bd959d28-94fe-815e-4b00-7b3987e232ae} {Suffrage:Voter ID:server-b4d8b611-2d54-822b-6725-47eda9ddc080 Address:b4d8b611-2d54-822b-6725-47eda9ddc080}]" 2024-06-11T14:41:20.699Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: entering follower state: follower="Node at 8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 [Follower]" leader-address= leader-id= 2024-06-11T14:41:20.700Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 Address:8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3} {Suffrage:Voter ID:server-bd959d28-94fe-815e-4b00-7b3987e232ae Address:bd959d28-94fe-815e-4b00-7b3987e232ae} {Suffrage:Voter ID:server-b4d8b611-2d54-822b-6725-47eda9ddc080 Address:b4d8b611-2d54-822b-6725-47eda9ddc080}]" 2024-06-11T14:41:20.700Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 Address:8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3} {Suffrage:Voter ID:server-bd959d28-94fe-815e-4b00-7b3987e232ae Address:bd959d28-94fe-815e-4b00-7b3987e232ae} {Suffrage:Voter ID:server-b4d8b611-2d54-822b-6725-47eda9ddc080 Address:b4d8b611-2d54-822b-6725-47eda9ddc080}]" 2024-06-11T14:41:20.700Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:20.700Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: entering follower state: follower="Node at b4d8b611-2d54-822b-6725-47eda9ddc080 [Follower]" leader-address= leader-id= 2024-06-11T14:41:20.700Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: entering follower state: follower="Node at bd959d28-94fe-815e-4b00-7b3987e232ae [Follower]" leader-address= leader-id= 2024-06-11T14:41:21.461Z [WARN] server-b4d8b611-2d54-822b-6725-47eda9ddc080: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:21.461Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: entering candidate state: node="Node at b4d8b611-2d54-822b-6725-47eda9ddc080 [Candidate]" term=2 2024-06-11T14:41:21.461Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: election won: term=2 tally=2 2024-06-11T14:41:21.461Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: entering leader state: leader="Node at b4d8b611-2d54-822b-6725-47eda9ddc080 [Leader]" 2024-06-11T14:41:21.461Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: added peer, starting replication: peer=server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 2024-06-11T14:41:21.461Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: added peer, starting replication: peer=server-bd959d28-94fe-815e-4b00-7b3987e232ae 2024-06-11T14:41:21.461Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: pipelining replication: peer="{Voter server-bd959d28-94fe-815e-4b00-7b3987e232ae bd959d28-94fe-815e-4b00-7b3987e232ae}" 2024-06-11T14:41:21.461Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: pipelining replication: peer="{Voter server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3}" 2024-06-11T14:41:22.467Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:21.46203018 +0000 UTC m=+88.045109132 (1 nodes), 761.546602ms from start of poll, 764.178867ms from cluster start. Timeout at 2024-06-11 14:41:22.46776745 +0000 UTC m=+89.050846402, 1.00573727s after stability 2024-06-11T14:41:22.469Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: starting snapshot up to: index=12 2024-06-11T14:41:22.469Z [INFO] creating new snapshot: path=/tmp/raft3817529648/snapshots/2-12-1718116882469.tmp 2024-06-11T14:41:22.471Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: snapshot complete up to: index=12 2024-06-11T14:41:22.472Z [INFO] creating new snapshot: path=/tmp/raft3817529648/snapshots/2-10013-1718116882472.tmp 2024-06-11T14:41:22.473Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: copied to local snapshot: bytes=71 2024-06-11T14:41:22.474Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: snapshot restore progress: id=2-10013-1718116882472 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:22.474Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: removing all old logs from log store 2024-06-11T14:41:22.474Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: compacting logs: from=1 to=22 2024-06-11T14:41:22.474Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: restored user snapshot: index=10013 2024-06-11T14:41:22.474Z [ERROR] server-b4d8b611-2d54-822b-6725-47eda9ddc080: failed to get log: index=22 error="log not found" 2024-06-11T14:41:22.474Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: aborting pipeline replication: peer="{Voter server-bd959d28-94fe-815e-4b00-7b3987e232ae bd959d28-94fe-815e-4b00-7b3987e232ae}" 2024-06-11T14:41:22.475Z [ERROR] server-b4d8b611-2d54-822b-6725-47eda9ddc080: failed to get log: index=22 error="log not found" 2024-06-11T14:41:22.475Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: aborting pipeline replication: peer="{Voter server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3}" 2024-06-11T14:41:22.481Z [ERROR] server-b4d8b611-2d54-822b-6725-47eda9ddc080: failed to get log: index=22 error="log not found" 2024-06-11T14:41:22.481Z [ERROR] server-b4d8b611-2d54-822b-6725-47eda9ddc080: failed to get log: index=22 error="log not found" 2024-06-11T14:41:22.483Z [INFO] creating new snapshot: path=/tmp/raft1150868804/snapshots/2-10013-1718116882483.tmp 2024-06-11T14:41:22.483Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:22.484Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: copied to local snapshot: bytes=71 2024-06-11T14:41:22.487Z [INFO] creating new snapshot: path=/tmp/raft377044600/snapshots/2-10013-1718116882487.tmp 2024-06-11T14:41:22.487Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: snapshot restore progress: id=2-10013-1718116882483 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:22.487Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: removing all old logs from log store 2024-06-11T14:41:22.487Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: compacting logs: from=1 to=22 2024-06-11T14:41:22.487Z [INFO] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: Installed remote snapshot 2024-06-11T14:41:22.488Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:22.489Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: copied to local snapshot: bytes=71 2024-06-11T14:41:22.490Z [WARN] server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3: skipping application of old log: index=22 2024-06-11T14:41:22.491Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: pipelining replication: peer="{Voter server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3}" 2024-06-11T14:41:22.491Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:22.492Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: snapshot restore progress: id=2-10013-1718116882487 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:22.492Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: removing all old logs from log store 2024-06-11T14:41:22.492Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: compacting logs: from=1 to=22 2024-06-11T14:41:22.492Z [INFO] server-bd959d28-94fe-815e-4b00-7b3987e232ae: Installed remote snapshot 2024-06-11T14:41:22.493Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: pipelining replication: peer="{Voter server-bd959d28-94fe-815e-4b00-7b3987e232ae bd959d28-94fe-815e-4b00-7b3987e232ae}" 2024-06-11T14:41:23.496Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:22.491349599 +0000 UTC m=+89.074428511 (2 nodes), 1.68µs from start of poll, 1.793498246s from cluster start. Timeout at 2024-06-11 14:41:23.496810479 +0000 UTC m=+90.079889431, 1.00546092s after stability 2024-06-11T14:41:23.508Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: aborting pipeline replication: peer="{Voter server-bd959d28-94fe-815e-4b00-7b3987e232ae bd959d28-94fe-815e-4b00-7b3987e232ae}" 2024-06-11T14:41:23.509Z [INFO] server-b4d8b611-2d54-822b-6725-47eda9ddc080: aborting pipeline replication: peer="{Voter server-8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3 8c1aa6f8-cba5-b9a2-6cd1-34da67613ec3}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2024-06-11T14:41:23.512Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8559304-ae2b-50d0-71c4-19a191e234c9 Address:d8559304-ae2b-50d0-71c4-19a191e234c9} {Suffrage:Voter ID:server-372c1896-7bbd-4f86-9854-405405889783 Address:372c1896-7bbd-4f86-9854-405405889783} {Suffrage:Voter ID:server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7 Address:c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7}]" 2024-06-11T14:41:23.512Z [INFO] server-372c1896-7bbd-4f86-9854-405405889783: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8559304-ae2b-50d0-71c4-19a191e234c9 Address:d8559304-ae2b-50d0-71c4-19a191e234c9} {Suffrage:Voter ID:server-372c1896-7bbd-4f86-9854-405405889783 Address:372c1896-7bbd-4f86-9854-405405889783} {Suffrage:Voter ID:server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7 Address:c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7}]" 2024-06-11T14:41:23.512Z [INFO] server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8559304-ae2b-50d0-71c4-19a191e234c9 Address:d8559304-ae2b-50d0-71c4-19a191e234c9} {Suffrage:Voter ID:server-372c1896-7bbd-4f86-9854-405405889783 Address:372c1896-7bbd-4f86-9854-405405889783} {Suffrage:Voter ID:server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7 Address:c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7}]" 2024-06-11T14:41:23.512Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:23.512Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: entering follower state: follower="Node at d8559304-ae2b-50d0-71c4-19a191e234c9 [Follower]" leader-address= leader-id= 2024-06-11T14:41:23.512Z [INFO] server-372c1896-7bbd-4f86-9854-405405889783: entering follower state: follower="Node at 372c1896-7bbd-4f86-9854-405405889783 [Follower]" leader-address= leader-id= 2024-06-11T14:41:23.512Z [INFO] server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7: entering follower state: follower="Node at c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7 [Follower]" leader-address= leader-id= 2024-06-11T14:41:24.141Z [WARN] server-d8559304-ae2b-50d0-71c4-19a191e234c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:24.141Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: entering candidate state: node="Node at d8559304-ae2b-50d0-71c4-19a191e234c9 [Candidate]" term=2 2024-06-11T14:41:24.141Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: election won: term=2 tally=2 2024-06-11T14:41:24.141Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: entering leader state: leader="Node at d8559304-ae2b-50d0-71c4-19a191e234c9 [Leader]" 2024-06-11T14:41:24.141Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: added peer, starting replication: peer=server-372c1896-7bbd-4f86-9854-405405889783 2024-06-11T14:41:24.141Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: added peer, starting replication: peer=server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7 2024-06-11T14:41:24.142Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: pipelining replication: peer="{Voter server-372c1896-7bbd-4f86-9854-405405889783 372c1896-7bbd-4f86-9854-405405889783}" 2024-06-11T14:41:24.142Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: pipelining replication: peer="{Voter server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7 c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7}" 2024-06-11T14:41:25.148Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:24.141962041 +0000 UTC m=+90.725040953 (1 nodes), 629.533405ms from start of poll, 630.137383ms from cluster start. Timeout at 2024-06-11 14:41:25.147907144 +0000 UTC m=+91.730986056, 1.005945103s after stability 2024-06-11T14:41:25.149Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: starting snapshot up to: index=12 2024-06-11T14:41:25.149Z [INFO] creating new snapshot: path=/tmp/raft940039016/snapshots/2-12-1718116885149.tmp 2024-06-11T14:41:25.151Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: snapshot complete up to: index=12 2024-06-11T14:41:25.153Z [INFO] server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec Address:bcb4173f-9700-f688-6d3e-dfc3bc76a7ec} {Suffrage:Voter ID:server-083a668f-a154-0c0b-210a-950c1f255128 Address:083a668f-a154-0c0b-210a-950c1f255128} {Suffrage:Voter ID:server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 Address:e88140c8-4a1a-ee2d-2d9e-6c7881067ef9}]" 2024-06-11T14:41:25.154Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec Address:bcb4173f-9700-f688-6d3e-dfc3bc76a7ec} {Suffrage:Voter ID:server-083a668f-a154-0c0b-210a-950c1f255128 Address:083a668f-a154-0c0b-210a-950c1f255128} {Suffrage:Voter ID:server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 Address:e88140c8-4a1a-ee2d-2d9e-6c7881067ef9}]" 2024-06-11T14:41:25.154Z [INFO] server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec: entering follower state: follower="Node at bcb4173f-9700-f688-6d3e-dfc3bc76a7ec [Follower]" leader-address= leader-id= 2024-06-11T14:41:25.154Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: entering follower state: follower="Node at 083a668f-a154-0c0b-210a-950c1f255128 [Follower]" leader-address= leader-id= 2024-06-11T14:41:25.154Z [INFO] server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec Address:bcb4173f-9700-f688-6d3e-dfc3bc76a7ec} {Suffrage:Voter ID:server-083a668f-a154-0c0b-210a-950c1f255128 Address:083a668f-a154-0c0b-210a-950c1f255128} {Suffrage:Voter ID:server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 Address:e88140c8-4a1a-ee2d-2d9e-6c7881067ef9}]" 2024-06-11T14:41:25.154Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:25.154Z [INFO] server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9: entering follower state: follower="Node at e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 [Follower]" leader-address= leader-id= 2024-06-11T14:41:25.681Z [WARN] server-083a668f-a154-0c0b-210a-950c1f255128: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:25.681Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: entering candidate state: node="Node at 083a668f-a154-0c0b-210a-950c1f255128 [Candidate]" term=2 2024-06-11T14:41:25.681Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: election won: term=2 tally=2 2024-06-11T14:41:25.681Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: entering leader state: leader="Node at 083a668f-a154-0c0b-210a-950c1f255128 [Leader]" 2024-06-11T14:41:25.681Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: added peer, starting replication: peer=server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec 2024-06-11T14:41:25.681Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: added peer, starting replication: peer=server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 2024-06-11T14:41:25.681Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: pipelining replication: peer="{Voter server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec bcb4173f-9700-f688-6d3e-dfc3bc76a7ec}" 2024-06-11T14:41:25.681Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: pipelining replication: peer="{Voter server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 e88140c8-4a1a-ee2d-2d9e-6c7881067ef9}" 2024-06-11T14:41:26.686Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:25.681647285 +0000 UTC m=+92.264726237 (1 nodes), 526.940747ms from start of poll, 527.96019ms from cluster start. Timeout at 2024-06-11 14:41:26.686718499 +0000 UTC m=+93.269797451, 1.005071214s after stability 2024-06-11T14:41:26.687Z [INFO] creating new snapshot: path=/tmp/raft3173956622/snapshots/2-10013-1718116886687.tmp 2024-06-11T14:41:26.688Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: copied to local snapshot: bytes=71 2024-06-11T14:41:26.688Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: snapshot restore progress: id=2-10013-1718116886687 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:26.688Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: restored user snapshot: index=10013 2024-06-11T14:41:26.688Z [ERROR] server-083a668f-a154-0c0b-210a-950c1f255128: failed to get log: index=3 error="log not found" 2024-06-11T14:41:26.688Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: aborting pipeline replication: peer="{Voter server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 e88140c8-4a1a-ee2d-2d9e-6c7881067ef9}" 2024-06-11T14:41:26.688Z [ERROR] server-083a668f-a154-0c0b-210a-950c1f255128: failed to get log: index=3 error="log not found" 2024-06-11T14:41:26.688Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: aborting pipeline replication: peer="{Voter server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec bcb4173f-9700-f688-6d3e-dfc3bc76a7ec}" 2024-06-11T14:41:26.697Z [ERROR] server-083a668f-a154-0c0b-210a-950c1f255128: failed to get log: index=3 error="log not found" 2024-06-11T14:41:26.697Z [ERROR] server-083a668f-a154-0c0b-210a-950c1f255128: failed to get log: index=3 error="log not found" 2024-06-11T14:41:26.697Z [INFO] creating new snapshot: path=/tmp/raft3189872522/snapshots/2-10013-1718116886697.tmp 2024-06-11T14:41:26.698Z [INFO] creating new snapshot: path=/tmp/raft793703742/snapshots/2-10013-1718116886698.tmp 2024-06-11T14:41:26.698Z [INFO] server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:26.698Z [INFO] server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:26.698Z [INFO] server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec: copied to local snapshot: bytes=71 2024-06-11T14:41:26.699Z [INFO] server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9: copied to local snapshot: bytes=71 2024-06-11T14:41:26.699Z [INFO] server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec: snapshot restore progress: id=2-10013-1718116886697 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:26.699Z [INFO] server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec: Installed remote snapshot 2024-06-11T14:41:26.699Z [INFO] server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9: snapshot restore progress: id=2-10013-1718116886698 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:26.699Z [INFO] server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9: Installed remote snapshot 2024-06-11T14:41:26.699Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: pipelining replication: peer="{Voter server-bcb4173f-9700-f688-6d3e-dfc3bc76a7ec bcb4173f-9700-f688-6d3e-dfc3bc76a7ec}" 2024-06-11T14:41:26.699Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:26.700Z [INFO] server-083a668f-a154-0c0b-210a-950c1f255128: pipelining replication: peer="{Voter server-e88140c8-4a1a-ee2d-2d9e-6c7881067ef9 e88140c8-4a1a-ee2d-2d9e-6c7881067ef9}" 2024-06-11T14:41:27.705Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:26.699886744 +0000 UTC m=+93.282965696 (2 nodes), 1.64µs from start of poll, 1.546199649s from cluster start. Timeout at 2024-06-11 14:41:27.705056715 +0000 UTC m=+94.288135667, 1.005169971s after stability 2024-06-11T14:41:27.720Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: aborting pipeline replication: peer="{Voter server-c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7 c7cf9a54-3dfb-b9dd-cdb7-a89ac4a27da7}" 2024-06-11T14:41:27.720Z [INFO] server-d8559304-ae2b-50d0-71c4-19a191e234c9: aborting pipeline replication: peer="{Voter server-372c1896-7bbd-4f86-9854-405405889783 372c1896-7bbd-4f86-9854-405405889783}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2024-06-11T14:41:27.722Z [INFO] server-4a373e0c-83df-6312-389b-5a719682bcb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a373e0c-83df-6312-389b-5a719682bcb2 Address:4a373e0c-83df-6312-389b-5a719682bcb2} {Suffrage:Voter ID:server-1ca0967d-80f0-b184-6ca4-b5de46db3466 Address:1ca0967d-80f0-b184-6ca4-b5de46db3466} {Suffrage:Voter ID:server-13293094-936e-4ab1-feca-b7328db58c67 Address:13293094-936e-4ab1-feca-b7328db58c67}]" 2024-06-11T14:41:27.723Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a373e0c-83df-6312-389b-5a719682bcb2 Address:4a373e0c-83df-6312-389b-5a719682bcb2} {Suffrage:Voter ID:server-1ca0967d-80f0-b184-6ca4-b5de46db3466 Address:1ca0967d-80f0-b184-6ca4-b5de46db3466} {Suffrage:Voter ID:server-13293094-936e-4ab1-feca-b7328db58c67 Address:13293094-936e-4ab1-feca-b7328db58c67}]" 2024-06-11T14:41:27.723Z [INFO] server-13293094-936e-4ab1-feca-b7328db58c67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a373e0c-83df-6312-389b-5a719682bcb2 Address:4a373e0c-83df-6312-389b-5a719682bcb2} {Suffrage:Voter ID:server-1ca0967d-80f0-b184-6ca4-b5de46db3466 Address:1ca0967d-80f0-b184-6ca4-b5de46db3466} {Suffrage:Voter ID:server-13293094-936e-4ab1-feca-b7328db58c67 Address:13293094-936e-4ab1-feca-b7328db58c67}]" 2024-06-11T14:41:27.723Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:27.723Z [INFO] server-4a373e0c-83df-6312-389b-5a719682bcb2: entering follower state: follower="Node at 4a373e0c-83df-6312-389b-5a719682bcb2 [Follower]" leader-address= leader-id= 2024-06-11T14:41:27.723Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: entering follower state: follower="Node at 1ca0967d-80f0-b184-6ca4-b5de46db3466 [Follower]" leader-address= leader-id= 2024-06-11T14:41:27.723Z [INFO] server-13293094-936e-4ab1-feca-b7328db58c67: entering follower state: follower="Node at 13293094-936e-4ab1-feca-b7328db58c67 [Follower]" leader-address= leader-id= 2024-06-11T14:41:28.238Z [WARN] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:28.238Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: entering candidate state: node="Node at 1ca0967d-80f0-b184-6ca4-b5de46db3466 [Candidate]" term=2 2024-06-11T14:41:28.238Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: election won: term=2 tally=2 2024-06-11T14:41:28.238Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: entering leader state: leader="Node at 1ca0967d-80f0-b184-6ca4-b5de46db3466 [Leader]" 2024-06-11T14:41:28.239Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: added peer, starting replication: peer=server-4a373e0c-83df-6312-389b-5a719682bcb2 2024-06-11T14:41:28.239Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: added peer, starting replication: peer=server-13293094-936e-4ab1-feca-b7328db58c67 2024-06-11T14:41:28.239Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: pipelining replication: peer="{Voter server-13293094-936e-4ab1-feca-b7328db58c67 13293094-936e-4ab1-feca-b7328db58c67}" 2024-06-11T14:41:28.239Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: pipelining replication: peer="{Voter server-4a373e0c-83df-6312-389b-5a719682bcb2 4a373e0c-83df-6312-389b-5a719682bcb2}" 2024-06-11T14:41:29.244Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:28.239145764 +0000 UTC m=+94.822224716 (1 nodes), 515.706312ms from start of poll, 516.381448ms from cluster start. Timeout at 2024-06-11 14:41:29.244583165 +0000 UTC m=+95.827662117, 1.005437401s after stability 2024-06-11T14:41:29.245Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: starting snapshot up to: index=12 2024-06-11T14:41:29.245Z [INFO] creating new snapshot: path=/tmp/raft812477429/snapshots/2-12-1718116889245.tmp 2024-06-11T14:41:29.247Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: snapshot complete up to: index=12 2024-06-11T14:41:29.249Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85f5b80e-fc38-79c1-365d-b92cda6f7572 Address:85f5b80e-fc38-79c1-365d-b92cda6f7572} {Suffrage:Voter ID:server-ffcdfe77-3681-bb31-6b3d-961e960ece22 Address:ffcdfe77-3681-bb31-6b3d-961e960ece22} {Suffrage:Voter ID:server-0395e3db-4634-6580-620c-0d72d08c3273 Address:0395e3db-4634-6580-620c-0d72d08c3273}]" 2024-06-11T14:41:29.249Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85f5b80e-fc38-79c1-365d-b92cda6f7572 Address:85f5b80e-fc38-79c1-365d-b92cda6f7572} {Suffrage:Voter ID:server-ffcdfe77-3681-bb31-6b3d-961e960ece22 Address:ffcdfe77-3681-bb31-6b3d-961e960ece22} {Suffrage:Voter ID:server-0395e3db-4634-6580-620c-0d72d08c3273 Address:0395e3db-4634-6580-620c-0d72d08c3273}]" 2024-06-11T14:41:29.250Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: entering follower state: follower="Node at 85f5b80e-fc38-79c1-365d-b92cda6f7572 [Follower]" leader-address= leader-id= 2024-06-11T14:41:29.250Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: entering follower state: follower="Node at ffcdfe77-3681-bb31-6b3d-961e960ece22 [Follower]" leader-address= leader-id= 2024-06-11T14:41:29.250Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85f5b80e-fc38-79c1-365d-b92cda6f7572 Address:85f5b80e-fc38-79c1-365d-b92cda6f7572} {Suffrage:Voter ID:server-ffcdfe77-3681-bb31-6b3d-961e960ece22 Address:ffcdfe77-3681-bb31-6b3d-961e960ece22} {Suffrage:Voter ID:server-0395e3db-4634-6580-620c-0d72d08c3273 Address:0395e3db-4634-6580-620c-0d72d08c3273}]" 2024-06-11T14:41:29.250Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:29.250Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: entering follower state: follower="Node at 0395e3db-4634-6580-620c-0d72d08c3273 [Follower]" leader-address= leader-id= 2024-06-11T14:41:30.065Z [WARN] server-0395e3db-4634-6580-620c-0d72d08c3273: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:30.065Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: entering candidate state: node="Node at 0395e3db-4634-6580-620c-0d72d08c3273 [Candidate]" term=2 2024-06-11T14:41:30.065Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: election won: term=2 tally=2 2024-06-11T14:41:30.065Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: entering leader state: leader="Node at 0395e3db-4634-6580-620c-0d72d08c3273 [Leader]" 2024-06-11T14:41:30.065Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: added peer, starting replication: peer=server-85f5b80e-fc38-79c1-365d-b92cda6f7572 2024-06-11T14:41:30.065Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: added peer, starting replication: peer=server-ffcdfe77-3681-bb31-6b3d-961e960ece22 2024-06-11T14:41:30.065Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: pipelining replication: peer="{Voter server-85f5b80e-fc38-79c1-365d-b92cda6f7572 85f5b80e-fc38-79c1-365d-b92cda6f7572}" 2024-06-11T14:41:30.065Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: pipelining replication: peer="{Voter server-ffcdfe77-3681-bb31-6b3d-961e960ece22 ffcdfe77-3681-bb31-6b3d-961e960ece22}" 2024-06-11T14:41:31.070Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:30.06534695 +0000 UTC m=+96.648425902 (1 nodes), 814.701364ms from start of poll, 815.823043ms from cluster start. Timeout at 2024-06-11 14:41:31.07053192 +0000 UTC m=+97.653610832, 1.00518493s after stability 2024-06-11T14:41:31.082Z [INFO] creating new snapshot: path=/tmp/raft1551059262/snapshots/2-10013-1718116891075.tmp 2024-06-11T14:41:31.083Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: copied to local snapshot: bytes=71 2024-06-11T14:41:31.084Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: snapshot restore progress: id=2-10013-1718116891075 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:31.084Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: removing all old logs from log store 2024-06-11T14:41:31.084Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: compacting logs: from=1 to=2 2024-06-11T14:41:31.084Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: restored user snapshot: index=10013 2024-06-11T14:41:31.084Z [ERROR] server-0395e3db-4634-6580-620c-0d72d08c3273: failed to get log: index=2 error="log not found" 2024-06-11T14:41:31.084Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: aborting pipeline replication: peer="{Voter server-ffcdfe77-3681-bb31-6b3d-961e960ece22 ffcdfe77-3681-bb31-6b3d-961e960ece22}" 2024-06-11T14:41:31.085Z [ERROR] server-0395e3db-4634-6580-620c-0d72d08c3273: failed to get log: index=2 error="log not found" 2024-06-11T14:41:31.085Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: aborting pipeline replication: peer="{Voter server-85f5b80e-fc38-79c1-365d-b92cda6f7572 85f5b80e-fc38-79c1-365d-b92cda6f7572}" 2024-06-11T14:41:31.091Z [ERROR] server-0395e3db-4634-6580-620c-0d72d08c3273: failed to get log: index=2 error="log not found" 2024-06-11T14:41:31.092Z [INFO] creating new snapshot: path=/tmp/raft1180683075/snapshots/2-10013-1718116891092.tmp 2024-06-11T14:41:31.092Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:31.093Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: copied to local snapshot: bytes=71 2024-06-11T14:41:31.094Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: snapshot restore progress: id=2-10013-1718116891092 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:31.094Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: removing all old logs from log store 2024-06-11T14:41:31.094Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: compacting logs: from=1 to=2 2024-06-11T14:41:31.094Z [INFO] server-ffcdfe77-3681-bb31-6b3d-961e960ece22: Installed remote snapshot 2024-06-11T14:41:31.094Z [ERROR] server-0395e3db-4634-6580-620c-0d72d08c3273: failed to get log: index=2 error="log not found" 2024-06-11T14:41:31.094Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: pipelining replication: peer="{Voter server-ffcdfe77-3681-bb31-6b3d-961e960ece22 ffcdfe77-3681-bb31-6b3d-961e960ece22}" 2024-06-11T14:41:31.094Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:31.095Z [INFO] creating new snapshot: path=/tmp/raft2322217095/snapshots/2-10013-1718116891095.tmp 2024-06-11T14:41:31.095Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:31.096Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: copied to local snapshot: bytes=71 2024-06-11T14:41:31.096Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: snapshot restore progress: id=2-10013-1718116891095 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-11T14:41:31.096Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: removing all old logs from log store 2024-06-11T14:41:31.096Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: compacting logs: from=1 to=2 2024-06-11T14:41:31.096Z [INFO] server-85f5b80e-fc38-79c1-365d-b92cda6f7572: Installed remote snapshot 2024-06-11T14:41:31.096Z [INFO] server-0395e3db-4634-6580-620c-0d72d08c3273: pipelining replication: peer="{Voter server-85f5b80e-fc38-79c1-365d-b92cda6f7572 85f5b80e-fc38-79c1-365d-b92cda6f7572}" 2024-06-11T14:41:32.101Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:31.094744767 +0000 UTC m=+97.677823719 (2 nodes), 1.48µs from start of poll, 1.84522086s from cluster start. Timeout at 2024-06-11 14:41:32.101100735 +0000 UTC m=+98.684179687, 1.006355968s after stability 2024-06-11T14:41:32.113Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: aborting pipeline replication: peer="{Voter server-4a373e0c-83df-6312-389b-5a719682bcb2 4a373e0c-83df-6312-389b-5a719682bcb2}" 2024-06-11T14:41:32.113Z [INFO] server-1ca0967d-80f0-b184-6ca4-b5de46db3466: aborting pipeline replication: peer="{Voter server-13293094-936e-4ab1-feca-b7328db58c67 13293094-936e-4ab1-feca-b7328db58c67}" --- PASS: TestRaft_UserRestore (84.39s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.59s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.57s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.37s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.21s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.70s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.76s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.49s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.56s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.59s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.57s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.19s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.49s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.82s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.76s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.39s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.21s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.68s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.75s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.14s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.54s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.58s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.81s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.21s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.40s) === RUN TestRaft_SendSnapshotFollower 2024-06-11T14:41:32.120Z [INFO] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e Address:cb344f00-4b1f-b3e7-cc70-e79898c93b3e} {Suffrage:Voter ID:server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef Address:0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef} {Suffrage:Voter ID:server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a Address:33e6afe3-b4a7-0172-40b3-2eb91aa8d99a}]" 2024-06-11T14:41:32.120Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e Address:cb344f00-4b1f-b3e7-cc70-e79898c93b3e} {Suffrage:Voter ID:server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef Address:0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef} {Suffrage:Voter ID:server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a Address:33e6afe3-b4a7-0172-40b3-2eb91aa8d99a}]" 2024-06-11T14:41:32.120Z [INFO] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: entering follower state: follower="Node at cb344f00-4b1f-b3e7-cc70-e79898c93b3e [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.120Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e Address:cb344f00-4b1f-b3e7-cc70-e79898c93b3e} {Suffrage:Voter ID:server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef Address:0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef} {Suffrage:Voter ID:server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a Address:33e6afe3-b4a7-0172-40b3-2eb91aa8d99a}]" 2024-06-11T14:41:32.120Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: entering follower state: follower="Node at 0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.120Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:32.120Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: entering follower state: follower="Node at 33e6afe3-b4a7-0172-40b3-2eb91aa8d99a [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.193Z [WARN] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:32.193Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: entering candidate state: node="Node at 33e6afe3-b4a7-0172-40b3-2eb91aa8d99a [Candidate]" term=2 2024-06-11T14:41:32.193Z [WARN] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:32.193Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: entering candidate state: node="Node at 0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef [Candidate]" term=2 2024-06-11T14:41:32.193Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: duplicate requestVote for same term: term=2 2024-06-11T14:41:32.194Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: duplicate requestVote for same term: term=2 2024-06-11T14:41:32.194Z [INFO] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: duplicate requestVote for same term: term=2 2024-06-11T14:41:32.195Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: election won: term=2 tally=2 2024-06-11T14:41:32.195Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: entering leader state: leader="Node at 33e6afe3-b4a7-0172-40b3-2eb91aa8d99a [Leader]" 2024-06-11T14:41:32.195Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: added peer, starting replication: peer=server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e 2024-06-11T14:41:32.195Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: added peer, starting replication: peer=server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef 2024-06-11T14:41:32.196Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: pipelining replication: peer="{Voter server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e cb344f00-4b1f-b3e7-cc70-e79898c93b3e}" 2024-06-11T14:41:32.203Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: entering follower state: follower="Node at 0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef [Follower]" leader-address=33e6afe3-b4a7-0172-40b3-2eb91aa8d99a leader-id=server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a 2024-06-11T14:41:32.203Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: pipelining replication: peer="{Voter server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef 0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef}" 2024-06-11T14:41:32.309Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:32.203327726 +0000 UTC m=+98.786406638 (2 nodes), 82.628418ms from start of poll, 83.377711ms from cluster start. Timeout at 2024-06-11 14:41:32.308933796 +0000 UTC m=+98.892012748, 105.60611ms after stability 2024-06-11T14:41:32.309Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:32.415Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:32.309189666 +0000 UTC m=+98.892268618 (1 nodes), 1.48µs from start of poll, 189.239691ms from cluster start. Timeout at 2024-06-11 14:41:32.415026647 +0000 UTC m=+98.998105559, 105.836941ms after stability 2024-06-11T14:41:32.415Z [ERROR] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: failed to pipeline appendEntries: peer="{Voter server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e cb344f00-4b1f-b3e7-cc70-e79898c93b3e}" error="append pipeline closed" 2024-06-11T14:41:32.415Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: aborting pipeline replication: peer="{Voter server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e cb344f00-4b1f-b3e7-cc70-e79898c93b3e}" 2024-06-11T14:41:32.415Z [ERROR] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: failed to appendEntries to: peer="{Voter server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e cb344f00-4b1f-b3e7-cc70-e79898c93b3e}" error="failed to connect to peer: cb344f00-4b1f-b3e7-cc70-e79898c93b3e" 2024-06-11T14:41:32.415Z [ERROR] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: failed to heartbeat to: peer=cb344f00-4b1f-b3e7-cc70-e79898c93b3e backoff time=10ms error="failed to connect to peer: cb344f00-4b1f-b3e7-cc70-e79898c93b3e" raft_test.go:1627: [INFO] Finished apply without behind follower 2024-06-11T14:41:32.423Z [ERROR] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: failed to take snapshot: error="nothing new to snapshot" 2024-06-11T14:41:32.423Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: starting snapshot up to: index=99 2024-06-11T14:41:32.423Z [INFO] creating new snapshot: path=/tmp/raft3824074542/snapshots/2-99-1718116892423.tmp 2024-06-11T14:41:32.424Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: compacting logs: from=1 to=92 2024-06-11T14:41:32.424Z [INFO] server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef: snapshot complete up to: index=99 2024-06-11T14:41:32.425Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: starting snapshot up to: index=102 2024-06-11T14:41:32.425Z [INFO] creating new snapshot: path=/tmp/raft105827925/snapshots/2-102-1718116892425.tmp 2024-06-11T14:41:32.425Z [ERROR] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: failed to appendEntries to: peer="{Voter server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e cb344f00-4b1f-b3e7-cc70-e79898c93b3e}" error="failed to connect to peer: cb344f00-4b1f-b3e7-cc70-e79898c93b3e" 2024-06-11T14:41:32.426Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: compacting logs: from=1 to=92 2024-06-11T14:41:32.426Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: snapshot complete up to: index=102 2024-06-11T14:41:32.435Z [ERROR] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: failed to get log: index=2 error="log not found" 2024-06-11T14:41:32.436Z [INFO] creating new snapshot: path=/tmp/raft1463489575/snapshots/2-102-1718116892436.tmp 2024-06-11T14:41:32.437Z [INFO] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2024-06-11T14:41:32.437Z [INFO] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: copied to local snapshot: bytes=693 2024-06-11T14:41:32.438Z [INFO] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: snapshot restore progress: id=2-102-1718116892436 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-11T14:41:32.438Z [INFO] server-cb344f00-4b1f-b3e7-cc70-e79898c93b3e: Installed remote snapshot 2024-06-11T14:41:32.440Z [INFO] server-33e6afe3-b4a7-0172-40b3-2eb91aa8d99a: aborting pipeline replication: peer="{Voter server-0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef 0c6c8e5f-241b-2d76-0ca9-2e1e1b9690ef}" --- PASS: TestRaft_SendSnapshotFollower (0.32s) === RUN TestRaft_SendSnapshotAndLogsFollower 2024-06-11T14:41:32.443Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f5083a-2171-4250-b62d-f6853f4efb0e Address:b9f5083a-2171-4250-b62d-f6853f4efb0e} {Suffrage:Voter ID:server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c Address:fd5b1c39-9db8-4a03-e9fd-cf199ef5862c} {Suffrage:Voter ID:server-14588a3d-4751-9789-b8aa-f2789bf17a31 Address:14588a3d-4751-9789-b8aa-f2789bf17a31}]" 2024-06-11T14:41:32.443Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: entering follower state: follower="Node at b9f5083a-2171-4250-b62d-f6853f4efb0e [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.443Z [INFO] server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f5083a-2171-4250-b62d-f6853f4efb0e Address:b9f5083a-2171-4250-b62d-f6853f4efb0e} {Suffrage:Voter ID:server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c Address:fd5b1c39-9db8-4a03-e9fd-cf199ef5862c} {Suffrage:Voter ID:server-14588a3d-4751-9789-b8aa-f2789bf17a31 Address:14588a3d-4751-9789-b8aa-f2789bf17a31}]" 2024-06-11T14:41:32.443Z [INFO] server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c: entering follower state: follower="Node at fd5b1c39-9db8-4a03-e9fd-cf199ef5862c [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.444Z [INFO] server-14588a3d-4751-9789-b8aa-f2789bf17a31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f5083a-2171-4250-b62d-f6853f4efb0e Address:b9f5083a-2171-4250-b62d-f6853f4efb0e} {Suffrage:Voter ID:server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c Address:fd5b1c39-9db8-4a03-e9fd-cf199ef5862c} {Suffrage:Voter ID:server-14588a3d-4751-9789-b8aa-f2789bf17a31 Address:14588a3d-4751-9789-b8aa-f2789bf17a31}]" 2024-06-11T14:41:32.444Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:32.444Z [INFO] server-14588a3d-4751-9789-b8aa-f2789bf17a31: entering follower state: follower="Node at 14588a3d-4751-9789-b8aa-f2789bf17a31 [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.520Z [WARN] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:32.520Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: entering candidate state: node="Node at b9f5083a-2171-4250-b62d-f6853f4efb0e [Candidate]" term=2 2024-06-11T14:41:32.520Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: election won: term=2 tally=2 2024-06-11T14:41:32.520Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: entering leader state: leader="Node at b9f5083a-2171-4250-b62d-f6853f4efb0e [Leader]" 2024-06-11T14:41:32.520Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: added peer, starting replication: peer=server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c 2024-06-11T14:41:32.520Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: added peer, starting replication: peer=server-14588a3d-4751-9789-b8aa-f2789bf17a31 2024-06-11T14:41:32.521Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: pipelining replication: peer="{Voter server-14588a3d-4751-9789-b8aa-f2789bf17a31 14588a3d-4751-9789-b8aa-f2789bf17a31}" 2024-06-11T14:41:32.521Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: pipelining replication: peer="{Voter server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c fd5b1c39-9db8-4a03-e9fd-cf199ef5862c}" 2024-06-11T14:41:32.627Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:32.521462447 +0000 UTC m=+99.104541359 (2 nodes), 77.353129ms from start of poll, 78.107302ms from cluster start. Timeout at 2024-06-11 14:41:32.627902526 +0000 UTC m=+99.210981478, 106.440119ms after stability 2024-06-11T14:41:32.628Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:32.733Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:32.62808956 +0000 UTC m=+99.211168512 (1 nodes), 1.28µs from start of poll, 184.734455ms from cluster start. Timeout at 2024-06-11 14:41:32.733564914 +0000 UTC m=+99.316643826, 105.475314ms after stability 2024-06-11T14:41:32.733Z [ERROR] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: failed to pipeline appendEntries: peer="{Voter server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c fd5b1c39-9db8-4a03-e9fd-cf199ef5862c}" error="append pipeline closed" 2024-06-11T14:41:32.733Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: aborting pipeline replication: peer="{Voter server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c fd5b1c39-9db8-4a03-e9fd-cf199ef5862c}" 2024-06-11T14:41:32.733Z [ERROR] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: failed to appendEntries to: peer="{Voter server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c fd5b1c39-9db8-4a03-e9fd-cf199ef5862c}" error="failed to connect to peer: fd5b1c39-9db8-4a03-e9fd-cf199ef5862c" raft_test.go:1669: [INFO] Finished apply without behind follower 2024-06-11T14:41:32.741Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: starting snapshot up to: index=102 2024-06-11T14:41:32.741Z [INFO] creating new snapshot: path=/tmp/raft301683537/snapshots/2-102-1718116892741.tmp 2024-06-11T14:41:32.742Z [ERROR] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: failed to heartbeat to: peer=fd5b1c39-9db8-4a03-e9fd-cf199ef5862c backoff time=10ms error="failed to connect to peer: fd5b1c39-9db8-4a03-e9fd-cf199ef5862c" 2024-06-11T14:41:32.742Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: compacting logs: from=1 to=92 2024-06-11T14:41:32.742Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: snapshot complete up to: index=102 2024-06-11T14:41:32.742Z [ERROR] server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c: failed to take snapshot: error="nothing new to snapshot" 2024-06-11T14:41:32.742Z [INFO] server-14588a3d-4751-9789-b8aa-f2789bf17a31: starting snapshot up to: index=99 2024-06-11T14:41:32.743Z [INFO] creating new snapshot: path=/tmp/raft2056677756/snapshots/2-99-1718116892743.tmp 2024-06-11T14:41:32.744Z [ERROR] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: failed to get log: index=2 error="log not found" 2024-06-11T14:41:32.744Z [INFO] server-14588a3d-4751-9789-b8aa-f2789bf17a31: compacting logs: from=1 to=92 2024-06-11T14:41:32.744Z [INFO] server-14588a3d-4751-9789-b8aa-f2789bf17a31: snapshot complete up to: index=99 2024-06-11T14:41:32.744Z [ERROR] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: failed to install snapshot: id=2-102-1718116892741 error="failed to connect to peer: fd5b1c39-9db8-4a03-e9fd-cf199ef5862c" 2024-06-11T14:41:32.744Z [ERROR] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: failed to send snapshot to: peer="{Voter server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c fd5b1c39-9db8-4a03-e9fd-cf199ef5862c}" error="failed to connect to peer: fd5b1c39-9db8-4a03-e9fd-cf199ef5862c" raft_test.go:1690: [INFO] Finished apply without behind follower 2024-06-11T14:41:32.755Z [ERROR] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: failed to get log: index=2 error="log not found" 2024-06-11T14:41:32.756Z [INFO] creating new snapshot: path=/tmp/raft974980324/snapshots/2-102-1718116892756.tmp 2024-06-11T14:41:32.757Z [INFO] server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2024-06-11T14:41:32.757Z [INFO] server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c: copied to local snapshot: bytes=693 2024-06-11T14:41:32.759Z [INFO] server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c: snapshot restore progress: id=2-102-1718116892756 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-11T14:41:32.759Z [INFO] server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c: Installed remote snapshot 2024-06-11T14:41:32.759Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: pipelining replication: peer="{Voter server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c fd5b1c39-9db8-4a03-e9fd-cf199ef5862c}" 2024-06-11T14:41:32.764Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: aborting pipeline replication: peer="{Voter server-14588a3d-4751-9789-b8aa-f2789bf17a31 14588a3d-4751-9789-b8aa-f2789bf17a31}" 2024-06-11T14:41:32.764Z [INFO] server-b9f5083a-2171-4250-b62d-f6853f4efb0e: aborting pipeline replication: peer="{Voter server-fd5b1c39-9db8-4a03-e9fd-cf199ef5862c fd5b1c39-9db8-4a03-e9fd-cf199ef5862c}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.32s) === RUN TestRaft_ReJoinFollower 2024-06-11T14:41:32.767Z [INFO] server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 Address:6d61a54d-7658-6f8a-8a8e-f34ca9c08296} {Suffrage:Voter ID:server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 Address:aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5} {Suffrage:Voter ID:server-02d3eec2-2a28-be80-148a-65b185960ba0 Address:02d3eec2-2a28-be80-148a-65b185960ba0}]" 2024-06-11T14:41:32.767Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 Address:6d61a54d-7658-6f8a-8a8e-f34ca9c08296} {Suffrage:Voter ID:server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 Address:aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5} {Suffrage:Voter ID:server-02d3eec2-2a28-be80-148a-65b185960ba0 Address:02d3eec2-2a28-be80-148a-65b185960ba0}]" 2024-06-11T14:41:32.767Z [INFO] server-02d3eec2-2a28-be80-148a-65b185960ba0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 Address:6d61a54d-7658-6f8a-8a8e-f34ca9c08296} {Suffrage:Voter ID:server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 Address:aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5} {Suffrage:Voter ID:server-02d3eec2-2a28-be80-148a-65b185960ba0 Address:02d3eec2-2a28-be80-148a-65b185960ba0}]" 2024-06-11T14:41:32.767Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:32.767Z [INFO] server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296: entering follower state: follower="Node at 6d61a54d-7658-6f8a-8a8e-f34ca9c08296 [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.768Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: entering follower state: follower="Node at aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.768Z [INFO] server-02d3eec2-2a28-be80-148a-65b185960ba0: entering follower state: follower="Node at 02d3eec2-2a28-be80-148a-65b185960ba0 [Follower]" leader-address= leader-id= 2024-06-11T14:41:32.825Z [WARN] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:32.825Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: entering candidate state: node="Node at aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 [Candidate]" term=2 2024-06-11T14:41:32.825Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: election won: term=2 tally=2 2024-06-11T14:41:32.825Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: entering leader state: leader="Node at aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 [Leader]" 2024-06-11T14:41:32.825Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: added peer, starting replication: peer=server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 2024-06-11T14:41:32.825Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: added peer, starting replication: peer=server-02d3eec2-2a28-be80-148a-65b185960ba0 2024-06-11T14:41:32.825Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: pipelining replication: peer="{Voter server-02d3eec2-2a28-be80-148a-65b185960ba0 02d3eec2-2a28-be80-148a-65b185960ba0}" 2024-06-11T14:41:32.825Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: pipelining replication: peer="{Voter server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 6d61a54d-7658-6f8a-8a8e-f34ca9c08296}" 2024-06-11T14:41:32.931Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:32.825423119 +0000 UTC m=+99.408502071 (1 nodes), 57.502525ms from start of poll, 58.151941ms from cluster start. Timeout at 2024-06-11 14:41:32.931574609 +0000 UTC m=+99.514653521, 106.15145ms after stability 2024-06-11T14:41:32.937Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:33.043Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:32.937776905 +0000 UTC m=+99.520855857 (2 nodes), 2.32µs from start of poll, 170.505727ms from cluster start. Timeout at 2024-06-11 14:41:33.043922035 +0000 UTC m=+99.627000987, 106.14513ms after stability 2024-06-11T14:41:33.044Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: updating configuration: command=RemoveServer server-id=server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 server-addr= servers="[{Suffrage:Voter ID:server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 Address:aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5} {Suffrage:Voter ID:server-02d3eec2-2a28-be80-148a-65b185960ba0 Address:02d3eec2-2a28-be80-148a-65b185960ba0}]" 2024-06-11T14:41:33.044Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: removed peer, stopping replication: peer=server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 last-index=3 2024-06-11T14:41:33.044Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: aborting pipeline replication: peer="{Voter server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 6d61a54d-7658-6f8a-8a8e-f34ca9c08296}" 2024-06-11T14:41:33.130Z [WARN] server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2024-06-11T14:41:33.150Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: updating configuration: command=AddVoter server-id=server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 server-addr=6d61a54d-7658-6f8a-8a8e-f34ca9c08296 servers="[{Suffrage:Voter ID:server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5 Address:aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5} {Suffrage:Voter ID:server-02d3eec2-2a28-be80-148a-65b185960ba0 Address:02d3eec2-2a28-be80-148a-65b185960ba0} {Suffrage:Voter ID:server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 Address:6d61a54d-7658-6f8a-8a8e-f34ca9c08296}]" 2024-06-11T14:41:33.150Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: added peer, starting replication: peer=server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 2024-06-11T14:41:33.150Z [WARN] server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2024-06-11T14:41:33.150Z [WARN] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: appendEntries rejected, sending older logs: peer="{Voter server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 6d61a54d-7658-6f8a-8a8e-f34ca9c08296}" next=4 2024-06-11T14:41:33.151Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: pipelining replication: peer="{Voter server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 6d61a54d-7658-6f8a-8a8e-f34ca9c08296}" 2024-06-11T14:41:33.151Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:33.256Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:33.151084209 +0000 UTC m=+99.734163161 (1 nodes), 1.16µs from start of poll, 383.813031ms from cluster start. Timeout at 2024-06-11 14:41:33.256378969 +0000 UTC m=+99.839457921, 105.29476ms after stability 2024-06-11T14:41:33.256Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: aborting pipeline replication: peer="{Voter server-02d3eec2-2a28-be80-148a-65b185960ba0 02d3eec2-2a28-be80-148a-65b185960ba0}" 2024-06-11T14:41:33.256Z [INFO] server-aaa0cefb-6f1e-5d6a-b96b-a2a8c7a801a5: aborting pipeline replication: peer="{Voter server-6d61a54d-7658-6f8a-8a8e-f34ca9c08296 6d61a54d-7658-6f8a-8a8e-f34ca9c08296}" --- PASS: TestRaft_ReJoinFollower (0.49s) === RUN TestRaft_LeaderLeaseExpire 2024-06-11T14:41:33.258Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0 Address:34a8d32f-0c49-75d4-7a8f-2f4617625ce0} {Suffrage:Voter ID:server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 Address:9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}]" 2024-06-11T14:41:33.258Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: entering follower state: follower="Node at 34a8d32f-0c49-75d4-7a8f-2f4617625ce0 [Follower]" leader-address= leader-id= 2024-06-11T14:41:33.258Z [INFO] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0 Address:34a8d32f-0c49-75d4-7a8f-2f4617625ce0} {Suffrage:Voter ID:server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 Address:9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}]" 2024-06-11T14:41:33.259Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:33.259Z [INFO] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: entering follower state: follower="Node at 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 [Follower]" leader-address= leader-id= 2024-06-11T14:41:33.314Z [WARN] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:33.314Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: entering candidate state: node="Node at 34a8d32f-0c49-75d4-7a8f-2f4617625ce0 [Candidate]" term=2 2024-06-11T14:41:33.314Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: election won: term=2 tally=2 2024-06-11T14:41:33.314Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: entering leader state: leader="Node at 34a8d32f-0c49-75d4-7a8f-2f4617625ce0 [Leader]" 2024-06-11T14:41:33.314Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: added peer, starting replication: peer=server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 2024-06-11T14:41:33.314Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: pipelining replication: peer="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" 2024-06-11T14:41:33.419Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:33.314640427 +0000 UTC m=+99.897719339 (1 nodes), 55.508877ms from start of poll, 56.054737ms from cluster start. Timeout at 2024-06-11 14:41:33.419710276 +0000 UTC m=+100.002789228, 105.069889ms after stability 2024-06-11T14:41:33.424Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:33.530Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:33.424925808 +0000 UTC m=+100.008004720 (1 nodes), 1.4µs from start of poll, 166.340118ms from cluster start. Timeout at 2024-06-11 14:41:33.530003896 +0000 UTC m=+100.113082848, 105.078128ms after stability raft_test.go:1799: [INFO] Disconnecting Node at 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 [Follower] 2024-06-11T14:41:33.531Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to heartbeat to: peer=9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 backoff time=10ms error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" 2024-06-11T14:41:33.534Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to pipeline appendEntries: peer="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" error="append pipeline closed" 2024-06-11T14:41:33.535Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: aborting pipeline replication: peer="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" 2024-06-11T14:41:33.544Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to appendEntries to: peer="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" 2024-06-11T14:41:33.548Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to heartbeat to: peer=9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 backoff time=10ms error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" 2024-06-11T14:41:33.560Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to appendEntries to: peer="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" 2024-06-11T14:41:33.565Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to heartbeat to: peer=9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 backoff time=10ms error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" 2024-06-11T14:41:33.577Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to appendEntries to: peer="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" 2024-06-11T14:41:33.579Z [WARN] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to contact: server-id=server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 time=51.302268ms 2024-06-11T14:41:33.579Z [WARN] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to contact quorum of nodes, stepping down 2024-06-11T14:41:33.579Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: entering follower state: follower="Node at 34a8d32f-0c49-75d4-7a8f-2f4617625ce0 [Follower]" leader-address= leader-id= 2024-06-11T14:41:33.579Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:33.650Z [WARN] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: heartbeat timeout reached, starting election: last-leader-addr=34a8d32f-0c49-75d4-7a8f-2f4617625ce0 last-leader-id=server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0 2024-06-11T14:41:33.650Z [INFO] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: entering candidate state: node="Node at 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 [Candidate]" term=3 2024-06-11T14:41:33.650Z [ERROR] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: failed to make requestVote RPC: target="{Voter server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0 34a8d32f-0c49-75d4-7a8f-2f4617625ce0}" error="failed to connect to peer: 34a8d32f-0c49-75d4-7a8f-2f4617625ce0" term=3 2024-06-11T14:41:33.668Z [WARN] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:33.668Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: entering candidate state: node="Node at 34a8d32f-0c49-75d4-7a8f-2f4617625ce0 [Candidate]" term=3 2024-06-11T14:41:33.668Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to make requestVote RPC: target="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" term=3 2024-06-11T14:41:33.712Z [WARN] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: Election timeout reached, restarting election 2024-06-11T14:41:33.712Z [INFO] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: entering candidate state: node="Node at 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 [Candidate]" term=4 2024-06-11T14:41:33.712Z [ERROR] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: failed to make requestVote RPC: target="{Voter server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0 34a8d32f-0c49-75d4-7a8f-2f4617625ce0}" error="failed to connect to peer: 34a8d32f-0c49-75d4-7a8f-2f4617625ce0" term=4 2024-06-11T14:41:33.748Z [WARN] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: Election timeout reached, restarting election 2024-06-11T14:41:33.748Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: entering candidate state: node="Node at 34a8d32f-0c49-75d4-7a8f-2f4617625ce0 [Candidate]" term=4 2024-06-11T14:41:33.748Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to make requestVote RPC: target="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" term=4 2024-06-11T14:41:33.774Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:33.668976362 +0000 UTC m=+100.252055274 (0 nodes), 89.465692ms from start of poll, 410.390672ms from cluster start. Timeout at 2024-06-11 14:41:33.774020052 +0000 UTC m=+100.357098964, 105.04369ms after stability 2024-06-11T14:41:33.780Z [WARN] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: Election timeout reached, restarting election 2024-06-11T14:41:33.780Z [INFO] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: entering candidate state: node="Node at 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 [Candidate]" term=5 2024-06-11T14:41:33.780Z [ERROR] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: failed to make requestVote RPC: target="{Voter server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0 34a8d32f-0c49-75d4-7a8f-2f4617625ce0}" error="failed to connect to peer: 34a8d32f-0c49-75d4-7a8f-2f4617625ce0" term=5 2024-06-11T14:41:33.822Z [WARN] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: Election timeout reached, restarting election 2024-06-11T14:41:33.822Z [INFO] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: entering candidate state: node="Node at 34a8d32f-0c49-75d4-7a8f-2f4617625ce0 [Candidate]" term=5 2024-06-11T14:41:33.822Z [ERROR] server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0: failed to make requestVote RPC: target="{Voter server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0}" error="failed to connect to peer: 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0" term=5 2024-06-11T14:41:33.854Z [WARN] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: Election timeout reached, restarting election 2024-06-11T14:41:33.854Z [INFO] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: entering candidate state: node="Node at 9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0 [Candidate]" term=6 2024-06-11T14:41:33.854Z [ERROR] server-9d365fd8-2042-a6cf-3a77-3d0f3fc6afb0: failed to make requestVote RPC: target="{Voter server-34a8d32f-0c49-75d4-7a8f-2f4617625ce0 34a8d32f-0c49-75d4-7a8f-2f4617625ce0}" error="failed to connect to peer: 34a8d32f-0c49-75d4-7a8f-2f4617625ce0" term=6 --- PASS: TestRaft_LeaderLeaseExpire (0.62s) === RUN TestRaft_Barrier 2024-06-11T14:41:33.881Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a28b84d-5fbd-6022-cf67-aae8871b0b30 Address:6a28b84d-5fbd-6022-cf67-aae8871b0b30} {Suffrage:Voter ID:server-f2af058b-4544-910d-76dd-0ea1612740f2 Address:f2af058b-4544-910d-76dd-0ea1612740f2} {Suffrage:Voter ID:server-f0384713-0951-4b6d-38f9-e232cb9d36f3 Address:f0384713-0951-4b6d-38f9-e232cb9d36f3}]" 2024-06-11T14:41:33.882Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: entering follower state: follower="Node at 6a28b84d-5fbd-6022-cf67-aae8871b0b30 [Follower]" leader-address= leader-id= 2024-06-11T14:41:33.882Z [INFO] server-f2af058b-4544-910d-76dd-0ea1612740f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a28b84d-5fbd-6022-cf67-aae8871b0b30 Address:6a28b84d-5fbd-6022-cf67-aae8871b0b30} {Suffrage:Voter ID:server-f2af058b-4544-910d-76dd-0ea1612740f2 Address:f2af058b-4544-910d-76dd-0ea1612740f2} {Suffrage:Voter ID:server-f0384713-0951-4b6d-38f9-e232cb9d36f3 Address:f0384713-0951-4b6d-38f9-e232cb9d36f3}]" 2024-06-11T14:41:33.882Z [INFO] server-f2af058b-4544-910d-76dd-0ea1612740f2: entering follower state: follower="Node at f2af058b-4544-910d-76dd-0ea1612740f2 [Follower]" leader-address= leader-id= 2024-06-11T14:41:33.882Z [INFO] server-f0384713-0951-4b6d-38f9-e232cb9d36f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a28b84d-5fbd-6022-cf67-aae8871b0b30 Address:6a28b84d-5fbd-6022-cf67-aae8871b0b30} {Suffrage:Voter ID:server-f2af058b-4544-910d-76dd-0ea1612740f2 Address:f2af058b-4544-910d-76dd-0ea1612740f2} {Suffrage:Voter ID:server-f0384713-0951-4b6d-38f9-e232cb9d36f3 Address:f0384713-0951-4b6d-38f9-e232cb9d36f3}]" 2024-06-11T14:41:33.882Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:33.882Z [INFO] server-f0384713-0951-4b6d-38f9-e232cb9d36f3: entering follower state: follower="Node at f0384713-0951-4b6d-38f9-e232cb9d36f3 [Follower]" leader-address= leader-id= 2024-06-11T14:41:33.945Z [WARN] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:33.945Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: entering candidate state: node="Node at 6a28b84d-5fbd-6022-cf67-aae8871b0b30 [Candidate]" term=2 2024-06-11T14:41:33.945Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: election won: term=2 tally=2 2024-06-11T14:41:33.945Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: entering leader state: leader="Node at 6a28b84d-5fbd-6022-cf67-aae8871b0b30 [Leader]" 2024-06-11T14:41:33.945Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: added peer, starting replication: peer=server-f2af058b-4544-910d-76dd-0ea1612740f2 2024-06-11T14:41:33.945Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: added peer, starting replication: peer=server-f0384713-0951-4b6d-38f9-e232cb9d36f3 2024-06-11T14:41:33.945Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: pipelining replication: peer="{Voter server-f2af058b-4544-910d-76dd-0ea1612740f2 f2af058b-4544-910d-76dd-0ea1612740f2}" 2024-06-11T14:41:33.945Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: pipelining replication: peer="{Voter server-f0384713-0951-4b6d-38f9-e232cb9d36f3 f0384713-0951-4b6d-38f9-e232cb9d36f3}" 2024-06-11T14:41:34.051Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:33.945428707 +0000 UTC m=+100.528507659 (1 nodes), 62.808134ms from start of poll, 63.642424ms from cluster start. Timeout at 2024-06-11 14:41:34.051104974 +0000 UTC m=+100.634183926, 105.676267ms after stability 2024-06-11T14:41:34.071Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: aborting pipeline replication: peer="{Voter server-f0384713-0951-4b6d-38f9-e232cb9d36f3 f0384713-0951-4b6d-38f9-e232cb9d36f3}" 2024-06-11T14:41:34.071Z [INFO] server-6a28b84d-5fbd-6022-cf67-aae8871b0b30: aborting pipeline replication: peer="{Voter server-f2af058b-4544-910d-76dd-0ea1612740f2 f2af058b-4544-910d-76dd-0ea1612740f2}" --- PASS: TestRaft_Barrier (0.19s) === RUN TestRaft_VerifyLeader 2024-06-11T14:41:34.073Z [INFO] server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b Address:daca0bb9-86d3-bbe7-4f22-ae459dfba79b} {Suffrage:Voter ID:server-d21626fc-4f12-877f-21ac-fd67b4bba526 Address:d21626fc-4f12-877f-21ac-fd67b4bba526} {Suffrage:Voter ID:server-2f284220-e97d-4b1c-2b58-8ad54cfda791 Address:2f284220-e97d-4b1c-2b58-8ad54cfda791}]" 2024-06-11T14:41:34.073Z [INFO] server-d21626fc-4f12-877f-21ac-fd67b4bba526: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b Address:daca0bb9-86d3-bbe7-4f22-ae459dfba79b} {Suffrage:Voter ID:server-d21626fc-4f12-877f-21ac-fd67b4bba526 Address:d21626fc-4f12-877f-21ac-fd67b4bba526} {Suffrage:Voter ID:server-2f284220-e97d-4b1c-2b58-8ad54cfda791 Address:2f284220-e97d-4b1c-2b58-8ad54cfda791}]" 2024-06-11T14:41:34.075Z [INFO] server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b: entering follower state: follower="Node at daca0bb9-86d3-bbe7-4f22-ae459dfba79b [Follower]" leader-address= leader-id= 2024-06-11T14:41:34.075Z [INFO] server-d21626fc-4f12-877f-21ac-fd67b4bba526: entering follower state: follower="Node at d21626fc-4f12-877f-21ac-fd67b4bba526 [Follower]" leader-address= leader-id= 2024-06-11T14:41:34.076Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b Address:daca0bb9-86d3-bbe7-4f22-ae459dfba79b} {Suffrage:Voter ID:server-d21626fc-4f12-877f-21ac-fd67b4bba526 Address:d21626fc-4f12-877f-21ac-fd67b4bba526} {Suffrage:Voter ID:server-2f284220-e97d-4b1c-2b58-8ad54cfda791 Address:2f284220-e97d-4b1c-2b58-8ad54cfda791}]" 2024-06-11T14:41:34.076Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:34.076Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: entering follower state: follower="Node at 2f284220-e97d-4b1c-2b58-8ad54cfda791 [Follower]" leader-address= leader-id= 2024-06-11T14:41:34.146Z [WARN] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:34.146Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: entering candidate state: node="Node at 2f284220-e97d-4b1c-2b58-8ad54cfda791 [Candidate]" term=2 2024-06-11T14:41:34.146Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: election won: term=2 tally=2 2024-06-11T14:41:34.146Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: entering leader state: leader="Node at 2f284220-e97d-4b1c-2b58-8ad54cfda791 [Leader]" 2024-06-11T14:41:34.146Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: added peer, starting replication: peer=server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b 2024-06-11T14:41:34.146Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: added peer, starting replication: peer=server-d21626fc-4f12-877f-21ac-fd67b4bba526 2024-06-11T14:41:34.146Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: pipelining replication: peer="{Voter server-d21626fc-4f12-877f-21ac-fd67b4bba526 d21626fc-4f12-877f-21ac-fd67b4bba526}" 2024-06-11T14:41:34.146Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: pipelining replication: peer="{Voter server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b daca0bb9-86d3-bbe7-4f22-ae459dfba79b}" 2024-06-11T14:41:34.252Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:34.147114549 +0000 UTC m=+100.730193501 (1 nodes), 70.314783ms from start of poll, 73.586984ms from cluster start. Timeout at 2024-06-11 14:41:34.252669541 +0000 UTC m=+100.835748493, 105.554992ms after stability 2024-06-11T14:41:34.253Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: aborting pipeline replication: peer="{Voter server-d21626fc-4f12-877f-21ac-fd67b4bba526 d21626fc-4f12-877f-21ac-fd67b4bba526}" 2024-06-11T14:41:34.253Z [INFO] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: aborting pipeline replication: peer="{Voter server-daca0bb9-86d3-bbe7-4f22-ae459dfba79b daca0bb9-86d3-bbe7-4f22-ae459dfba79b}" 2024-06-11T14:41:34.754Z [ERROR] server-2f284220-e97d-4b1c-2b58-8ad54cfda791: failed to heartbeat to: peer=d21626fc-4f12-877f-21ac-fd67b4bba526 backoff time=10ms error="command timed out" --- PASS: TestRaft_VerifyLeader (0.69s) === RUN TestRaft_VerifyLeader_Single 2024-06-11T14:41:34.761Z [INFO] server-d5c03a3c-063e-cee8-cbee-f7ef2388b4bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5c03a3c-063e-cee8-cbee-f7ef2388b4bb Address:d5c03a3c-063e-cee8-cbee-f7ef2388b4bb}]" 2024-06-11T14:41:34.761Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:34.761Z [INFO] server-d5c03a3c-063e-cee8-cbee-f7ef2388b4bb: entering follower state: follower="Node at d5c03a3c-063e-cee8-cbee-f7ef2388b4bb [Follower]" leader-address= leader-id= 2024-06-11T14:41:34.815Z [WARN] server-d5c03a3c-063e-cee8-cbee-f7ef2388b4bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:34.815Z [INFO] server-d5c03a3c-063e-cee8-cbee-f7ef2388b4bb: entering candidate state: node="Node at d5c03a3c-063e-cee8-cbee-f7ef2388b4bb [Candidate]" term=2 2024-06-11T14:41:34.815Z [INFO] server-d5c03a3c-063e-cee8-cbee-f7ef2388b4bb: election won: term=2 tally=1 2024-06-11T14:41:34.815Z [INFO] server-d5c03a3c-063e-cee8-cbee-f7ef2388b4bb: entering leader state: leader="Node at d5c03a3c-063e-cee8-cbee-f7ef2388b4bb [Leader]" 2024-06-11T14:41:34.921Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:34.815774702 +0000 UTC m=+101.398853654 (1 nodes), 54.58727ms from start of poll, 54.8645ms from cluster start. Timeout at 2024-06-11 14:41:34.921289055 +0000 UTC m=+101.504368007, 105.514353ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.16s) === RUN TestRaft_VerifyLeader_Fail 2024-06-11T14:41:34.923Z [INFO] server-efd91e01-e28a-ccf8-0f4b-114b6458fa67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efd91e01-e28a-ccf8-0f4b-114b6458fa67 Address:efd91e01-e28a-ccf8-0f4b-114b6458fa67} {Suffrage:Voter ID:server-70ef880b-f529-1b13-d2af-1dcffdfec499 Address:70ef880b-f529-1b13-d2af-1dcffdfec499}]" 2024-06-11T14:41:34.923Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efd91e01-e28a-ccf8-0f4b-114b6458fa67 Address:efd91e01-e28a-ccf8-0f4b-114b6458fa67} {Suffrage:Voter ID:server-70ef880b-f529-1b13-d2af-1dcffdfec499 Address:70ef880b-f529-1b13-d2af-1dcffdfec499}]" 2024-06-11T14:41:34.923Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:34.923Z [INFO] server-efd91e01-e28a-ccf8-0f4b-114b6458fa67: entering follower state: follower="Node at efd91e01-e28a-ccf8-0f4b-114b6458fa67 [Follower]" leader-address= leader-id= 2024-06-11T14:41:34.923Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: entering follower state: follower="Node at 70ef880b-f529-1b13-d2af-1dcffdfec499 [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.002Z [WARN] server-70ef880b-f529-1b13-d2af-1dcffdfec499: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:35.002Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: entering candidate state: node="Node at 70ef880b-f529-1b13-d2af-1dcffdfec499 [Candidate]" term=2 2024-06-11T14:41:35.002Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: election won: term=2 tally=2 2024-06-11T14:41:35.002Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: entering leader state: leader="Node at 70ef880b-f529-1b13-d2af-1dcffdfec499 [Leader]" 2024-06-11T14:41:35.002Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: added peer, starting replication: peer=server-efd91e01-e28a-ccf8-0f4b-114b6458fa67 2024-06-11T14:41:35.002Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: pipelining replication: peer="{Voter server-efd91e01-e28a-ccf8-0f4b-114b6458fa67 efd91e01-e28a-ccf8-0f4b-114b6458fa67}" 2024-06-11T14:41:35.108Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:35.003052025 +0000 UTC m=+101.586130977 (1 nodes), 79.722643ms from start of poll, 80.185627ms from cluster start. Timeout at 2024-06-11 14:41:35.108558538 +0000 UTC m=+101.691637490, 105.506513ms after stability 2024-06-11T14:41:35.108Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:35.214Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:35.108736532 +0000 UTC m=+101.691815484 (1 nodes), 1.36µs from start of poll, 185.870134ms from cluster start. Timeout at 2024-06-11 14:41:35.214216926 +0000 UTC m=+101.797295838, 105.480354ms after stability 2024-06-11T14:41:35.215Z [ERROR] server-70ef880b-f529-1b13-d2af-1dcffdfec499: peer has newer term, stopping replication: peer="{Voter server-efd91e01-e28a-ccf8-0f4b-114b6458fa67 efd91e01-e28a-ccf8-0f4b-114b6458fa67}" 2024-06-11T14:41:35.215Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: aborting pipeline replication: peer="{Voter server-efd91e01-e28a-ccf8-0f4b-114b6458fa67 efd91e01-e28a-ccf8-0f4b-114b6458fa67}" 2024-06-11T14:41:35.215Z [INFO] server-70ef880b-f529-1b13-d2af-1dcffdfec499: entering follower state: follower="Node at 70ef880b-f529-1b13-d2af-1dcffdfec499 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.29s) === RUN TestRaft_VerifyLeader_PartialConnect 2024-06-11T14:41:35.217Z [INFO] server-cc158ab0-34e2-79a0-74f0-212939f1ef7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc158ab0-34e2-79a0-74f0-212939f1ef7c Address:cc158ab0-34e2-79a0-74f0-212939f1ef7c} {Suffrage:Voter ID:server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9 Address:1999cf8c-6a1b-1556-294e-e0b6fa70d8e9} {Suffrage:Voter ID:server-34fe3831-18ce-160f-f81d-4e15605355dd Address:34fe3831-18ce-160f-f81d-4e15605355dd}]" 2024-06-11T14:41:35.218Z [INFO] server-cc158ab0-34e2-79a0-74f0-212939f1ef7c: entering follower state: follower="Node at cc158ab0-34e2-79a0-74f0-212939f1ef7c [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.218Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc158ab0-34e2-79a0-74f0-212939f1ef7c Address:cc158ab0-34e2-79a0-74f0-212939f1ef7c} {Suffrage:Voter ID:server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9 Address:1999cf8c-6a1b-1556-294e-e0b6fa70d8e9} {Suffrage:Voter ID:server-34fe3831-18ce-160f-f81d-4e15605355dd Address:34fe3831-18ce-160f-f81d-4e15605355dd}]" 2024-06-11T14:41:35.218Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: entering follower state: follower="Node at 1999cf8c-6a1b-1556-294e-e0b6fa70d8e9 [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.218Z [INFO] server-34fe3831-18ce-160f-f81d-4e15605355dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc158ab0-34e2-79a0-74f0-212939f1ef7c Address:cc158ab0-34e2-79a0-74f0-212939f1ef7c} {Suffrage:Voter ID:server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9 Address:1999cf8c-6a1b-1556-294e-e0b6fa70d8e9} {Suffrage:Voter ID:server-34fe3831-18ce-160f-f81d-4e15605355dd Address:34fe3831-18ce-160f-f81d-4e15605355dd}]" 2024-06-11T14:41:35.218Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:35.218Z [INFO] server-34fe3831-18ce-160f-f81d-4e15605355dd: entering follower state: follower="Node at 34fe3831-18ce-160f-f81d-4e15605355dd [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.284Z [WARN] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:35.284Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: entering candidate state: node="Node at 1999cf8c-6a1b-1556-294e-e0b6fa70d8e9 [Candidate]" term=2 2024-06-11T14:41:35.284Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: election won: term=2 tally=2 2024-06-11T14:41:35.284Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: entering leader state: leader="Node at 1999cf8c-6a1b-1556-294e-e0b6fa70d8e9 [Leader]" 2024-06-11T14:41:35.284Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: added peer, starting replication: peer=server-cc158ab0-34e2-79a0-74f0-212939f1ef7c 2024-06-11T14:41:35.284Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: added peer, starting replication: peer=server-34fe3831-18ce-160f-f81d-4e15605355dd 2024-06-11T14:41:35.284Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: pipelining replication: peer="{Voter server-34fe3831-18ce-160f-f81d-4e15605355dd 34fe3831-18ce-160f-f81d-4e15605355dd}" 2024-06-11T14:41:35.284Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: pipelining replication: peer="{Voter server-cc158ab0-34e2-79a0-74f0-212939f1ef7c cc158ab0-34e2-79a0-74f0-212939f1ef7c}" 2024-06-11T14:41:35.389Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:35.284606626 +0000 UTC m=+101.867685578 (1 nodes), 65.768267ms from start of poll, 66.882427ms from cluster start. Timeout at 2024-06-11 14:41:35.389637476 +0000 UTC m=+101.972716428, 105.03085ms after stability 2024-06-11T14:41:35.395Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:35.501Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:35.395785894 +0000 UTC m=+101.978864806 (2 nodes), 2.24µs from start of poll, 178.061655ms from cluster start. Timeout at 2024-06-11 14:41:35.501268288 +0000 UTC m=+102.084347240, 105.482434ms after stability raft_test.go:1968: [INFO] Disconnecting Node at cc158ab0-34e2-79a0-74f0-212939f1ef7c [Follower] 2024-06-11T14:41:35.501Z [ERROR] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: failed to heartbeat to: peer=cc158ab0-34e2-79a0-74f0-212939f1ef7c backoff time=10ms error="failed to connect to peer: cc158ab0-34e2-79a0-74f0-212939f1ef7c" 2024-06-11T14:41:35.501Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: aborting pipeline replication: peer="{Voter server-34fe3831-18ce-160f-f81d-4e15605355dd 34fe3831-18ce-160f-f81d-4e15605355dd}" 2024-06-11T14:41:35.501Z [INFO] server-1999cf8c-6a1b-1556-294e-e0b6fa70d8e9: aborting pipeline replication: peer="{Voter server-cc158ab0-34e2-79a0-74f0-212939f1ef7c cc158ab0-34e2-79a0-74f0-212939f1ef7c}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s) === RUN TestRaft_NotifyCh 2024-06-11T14:41:35.503Z [INFO] server-4f02c548-98fe-b56b-bb88-be1ba4cf6243: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f02c548-98fe-b56b-bb88-be1ba4cf6243 Address:4f02c548-98fe-b56b-bb88-be1ba4cf6243}]" 2024-06-11T14:41:35.503Z [INFO] server-4f02c548-98fe-b56b-bb88-be1ba4cf6243: entering follower state: follower="Node at 4f02c548-98fe-b56b-bb88-be1ba4cf6243 [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.574Z [WARN] server-4f02c548-98fe-b56b-bb88-be1ba4cf6243: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:35.574Z [INFO] server-4f02c548-98fe-b56b-bb88-be1ba4cf6243: entering candidate state: node="Node at 4f02c548-98fe-b56b-bb88-be1ba4cf6243 [Candidate]" term=2 2024-06-11T14:41:35.574Z [INFO] server-4f02c548-98fe-b56b-bb88-be1ba4cf6243: election won: term=2 tally=1 2024-06-11T14:41:35.574Z [INFO] server-4f02c548-98fe-b56b-bb88-be1ba4cf6243: entering leader state: leader="Node at 4f02c548-98fe-b56b-bb88-be1ba4cf6243 [Leader]" --- PASS: TestRaft_NotifyCh (0.07s) === RUN TestRaft_AppendEntry 2024-06-11T14:41:35.576Z [INFO] server-bc61ed69-ea7d-0902-f5f7-364b439bafb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc61ed69-ea7d-0902-f5f7-364b439bafb9 Address:bc61ed69-ea7d-0902-f5f7-364b439bafb9} {Suffrage:Voter ID:server-bd2c8283-17b2-55be-b757-35bd41dda8b6 Address:bd2c8283-17b2-55be-b757-35bd41dda8b6} {Suffrage:Voter ID:server-c6dde733-3fc7-931b-3cc7-35f66f520e0a Address:c6dde733-3fc7-931b-3cc7-35f66f520e0a}]" 2024-06-11T14:41:35.576Z [INFO] server-bd2c8283-17b2-55be-b757-35bd41dda8b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc61ed69-ea7d-0902-f5f7-364b439bafb9 Address:bc61ed69-ea7d-0902-f5f7-364b439bafb9} {Suffrage:Voter ID:server-bd2c8283-17b2-55be-b757-35bd41dda8b6 Address:bd2c8283-17b2-55be-b757-35bd41dda8b6} {Suffrage:Voter ID:server-c6dde733-3fc7-931b-3cc7-35f66f520e0a Address:c6dde733-3fc7-931b-3cc7-35f66f520e0a}]" 2024-06-11T14:41:35.576Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc61ed69-ea7d-0902-f5f7-364b439bafb9 Address:bc61ed69-ea7d-0902-f5f7-364b439bafb9} {Suffrage:Voter ID:server-bd2c8283-17b2-55be-b757-35bd41dda8b6 Address:bd2c8283-17b2-55be-b757-35bd41dda8b6} {Suffrage:Voter ID:server-c6dde733-3fc7-931b-3cc7-35f66f520e0a Address:c6dde733-3fc7-931b-3cc7-35f66f520e0a}]" 2024-06-11T14:41:35.576Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:35.577Z [INFO] server-bc61ed69-ea7d-0902-f5f7-364b439bafb9: entering follower state: follower="Node at bc61ed69-ea7d-0902-f5f7-364b439bafb9 [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.577Z [INFO] server-bd2c8283-17b2-55be-b757-35bd41dda8b6: entering follower state: follower="Node at bd2c8283-17b2-55be-b757-35bd41dda8b6 [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.577Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: entering follower state: follower="Node at c6dde733-3fc7-931b-3cc7-35f66f520e0a [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.647Z [WARN] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:35.648Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: entering candidate state: node="Node at c6dde733-3fc7-931b-3cc7-35f66f520e0a [Candidate]" term=2 2024-06-11T14:41:35.648Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: election won: term=2 tally=2 2024-06-11T14:41:35.648Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: entering leader state: leader="Node at c6dde733-3fc7-931b-3cc7-35f66f520e0a [Leader]" 2024-06-11T14:41:35.648Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: added peer, starting replication: peer=server-bc61ed69-ea7d-0902-f5f7-364b439bafb9 2024-06-11T14:41:35.648Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: added peer, starting replication: peer=server-bd2c8283-17b2-55be-b757-35bd41dda8b6 2024-06-11T14:41:35.648Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: pipelining replication: peer="{Voter server-bd2c8283-17b2-55be-b757-35bd41dda8b6 bd2c8283-17b2-55be-b757-35bd41dda8b6}" 2024-06-11T14:41:35.648Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: pipelining replication: peer="{Voter server-bc61ed69-ea7d-0902-f5f7-364b439bafb9 bc61ed69-ea7d-0902-f5f7-364b439bafb9}" 2024-06-11T14:41:35.753Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:35.648451457 +0000 UTC m=+102.231530369 (2 nodes), 71.473621ms from start of poll, 72.067679ms from cluster start. Timeout at 2024-06-11 14:41:35.753753858 +0000 UTC m=+102.336832770, 105.302401ms after stability 2024-06-11T14:41:35.753Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:35.859Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:35.75397421 +0000 UTC m=+102.337053162 (1 nodes), 1.56µs from start of poll, 177.590472ms from cluster start. Timeout at 2024-06-11 14:41:35.859510922 +0000 UTC m=+102.442589834, 105.536672ms after stability 2024-06-11T14:41:35.859Z [WARN] server-bc61ed69-ea7d-0902-f5f7-364b439bafb9: clearing log suffix: from=1 to=2 2024-06-11T14:41:35.859Z [WARN] server-bc61ed69-ea7d-0902-f5f7-364b439bafb9: skipping application of old log: index=1 2024-06-11T14:41:35.859Z [WARN] server-bc61ed69-ea7d-0902-f5f7-364b439bafb9: skipping application of old log: index=1 2024-06-11T14:41:35.859Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: aborting pipeline replication: peer="{Voter server-bd2c8283-17b2-55be-b757-35bd41dda8b6 bd2c8283-17b2-55be-b757-35bd41dda8b6}" 2024-06-11T14:41:35.859Z [INFO] server-c6dde733-3fc7-931b-3cc7-35f66f520e0a: aborting pipeline replication: peer="{Voter server-bc61ed69-ea7d-0902-f5f7-364b439bafb9 bc61ed69-ea7d-0902-f5f7-364b439bafb9}" --- PASS: TestRaft_AppendEntry (0.29s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2024-06-11T14:41:35.862Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-386a30b9-ccdb-548d-07de-80c4c182c326 Address:386a30b9-ccdb-548d-07de-80c4c182c326} {Suffrage:Voter ID:server-dfb18aed-66d3-b472-a0ab-b5850746b957 Address:dfb18aed-66d3-b472-a0ab-b5850746b957} {Suffrage:Voter ID:server-ce751605-1fe1-c068-4491-1f0985d3f64b Address:ce751605-1fe1-c068-4491-1f0985d3f64b}]" 2024-06-11T14:41:35.862Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: entering follower state: follower="Node at 386a30b9-ccdb-548d-07de-80c4c182c326 [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.862Z [INFO] server-dfb18aed-66d3-b472-a0ab-b5850746b957: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-386a30b9-ccdb-548d-07de-80c4c182c326 Address:386a30b9-ccdb-548d-07de-80c4c182c326} {Suffrage:Voter ID:server-dfb18aed-66d3-b472-a0ab-b5850746b957 Address:dfb18aed-66d3-b472-a0ab-b5850746b957} {Suffrage:Voter ID:server-ce751605-1fe1-c068-4491-1f0985d3f64b Address:ce751605-1fe1-c068-4491-1f0985d3f64b}]" 2024-06-11T14:41:35.862Z [INFO] server-dfb18aed-66d3-b472-a0ab-b5850746b957: entering follower state: follower="Node at dfb18aed-66d3-b472-a0ab-b5850746b957 [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.862Z [INFO] server-ce751605-1fe1-c068-4491-1f0985d3f64b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-386a30b9-ccdb-548d-07de-80c4c182c326 Address:386a30b9-ccdb-548d-07de-80c4c182c326} {Suffrage:Voter ID:server-dfb18aed-66d3-b472-a0ab-b5850746b957 Address:dfb18aed-66d3-b472-a0ab-b5850746b957} {Suffrage:Voter ID:server-ce751605-1fe1-c068-4491-1f0985d3f64b Address:ce751605-1fe1-c068-4491-1f0985d3f64b}]" 2024-06-11T14:41:35.863Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:35.863Z [INFO] server-ce751605-1fe1-c068-4491-1f0985d3f64b: entering follower state: follower="Node at ce751605-1fe1-c068-4491-1f0985d3f64b [Follower]" leader-address= leader-id= 2024-06-11T14:41:35.924Z [WARN] server-386a30b9-ccdb-548d-07de-80c4c182c326: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:35.924Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: entering candidate state: node="Node at 386a30b9-ccdb-548d-07de-80c4c182c326 [Candidate]" term=2 2024-06-11T14:41:35.924Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: election won: term=2 tally=2 2024-06-11T14:41:35.924Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: entering leader state: leader="Node at 386a30b9-ccdb-548d-07de-80c4c182c326 [Leader]" 2024-06-11T14:41:35.924Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: added peer, starting replication: peer=server-dfb18aed-66d3-b472-a0ab-b5850746b957 2024-06-11T14:41:35.924Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: added peer, starting replication: peer=server-ce751605-1fe1-c068-4491-1f0985d3f64b 2024-06-11T14:41:35.924Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: pipelining replication: peer="{Voter server-ce751605-1fe1-c068-4491-1f0985d3f64b ce751605-1fe1-c068-4491-1f0985d3f64b}" 2024-06-11T14:41:35.924Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: pipelining replication: peer="{Voter server-dfb18aed-66d3-b472-a0ab-b5850746b957 dfb18aed-66d3-b472-a0ab-b5850746b957}" 2024-06-11T14:41:36.029Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:35.924643012 +0000 UTC m=+102.507721964 (2 nodes), 61.610537ms from start of poll, 62.544344ms from cluster start. Timeout at 2024-06-11 14:41:36.029660502 +0000 UTC m=+102.612739454, 105.01749ms after stability 2024-06-11T14:41:36.029Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:36.136Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:36.029814697 +0000 UTC m=+102.612893649 (1 nodes), 1.72µs from start of poll, 167.716029ms from cluster start. Timeout at 2024-06-11 14:41:36.135985106 +0000 UTC m=+102.719064058, 106.170409ms after stability 2024-06-11T14:41:36.136Z [WARN] server-ce751605-1fe1-c068-4491-1f0985d3f64b: rejecting vote request since we have a leader: from=dfb18aed-66d3-b472-a0ab-b5850746b957 leader=386a30b9-ccdb-548d-07de-80c4c182c326 leader-id=server-386a30b9-ccdb-548d-07de-80c4c182c326 2024-06-11T14:41:36.136Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: aborting pipeline replication: peer="{Voter server-ce751605-1fe1-c068-4491-1f0985d3f64b ce751605-1fe1-c068-4491-1f0985d3f64b}" 2024-06-11T14:41:36.136Z [INFO] server-386a30b9-ccdb-548d-07de-80c4c182c326: aborting pipeline replication: peer="{Voter server-dfb18aed-66d3-b472-a0ab-b5850746b957 dfb18aed-66d3-b472-a0ab-b5850746b957}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC 2024-06-11T14:41:36.138Z [INFO] server-60f359a0-5d4d-938f-1c0e-dd64bc19377b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60f359a0-5d4d-938f-1c0e-dd64bc19377b Address:60f359a0-5d4d-938f-1c0e-dd64bc19377b} {Suffrage:Voter ID:server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145 Address:cd23bfbd-7d6c-b016-e3f8-0966cf0c2145} {Suffrage:Voter ID:server-a2f969c5-237e-42a3-b572-7ee12799de57 Address:a2f969c5-237e-42a3-b572-7ee12799de57}]" 2024-06-11T14:41:36.138Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60f359a0-5d4d-938f-1c0e-dd64bc19377b Address:60f359a0-5d4d-938f-1c0e-dd64bc19377b} {Suffrage:Voter ID:server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145 Address:cd23bfbd-7d6c-b016-e3f8-0966cf0c2145} {Suffrage:Voter ID:server-a2f969c5-237e-42a3-b572-7ee12799de57 Address:a2f969c5-237e-42a3-b572-7ee12799de57}]" 2024-06-11T14:41:36.139Z [INFO] server-a2f969c5-237e-42a3-b572-7ee12799de57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60f359a0-5d4d-938f-1c0e-dd64bc19377b Address:60f359a0-5d4d-938f-1c0e-dd64bc19377b} {Suffrage:Voter ID:server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145 Address:cd23bfbd-7d6c-b016-e3f8-0966cf0c2145} {Suffrage:Voter ID:server-a2f969c5-237e-42a3-b572-7ee12799de57 Address:a2f969c5-237e-42a3-b572-7ee12799de57}]" 2024-06-11T14:41:36.139Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:36.139Z [INFO] server-60f359a0-5d4d-938f-1c0e-dd64bc19377b: entering follower state: follower="Node at 60f359a0-5d4d-938f-1c0e-dd64bc19377b [Follower]" leader-address= leader-id= 2024-06-11T14:41:36.139Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: entering follower state: follower="Node at cd23bfbd-7d6c-b016-e3f8-0966cf0c2145 [Follower]" leader-address= leader-id= 2024-06-11T14:41:36.139Z [INFO] server-a2f969c5-237e-42a3-b572-7ee12799de57: entering follower state: follower="Node at a2f969c5-237e-42a3-b572-7ee12799de57 [Follower]" leader-address= leader-id= 2024-06-11T14:41:36.212Z [WARN] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:36.212Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: entering candidate state: node="Node at cd23bfbd-7d6c-b016-e3f8-0966cf0c2145 [Candidate]" term=2 2024-06-11T14:41:36.212Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: election won: term=2 tally=2 2024-06-11T14:41:36.212Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: entering leader state: leader="Node at cd23bfbd-7d6c-b016-e3f8-0966cf0c2145 [Leader]" 2024-06-11T14:41:36.212Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: added peer, starting replication: peer=server-60f359a0-5d4d-938f-1c0e-dd64bc19377b 2024-06-11T14:41:36.212Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: added peer, starting replication: peer=server-a2f969c5-237e-42a3-b572-7ee12799de57 2024-06-11T14:41:36.212Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: pipelining replication: peer="{Voter server-a2f969c5-237e-42a3-b572-7ee12799de57 a2f969c5-237e-42a3-b572-7ee12799de57}" 2024-06-11T14:41:36.212Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: pipelining replication: peer="{Voter server-60f359a0-5d4d-938f-1c0e-dd64bc19377b 60f359a0-5d4d-938f-1c0e-dd64bc19377b}" 2024-06-11T14:41:36.317Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:36.212585542 +0000 UTC m=+102.795664494 (2 nodes), 73.372273ms from start of poll, 73.965051ms from cluster start. Timeout at 2024-06-11 14:41:36.317762467 +0000 UTC m=+102.900841379, 105.176885ms after stability 2024-06-11T14:41:36.317Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:36.423Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:36.3179471 +0000 UTC m=+102.901026052 (1 nodes), 2.08µs from start of poll, 179.326609ms from cluster start. Timeout at 2024-06-11 14:41:36.42299831 +0000 UTC m=+103.006077262, 105.05121ms after stability 2024-06-11T14:41:36.423Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: aborting pipeline replication: peer="{Voter server-a2f969c5-237e-42a3-b572-7ee12799de57 a2f969c5-237e-42a3-b572-7ee12799de57}" 2024-06-11T14:41:36.423Z [INFO] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: aborting pipeline replication: peer="{Voter server-60f359a0-5d4d-938f-1c0e-dd64bc19377b 60f359a0-5d4d-938f-1c0e-dd64bc19377b}" 2024-06-11T14:41:36.923Z [ERROR] server-cd23bfbd-7d6c-b016-e3f8-0966cf0c2145: failed to heartbeat to: peer=a2f969c5-237e-42a3-b572-7ee12799de57 backoff time=10ms error="command timed out" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.79s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2024-06-11T14:41:36.926Z [INFO] server-f5414fa2-6d15-0497-98b4-0485bae72bf3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5414fa2-6d15-0497-98b4-0485bae72bf3 Address:f5414fa2-6d15-0497-98b4-0485bae72bf3} {Suffrage:Voter ID:server-8630d1d5-df01-4896-91b7-0367c93cc932 Address:8630d1d5-df01-4896-91b7-0367c93cc932} {Suffrage:Voter ID:server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f Address:0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f}]" 2024-06-11T14:41:36.926Z [INFO] server-f5414fa2-6d15-0497-98b4-0485bae72bf3: entering follower state: follower="Node at f5414fa2-6d15-0497-98b4-0485bae72bf3 [Follower]" leader-address= leader-id= 2024-06-11T14:41:36.926Z [INFO] server-8630d1d5-df01-4896-91b7-0367c93cc932: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5414fa2-6d15-0497-98b4-0485bae72bf3 Address:f5414fa2-6d15-0497-98b4-0485bae72bf3} {Suffrage:Voter ID:server-8630d1d5-df01-4896-91b7-0367c93cc932 Address:8630d1d5-df01-4896-91b7-0367c93cc932} {Suffrage:Voter ID:server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f Address:0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f}]" 2024-06-11T14:41:36.926Z [INFO] server-8630d1d5-df01-4896-91b7-0367c93cc932: entering follower state: follower="Node at 8630d1d5-df01-4896-91b7-0367c93cc932 [Follower]" leader-address= leader-id= 2024-06-11T14:41:36.926Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5414fa2-6d15-0497-98b4-0485bae72bf3 Address:f5414fa2-6d15-0497-98b4-0485bae72bf3} {Suffrage:Voter ID:server-8630d1d5-df01-4896-91b7-0367c93cc932 Address:8630d1d5-df01-4896-91b7-0367c93cc932} {Suffrage:Voter ID:server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f Address:0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f}]" 2024-06-11T14:41:36.926Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:36.926Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: entering follower state: follower="Node at 0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f [Follower]" leader-address= leader-id= 2024-06-11T14:41:36.999Z [WARN] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:36.999Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: entering candidate state: node="Node at 0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f [Candidate]" term=2 2024-06-11T14:41:36.999Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: election won: term=2 tally=2 2024-06-11T14:41:36.999Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: entering leader state: leader="Node at 0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f [Leader]" 2024-06-11T14:41:36.999Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: added peer, starting replication: peer=server-f5414fa2-6d15-0497-98b4-0485bae72bf3 2024-06-11T14:41:36.999Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: added peer, starting replication: peer=server-8630d1d5-df01-4896-91b7-0367c93cc932 2024-06-11T14:41:36.999Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: pipelining replication: peer="{Voter server-8630d1d5-df01-4896-91b7-0367c93cc932 8630d1d5-df01-4896-91b7-0367c93cc932}" 2024-06-11T14:41:37.000Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: pipelining replication: peer="{Voter server-f5414fa2-6d15-0497-98b4-0485bae72bf3 f5414fa2-6d15-0497-98b4-0485bae72bf3}" 2024-06-11T14:41:37.105Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:36.999760419 +0000 UTC m=+103.582839371 (1 nodes), 72.984687ms from start of poll, 73.806377ms from cluster start. Timeout at 2024-06-11 14:41:37.105013661 +0000 UTC m=+103.688092613, 105.253242ms after stability 2024-06-11T14:41:37.105Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:37.211Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:37.105207094 +0000 UTC m=+103.688286046 (1 nodes), 1.2µs from start of poll, 179.253052ms from cluster start. Timeout at 2024-06-11 14:41:37.21118175 +0000 UTC m=+103.794260662, 105.974616ms after stability 2024-06-11T14:41:37.211Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:37.317Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:37.211812727 +0000 UTC m=+103.794891679 (1 nodes), 1.2µs from start of poll, 285.858685ms from cluster start. Timeout at 2024-06-11 14:41:37.317286842 +0000 UTC m=+103.900365794, 105.474115ms after stability 2024-06-11T14:41:37.317Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:37.423Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:37.317730625 +0000 UTC m=+103.900809537 (1 nodes), 1.359µs from start of poll, 391.776543ms from cluster start. Timeout at 2024-06-11 14:41:37.423779119 +0000 UTC m=+104.006858031, 106.048494ms after stability 2024-06-11T14:41:37.424Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:37.529Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:37.42431298 +0000 UTC m=+104.007391932 (1 nodes), 1.24µs from start of poll, 498.358938ms from cluster start. Timeout at 2024-06-11 14:41:37.529487225 +0000 UTC m=+104.112566177, 105.174245ms after stability 2024-06-11T14:41:37.530Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:37.636Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:37.530057964 +0000 UTC m=+104.113136916 (1 nodes), 1.36µs from start of poll, 604.103922ms from cluster start. Timeout at 2024-06-11 14:41:37.636151416 +0000 UTC m=+104.219230368, 106.093452ms after stability 2024-06-11T14:41:37.741Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:37.846Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:37.741725407 +0000 UTC m=+104.324804359 (1 nodes), 1.56µs from start of poll, 815.771365ms from cluster start. Timeout at 2024-06-11 14:41:37.846755417 +0000 UTC m=+104.429834329, 105.02997ms after stability 2024-06-11T14:41:37.847Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: aborting pipeline replication: peer="{Voter server-8630d1d5-df01-4896-91b7-0367c93cc932 8630d1d5-df01-4896-91b7-0367c93cc932}" 2024-06-11T14:41:37.847Z [INFO] server-0a0e33ba-9dd1-edbe-0266-5fa21cdaa26f: aborting pipeline replication: peer="{Voter server-f5414fa2-6d15-0497-98b4-0485bae72bf3 f5414fa2-6d15-0497-98b4-0485bae72bf3}" --- PASS: TestRaft_LeaderID_Propagated (0.92s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2024-06-11T14:41:37.849Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4 Address:e7cf2fe7-d894-d86b-6c22-8384c04b25c4} {Suffrage:Voter ID:server-d6d6d090-adce-774d-38eb-faa36d3753d9 Address:d6d6d090-adce-774d-38eb-faa36d3753d9} {Suffrage:Voter ID:server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 Address:c1567b57-f4f8-0368-f0d2-ae0a5fe32d05}]" 2024-06-11T14:41:37.850Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: entering follower state: follower="Node at e7cf2fe7-d894-d86b-6c22-8384c04b25c4 [Follower]" leader-address= leader-id= 2024-06-11T14:41:37.850Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4 Address:e7cf2fe7-d894-d86b-6c22-8384c04b25c4} {Suffrage:Voter ID:server-d6d6d090-adce-774d-38eb-faa36d3753d9 Address:d6d6d090-adce-774d-38eb-faa36d3753d9} {Suffrage:Voter ID:server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 Address:c1567b57-f4f8-0368-f0d2-ae0a5fe32d05}]" 2024-06-11T14:41:37.850Z [INFO] server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4 Address:e7cf2fe7-d894-d86b-6c22-8384c04b25c4} {Suffrage:Voter ID:server-d6d6d090-adce-774d-38eb-faa36d3753d9 Address:d6d6d090-adce-774d-38eb-faa36d3753d9} {Suffrage:Voter ID:server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 Address:c1567b57-f4f8-0368-f0d2-ae0a5fe32d05}]" 2024-06-11T14:41:37.850Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:37.850Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: entering follower state: follower="Node at d6d6d090-adce-774d-38eb-faa36d3753d9 [Follower]" leader-address= leader-id= 2024-06-11T14:41:37.850Z [INFO] server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05: entering follower state: follower="Node at c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 [Follower]" leader-address= leader-id= 2024-06-11T14:41:37.907Z [WARN] server-d6d6d090-adce-774d-38eb-faa36d3753d9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:37.907Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: entering candidate state: node="Node at d6d6d090-adce-774d-38eb-faa36d3753d9 [Candidate]" term=2 2024-06-11T14:41:37.907Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: election won: term=2 tally=2 2024-06-11T14:41:37.907Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: entering leader state: leader="Node at d6d6d090-adce-774d-38eb-faa36d3753d9 [Leader]" 2024-06-11T14:41:37.907Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: added peer, starting replication: peer=server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4 2024-06-11T14:41:37.907Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: added peer, starting replication: peer=server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 2024-06-11T14:41:37.908Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: pipelining replication: peer="{Voter server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4 e7cf2fe7-d894-d86b-6c22-8384c04b25c4}" 2024-06-11T14:41:37.908Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: pipelining replication: peer="{Voter server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 c1567b57-f4f8-0368-f0d2-ae0a5fe32d05}" 2024-06-11T14:41:38.013Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:37.907865412 +0000 UTC m=+104.490944324 (1 nodes), 56.974544ms from start of poll, 58.537328ms from cluster start. Timeout at 2024-06-11 14:41:38.012978339 +0000 UTC m=+104.596057251, 105.112927ms after stability 2024-06-11T14:41:38.013Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:38.119Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:38.013186052 +0000 UTC m=+104.596265004 (1 nodes), 2.12µs from start of poll, 163.858008ms from cluster start. Timeout at 2024-06-11 14:41:38.118996634 +0000 UTC m=+104.702075586, 105.810582ms after stability 2024-06-11T14:41:38.119Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: entering candidate state: node="Node at e7cf2fe7-d894-d86b-6c22-8384c04b25c4 [Candidate]" term=3 2024-06-11T14:41:38.119Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: election won: term=3 tally=2 2024-06-11T14:41:38.119Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: entering leader state: leader="Node at e7cf2fe7-d894-d86b-6c22-8384c04b25c4 [Leader]" 2024-06-11T14:41:38.119Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: added peer, starting replication: peer=server-d6d6d090-adce-774d-38eb-faa36d3753d9 2024-06-11T14:41:38.119Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: added peer, starting replication: peer=server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 2024-06-11T14:41:38.119Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: entering follower state: follower="Node at d6d6d090-adce-774d-38eb-faa36d3753d9 [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.119Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: aborting pipeline replication: peer="{Voter server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 c1567b57-f4f8-0368-f0d2-ae0a5fe32d05}" 2024-06-11T14:41:38.119Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: pipelining replication: peer="{Voter server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 c1567b57-f4f8-0368-f0d2-ae0a5fe32d05}" 2024-06-11T14:41:38.119Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:38.119Z [INFO] server-d6d6d090-adce-774d-38eb-faa36d3753d9: aborting pipeline replication: peer="{Voter server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4 e7cf2fe7-d894-d86b-6c22-8384c04b25c4}" 2024-06-11T14:41:38.119Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: pipelining replication: peer="{Voter server-d6d6d090-adce-774d-38eb-faa36d3753d9 d6d6d090-adce-774d-38eb-faa36d3753d9}" 2024-06-11T14:41:38.224Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:38.119834844 +0000 UTC m=+104.702913756 (1 nodes), 53.439µs from start of poll, 270.50676ms from cluster start. Timeout at 2024-06-11 14:41:38.224837415 +0000 UTC m=+104.807916367, 105.002611ms after stability 2024-06-11T14:41:38.225Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: aborting pipeline replication: peer="{Voter server-c1567b57-f4f8-0368-f0d2-ae0a5fe32d05 c1567b57-f4f8-0368-f0d2-ae0a5fe32d05}" 2024-06-11T14:41:38.225Z [INFO] server-e7cf2fe7-d894-d86b-6c22-8384c04b25c4: aborting pipeline replication: peer="{Voter server-d6d6d090-adce-774d-38eb-faa36d3753d9 d6d6d090-adce-774d-38eb-faa36d3753d9}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode 2024-06-11T14:41:38.227Z [INFO] server-77f65adb-a787-e7f6-d028-c4f452961eb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f65adb-a787-e7f6-d028-c4f452961eb0 Address:77f65adb-a787-e7f6-d028-c4f452961eb0}]" 2024-06-11T14:41:38.227Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:38.227Z [INFO] server-77f65adb-a787-e7f6-d028-c4f452961eb0: entering follower state: follower="Node at 77f65adb-a787-e7f6-d028-c4f452961eb0 [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.297Z [WARN] server-77f65adb-a787-e7f6-d028-c4f452961eb0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:38.297Z [INFO] server-77f65adb-a787-e7f6-d028-c4f452961eb0: entering candidate state: node="Node at 77f65adb-a787-e7f6-d028-c4f452961eb0 [Candidate]" term=2 2024-06-11T14:41:38.297Z [INFO] server-77f65adb-a787-e7f6-d028-c4f452961eb0: election won: term=2 tally=1 2024-06-11T14:41:38.297Z [INFO] server-77f65adb-a787-e7f6-d028-c4f452961eb0: entering leader state: leader="Node at 77f65adb-a787-e7f6-d028-c4f452961eb0 [Leader]" 2024-06-11T14:41:38.404Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:38.29869603 +0000 UTC m=+104.881774942 (1 nodes), 71.545179ms from start of poll, 71.857687ms from cluster start. Timeout at 2024-06-11 14:41:38.404475373 +0000 UTC m=+104.987554325, 105.779383ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2024-06-11T14:41:38.407Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 Address:c025ff85-7b6d-ece1-87aa-6f5d2f19fb84} {Suffrage:Voter ID:server-4ac9afc3-542c-959d-f81f-e8eb2172482d Address:4ac9afc3-542c-959d-f81f-e8eb2172482d} {Suffrage:Voter ID:server-3c968ed6-4734-74d2-50dc-36ec8b22b95b Address:3c968ed6-4734-74d2-50dc-36ec8b22b95b} {Suffrage:Voter ID:server-80376dbe-edb9-26ee-8708-22843b878160 Address:80376dbe-edb9-26ee-8708-22843b878160} {Suffrage:Voter ID:server-0170736c-6996-37cb-87cc-6a97ad955635 Address:0170736c-6996-37cb-87cc-6a97ad955635} {Suffrage:Voter ID:server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 Address:5c87e321-adbf-52fa-8f75-ccb8f897b7b7} {Suffrage:Voter ID:server-8f658095-ae5d-3172-184d-0f7da1be840d Address:8f658095-ae5d-3172-184d-0f7da1be840d}]" 2024-06-11T14:41:38.407Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 Address:c025ff85-7b6d-ece1-87aa-6f5d2f19fb84} {Suffrage:Voter ID:server-4ac9afc3-542c-959d-f81f-e8eb2172482d Address:4ac9afc3-542c-959d-f81f-e8eb2172482d} {Suffrage:Voter ID:server-3c968ed6-4734-74d2-50dc-36ec8b22b95b Address:3c968ed6-4734-74d2-50dc-36ec8b22b95b} {Suffrage:Voter ID:server-80376dbe-edb9-26ee-8708-22843b878160 Address:80376dbe-edb9-26ee-8708-22843b878160} {Suffrage:Voter ID:server-0170736c-6996-37cb-87cc-6a97ad955635 Address:0170736c-6996-37cb-87cc-6a97ad955635} {Suffrage:Voter ID:server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 Address:5c87e321-adbf-52fa-8f75-ccb8f897b7b7} {Suffrage:Voter ID:server-8f658095-ae5d-3172-184d-0f7da1be840d Address:8f658095-ae5d-3172-184d-0f7da1be840d}]" 2024-06-11T14:41:38.408Z [INFO] server-3c968ed6-4734-74d2-50dc-36ec8b22b95b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 Address:c025ff85-7b6d-ece1-87aa-6f5d2f19fb84} {Suffrage:Voter ID:server-4ac9afc3-542c-959d-f81f-e8eb2172482d Address:4ac9afc3-542c-959d-f81f-e8eb2172482d} {Suffrage:Voter ID:server-3c968ed6-4734-74d2-50dc-36ec8b22b95b Address:3c968ed6-4734-74d2-50dc-36ec8b22b95b} {Suffrage:Voter ID:server-80376dbe-edb9-26ee-8708-22843b878160 Address:80376dbe-edb9-26ee-8708-22843b878160} {Suffrage:Voter ID:server-0170736c-6996-37cb-87cc-6a97ad955635 Address:0170736c-6996-37cb-87cc-6a97ad955635} {Suffrage:Voter ID:server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 Address:5c87e321-adbf-52fa-8f75-ccb8f897b7b7} {Suffrage:Voter ID:server-8f658095-ae5d-3172-184d-0f7da1be840d Address:8f658095-ae5d-3172-184d-0f7da1be840d}]" 2024-06-11T14:41:38.408Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: entering follower state: follower="Node at 4ac9afc3-542c-959d-f81f-e8eb2172482d [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.408Z [INFO] server-80376dbe-edb9-26ee-8708-22843b878160: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 Address:c025ff85-7b6d-ece1-87aa-6f5d2f19fb84} {Suffrage:Voter ID:server-4ac9afc3-542c-959d-f81f-e8eb2172482d Address:4ac9afc3-542c-959d-f81f-e8eb2172482d} {Suffrage:Voter ID:server-3c968ed6-4734-74d2-50dc-36ec8b22b95b Address:3c968ed6-4734-74d2-50dc-36ec8b22b95b} {Suffrage:Voter ID:server-80376dbe-edb9-26ee-8708-22843b878160 Address:80376dbe-edb9-26ee-8708-22843b878160} {Suffrage:Voter ID:server-0170736c-6996-37cb-87cc-6a97ad955635 Address:0170736c-6996-37cb-87cc-6a97ad955635} {Suffrage:Voter ID:server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 Address:5c87e321-adbf-52fa-8f75-ccb8f897b7b7} {Suffrage:Voter ID:server-8f658095-ae5d-3172-184d-0f7da1be840d Address:8f658095-ae5d-3172-184d-0f7da1be840d}]" 2024-06-11T14:41:38.408Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: entering follower state: follower="Node at c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.408Z [INFO] server-3c968ed6-4734-74d2-50dc-36ec8b22b95b: entering follower state: follower="Node at 3c968ed6-4734-74d2-50dc-36ec8b22b95b [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.408Z [INFO] server-0170736c-6996-37cb-87cc-6a97ad955635: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 Address:c025ff85-7b6d-ece1-87aa-6f5d2f19fb84} {Suffrage:Voter ID:server-4ac9afc3-542c-959d-f81f-e8eb2172482d Address:4ac9afc3-542c-959d-f81f-e8eb2172482d} {Suffrage:Voter ID:server-3c968ed6-4734-74d2-50dc-36ec8b22b95b Address:3c968ed6-4734-74d2-50dc-36ec8b22b95b} {Suffrage:Voter ID:server-80376dbe-edb9-26ee-8708-22843b878160 Address:80376dbe-edb9-26ee-8708-22843b878160} {Suffrage:Voter ID:server-0170736c-6996-37cb-87cc-6a97ad955635 Address:0170736c-6996-37cb-87cc-6a97ad955635} {Suffrage:Voter ID:server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 Address:5c87e321-adbf-52fa-8f75-ccb8f897b7b7} {Suffrage:Voter ID:server-8f658095-ae5d-3172-184d-0f7da1be840d Address:8f658095-ae5d-3172-184d-0f7da1be840d}]" 2024-06-11T14:41:38.408Z [INFO] server-80376dbe-edb9-26ee-8708-22843b878160: entering follower state: follower="Node at 80376dbe-edb9-26ee-8708-22843b878160 [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.408Z [INFO] server-0170736c-6996-37cb-87cc-6a97ad955635: entering follower state: follower="Node at 0170736c-6996-37cb-87cc-6a97ad955635 [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.408Z [INFO] server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 Address:c025ff85-7b6d-ece1-87aa-6f5d2f19fb84} {Suffrage:Voter ID:server-4ac9afc3-542c-959d-f81f-e8eb2172482d Address:4ac9afc3-542c-959d-f81f-e8eb2172482d} {Suffrage:Voter ID:server-3c968ed6-4734-74d2-50dc-36ec8b22b95b Address:3c968ed6-4734-74d2-50dc-36ec8b22b95b} {Suffrage:Voter ID:server-80376dbe-edb9-26ee-8708-22843b878160 Address:80376dbe-edb9-26ee-8708-22843b878160} {Suffrage:Voter ID:server-0170736c-6996-37cb-87cc-6a97ad955635 Address:0170736c-6996-37cb-87cc-6a97ad955635} {Suffrage:Voter ID:server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 Address:5c87e321-adbf-52fa-8f75-ccb8f897b7b7} {Suffrage:Voter ID:server-8f658095-ae5d-3172-184d-0f7da1be840d Address:8f658095-ae5d-3172-184d-0f7da1be840d}]" 2024-06-11T14:41:38.408Z [INFO] server-8f658095-ae5d-3172-184d-0f7da1be840d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 Address:c025ff85-7b6d-ece1-87aa-6f5d2f19fb84} {Suffrage:Voter ID:server-4ac9afc3-542c-959d-f81f-e8eb2172482d Address:4ac9afc3-542c-959d-f81f-e8eb2172482d} {Suffrage:Voter ID:server-3c968ed6-4734-74d2-50dc-36ec8b22b95b Address:3c968ed6-4734-74d2-50dc-36ec8b22b95b} {Suffrage:Voter ID:server-80376dbe-edb9-26ee-8708-22843b878160 Address:80376dbe-edb9-26ee-8708-22843b878160} {Suffrage:Voter ID:server-0170736c-6996-37cb-87cc-6a97ad955635 Address:0170736c-6996-37cb-87cc-6a97ad955635} {Suffrage:Voter ID:server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 Address:5c87e321-adbf-52fa-8f75-ccb8f897b7b7} {Suffrage:Voter ID:server-8f658095-ae5d-3172-184d-0f7da1be840d Address:8f658095-ae5d-3172-184d-0f7da1be840d}]" 2024-06-11T14:41:38.408Z [INFO] server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7: entering follower state: follower="Node at 5c87e321-adbf-52fa-8f75-ccb8f897b7b7 [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.409Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:38.409Z [INFO] server-8f658095-ae5d-3172-184d-0f7da1be840d: entering follower state: follower="Node at 8f658095-ae5d-3172-184d-0f7da1be840d [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.474Z [WARN] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:38.474Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: entering candidate state: node="Node at c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 [Candidate]" term=2 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: election won: term=2 tally=4 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: entering leader state: leader="Node at c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 [Leader]" 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: added peer, starting replication: peer=server-4ac9afc3-542c-959d-f81f-e8eb2172482d 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: added peer, starting replication: peer=server-3c968ed6-4734-74d2-50dc-36ec8b22b95b 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: added peer, starting replication: peer=server-80376dbe-edb9-26ee-8708-22843b878160 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: added peer, starting replication: peer=server-0170736c-6996-37cb-87cc-6a97ad955635 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: added peer, starting replication: peer=server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: added peer, starting replication: peer=server-8f658095-ae5d-3172-184d-0f7da1be840d 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: pipelining replication: peer="{Voter server-3c968ed6-4734-74d2-50dc-36ec8b22b95b 3c968ed6-4734-74d2-50dc-36ec8b22b95b}" 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: pipelining replication: peer="{Voter server-80376dbe-edb9-26ee-8708-22843b878160 80376dbe-edb9-26ee-8708-22843b878160}" 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: pipelining replication: peer="{Voter server-4ac9afc3-542c-959d-f81f-e8eb2172482d 4ac9afc3-542c-959d-f81f-e8eb2172482d}" 2024-06-11T14:41:38.475Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: pipelining replication: peer="{Voter server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 5c87e321-adbf-52fa-8f75-ccb8f897b7b7}" 2024-06-11T14:41:38.476Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: pipelining replication: peer="{Voter server-8f658095-ae5d-3172-184d-0f7da1be840d 8f658095-ae5d-3172-184d-0f7da1be840d}" 2024-06-11T14:41:38.476Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: pipelining replication: peer="{Voter server-0170736c-6996-37cb-87cc-6a97ad955635 0170736c-6996-37cb-87cc-6a97ad955635}" 2024-06-11T14:41:38.581Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:38.475394694 +0000 UTC m=+105.058473646 (1 nodes), 66.335886ms from start of poll, 67.997866ms from cluster start. Timeout at 2024-06-11 14:41:38.581157478 +0000 UTC m=+105.164236390, 105.762744ms after stability 2024-06-11T14:41:38.581Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:38.686Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:38.58136287 +0000 UTC m=+105.164441782 (6 nodes), 4.839µs from start of poll, 173.966002ms from cluster start. Timeout at 2024-06-11 14:41:38.686870383 +0000 UTC m=+105.269949335, 105.507553ms after stability 2024-06-11T14:41:38.686Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:38.792Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:38.687041497 +0000 UTC m=+105.270120409 (1 nodes), 2.52µs from start of poll, 279.644629ms from cluster start. Timeout at 2024-06-11 14:41:38.792780882 +0000 UTC m=+105.375859794, 105.739385ms after stability 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: entering candidate state: node="Node at 4ac9afc3-542c-959d-f81f-e8eb2172482d [Candidate]" term=3 2024-06-11T14:41:38.793Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: election won: term=3 tally=4 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: entering leader state: leader="Node at 4ac9afc3-542c-959d-f81f-e8eb2172482d [Leader]" 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: added peer, starting replication: peer=server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: added peer, starting replication: peer=server-3c968ed6-4734-74d2-50dc-36ec8b22b95b 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: added peer, starting replication: peer=server-80376dbe-edb9-26ee-8708-22843b878160 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: added peer, starting replication: peer=server-0170736c-6996-37cb-87cc-6a97ad955635 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: added peer, starting replication: peer=server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 2024-06-11T14:41:38.793Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: added peer, starting replication: peer=server-8f658095-ae5d-3172-184d-0f7da1be840d 2024-06-11T14:41:38.794Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: pipelining replication: peer="{Voter server-0170736c-6996-37cb-87cc-6a97ad955635 0170736c-6996-37cb-87cc-6a97ad955635}" 2024-06-11T14:41:38.793Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: entering follower state: follower="Node at c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 [Follower]" leader-address= leader-id= 2024-06-11T14:41:38.794Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: pipelining replication: peer="{Voter server-8f658095-ae5d-3172-184d-0f7da1be840d 8f658095-ae5d-3172-184d-0f7da1be840d}" 2024-06-11T14:41:38.794Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: pipelining replication: peer="{Voter server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 5c87e321-adbf-52fa-8f75-ccb8f897b7b7}" 2024-06-11T14:41:38.793Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: aborting pipeline replication: peer="{Voter server-4ac9afc3-542c-959d-f81f-e8eb2172482d 4ac9afc3-542c-959d-f81f-e8eb2172482d}" 2024-06-11T14:41:38.793Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: aborting pipeline replication: peer="{Voter server-3c968ed6-4734-74d2-50dc-36ec8b22b95b 3c968ed6-4734-74d2-50dc-36ec8b22b95b}" 2024-06-11T14:41:38.793Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: aborting pipeline replication: peer="{Voter server-80376dbe-edb9-26ee-8708-22843b878160 80376dbe-edb9-26ee-8708-22843b878160}" 2024-06-11T14:41:38.793Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: aborting pipeline replication: peer="{Voter server-0170736c-6996-37cb-87cc-6a97ad955635 0170736c-6996-37cb-87cc-6a97ad955635}" 2024-06-11T14:41:38.793Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: aborting pipeline replication: peer="{Voter server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 5c87e321-adbf-52fa-8f75-ccb8f897b7b7}" 2024-06-11T14:41:38.793Z [INFO] server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84: aborting pipeline replication: peer="{Voter server-8f658095-ae5d-3172-184d-0f7da1be840d 8f658095-ae5d-3172-184d-0f7da1be840d}" 2024-06-11T14:41:38.794Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: pipelining replication: peer="{Voter server-80376dbe-edb9-26ee-8708-22843b878160 80376dbe-edb9-26ee-8708-22843b878160}" 2024-06-11T14:41:38.794Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: pipelining replication: peer="{Voter server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 c025ff85-7b6d-ece1-87aa-6f5d2f19fb84}" 2024-06-11T14:41:38.794Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: pipelining replication: peer="{Voter server-3c968ed6-4734-74d2-50dc-36ec8b22b95b 3c968ed6-4734-74d2-50dc-36ec8b22b95b}" 2024-06-11T14:41:38.899Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:38.79448874 +0000 UTC m=+105.377567692 (1 nodes), 1.125879ms from start of poll, 387.091912ms from cluster start. Timeout at 2024-06-11 14:41:38.899659986 +0000 UTC m=+105.482738937, 105.171245ms after stability 2024-06-11T14:41:38.899Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: aborting pipeline replication: peer="{Voter server-3c968ed6-4734-74d2-50dc-36ec8b22b95b 3c968ed6-4734-74d2-50dc-36ec8b22b95b}" 2024-06-11T14:41:38.900Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: aborting pipeline replication: peer="{Voter server-80376dbe-edb9-26ee-8708-22843b878160 80376dbe-edb9-26ee-8708-22843b878160}" 2024-06-11T14:41:38.900Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: aborting pipeline replication: peer="{Voter server-8f658095-ae5d-3172-184d-0f7da1be840d 8f658095-ae5d-3172-184d-0f7da1be840d}" 2024-06-11T14:41:38.900Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: aborting pipeline replication: peer="{Voter server-5c87e321-adbf-52fa-8f75-ccb8f897b7b7 5c87e321-adbf-52fa-8f75-ccb8f897b7b7}" 2024-06-11T14:41:38.900Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: aborting pipeline replication: peer="{Voter server-c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 c025ff85-7b6d-ece1-87aa-6f5d2f19fb84}" 2024-06-11T14:41:38.900Z [INFO] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: aborting pipeline replication: peer="{Voter server-0170736c-6996-37cb-87cc-6a97ad955635 0170736c-6996-37cb-87cc-6a97ad955635}" 2024-06-11T14:41:39.400Z [ERROR] server-4ac9afc3-542c-959d-f81f-e8eb2172482d: failed to heartbeat to: peer=c025ff85-7b6d-ece1-87aa-6f5d2f19fb84 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (1.00s) === RUN TestRaft_LeadershipTransferToInvalidID 2024-06-11T14:41:39.404Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c4e218-85c6-601d-0576-ef313bdb8baa Address:42c4e218-85c6-601d-0576-ef313bdb8baa} {Suffrage:Voter ID:server-b0018298-eccc-c5c5-0b46-2344a7da7216 Address:b0018298-eccc-c5c5-0b46-2344a7da7216} {Suffrage:Voter ID:server-17a6e523-118a-1ba8-4ceb-1141eae39125 Address:17a6e523-118a-1ba8-4ceb-1141eae39125}]" 2024-06-11T14:41:39.404Z [INFO] server-b0018298-eccc-c5c5-0b46-2344a7da7216: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c4e218-85c6-601d-0576-ef313bdb8baa Address:42c4e218-85c6-601d-0576-ef313bdb8baa} {Suffrage:Voter ID:server-b0018298-eccc-c5c5-0b46-2344a7da7216 Address:b0018298-eccc-c5c5-0b46-2344a7da7216} {Suffrage:Voter ID:server-17a6e523-118a-1ba8-4ceb-1141eae39125 Address:17a6e523-118a-1ba8-4ceb-1141eae39125}]" 2024-06-11T14:41:39.404Z [INFO] server-17a6e523-118a-1ba8-4ceb-1141eae39125: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c4e218-85c6-601d-0576-ef313bdb8baa Address:42c4e218-85c6-601d-0576-ef313bdb8baa} {Suffrage:Voter ID:server-b0018298-eccc-c5c5-0b46-2344a7da7216 Address:b0018298-eccc-c5c5-0b46-2344a7da7216} {Suffrage:Voter ID:server-17a6e523-118a-1ba8-4ceb-1141eae39125 Address:17a6e523-118a-1ba8-4ceb-1141eae39125}]" 2024-06-11T14:41:39.404Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:39.405Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: entering follower state: follower="Node at 42c4e218-85c6-601d-0576-ef313bdb8baa [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.405Z [INFO] server-b0018298-eccc-c5c5-0b46-2344a7da7216: entering follower state: follower="Node at b0018298-eccc-c5c5-0b46-2344a7da7216 [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.405Z [INFO] server-17a6e523-118a-1ba8-4ceb-1141eae39125: entering follower state: follower="Node at 17a6e523-118a-1ba8-4ceb-1141eae39125 [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.459Z [WARN] server-42c4e218-85c6-601d-0576-ef313bdb8baa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:39.460Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: entering candidate state: node="Node at 42c4e218-85c6-601d-0576-ef313bdb8baa [Candidate]" term=2 2024-06-11T14:41:39.460Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: election won: term=2 tally=2 2024-06-11T14:41:39.460Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: entering leader state: leader="Node at 42c4e218-85c6-601d-0576-ef313bdb8baa [Leader]" 2024-06-11T14:41:39.460Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: added peer, starting replication: peer=server-b0018298-eccc-c5c5-0b46-2344a7da7216 2024-06-11T14:41:39.460Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: added peer, starting replication: peer=server-17a6e523-118a-1ba8-4ceb-1141eae39125 2024-06-11T14:41:39.460Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: pipelining replication: peer="{Voter server-b0018298-eccc-c5c5-0b46-2344a7da7216 b0018298-eccc-c5c5-0b46-2344a7da7216}" 2024-06-11T14:41:39.460Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: pipelining replication: peer="{Voter server-17a6e523-118a-1ba8-4ceb-1141eae39125 17a6e523-118a-1ba8-4ceb-1141eae39125}" 2024-06-11T14:41:39.565Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:39.460304596 +0000 UTC m=+106.043383508 (1 nodes), 55.481478ms from start of poll, 56.170933ms from cluster start. Timeout at 2024-06-11 14:41:39.565323407 +0000 UTC m=+106.148402359, 105.018851ms after stability 2024-06-11T14:41:39.565Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: aborting pipeline replication: peer="{Voter server-b0018298-eccc-c5c5-0b46-2344a7da7216 b0018298-eccc-c5c5-0b46-2344a7da7216}" 2024-06-11T14:41:39.565Z [INFO] server-42c4e218-85c6-601d-0576-ef313bdb8baa: aborting pipeline replication: peer="{Voter server-17a6e523-118a-1ba8-4ceb-1141eae39125 17a6e523-118a-1ba8-4ceb-1141eae39125}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2024-06-11T14:41:39.567Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9518d936-bed6-ad1c-71b7-fef4c180430d Address:9518d936-bed6-ad1c-71b7-fef4c180430d} {Suffrage:Voter ID:server-151b625a-041b-161b-394b-e1fee9bd3717 Address:151b625a-041b-161b-394b-e1fee9bd3717} {Suffrage:Voter ID:server-16c0ed60-5139-8a05-d8e6-55b470dda081 Address:16c0ed60-5139-8a05-d8e6-55b470dda081}]" 2024-06-11T14:41:39.568Z [INFO] server-151b625a-041b-161b-394b-e1fee9bd3717: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9518d936-bed6-ad1c-71b7-fef4c180430d Address:9518d936-bed6-ad1c-71b7-fef4c180430d} {Suffrage:Voter ID:server-151b625a-041b-161b-394b-e1fee9bd3717 Address:151b625a-041b-161b-394b-e1fee9bd3717} {Suffrage:Voter ID:server-16c0ed60-5139-8a05-d8e6-55b470dda081 Address:16c0ed60-5139-8a05-d8e6-55b470dda081}]" 2024-06-11T14:41:39.568Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: entering follower state: follower="Node at 9518d936-bed6-ad1c-71b7-fef4c180430d [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.568Z [INFO] server-151b625a-041b-161b-394b-e1fee9bd3717: entering follower state: follower="Node at 151b625a-041b-161b-394b-e1fee9bd3717 [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.568Z [INFO] server-16c0ed60-5139-8a05-d8e6-55b470dda081: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9518d936-bed6-ad1c-71b7-fef4c180430d Address:9518d936-bed6-ad1c-71b7-fef4c180430d} {Suffrage:Voter ID:server-151b625a-041b-161b-394b-e1fee9bd3717 Address:151b625a-041b-161b-394b-e1fee9bd3717} {Suffrage:Voter ID:server-16c0ed60-5139-8a05-d8e6-55b470dda081 Address:16c0ed60-5139-8a05-d8e6-55b470dda081}]" 2024-06-11T14:41:39.568Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:39.568Z [INFO] server-16c0ed60-5139-8a05-d8e6-55b470dda081: entering follower state: follower="Node at 16c0ed60-5139-8a05-d8e6-55b470dda081 [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.620Z [WARN] server-9518d936-bed6-ad1c-71b7-fef4c180430d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:39.620Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: entering candidate state: node="Node at 9518d936-bed6-ad1c-71b7-fef4c180430d [Candidate]" term=2 2024-06-11T14:41:39.620Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: election won: term=2 tally=2 2024-06-11T14:41:39.620Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: entering leader state: leader="Node at 9518d936-bed6-ad1c-71b7-fef4c180430d [Leader]" 2024-06-11T14:41:39.620Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: added peer, starting replication: peer=server-151b625a-041b-161b-394b-e1fee9bd3717 2024-06-11T14:41:39.620Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: added peer, starting replication: peer=server-16c0ed60-5139-8a05-d8e6-55b470dda081 2024-06-11T14:41:39.620Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: pipelining replication: peer="{Voter server-16c0ed60-5139-8a05-d8e6-55b470dda081 16c0ed60-5139-8a05-d8e6-55b470dda081}" 2024-06-11T14:41:39.620Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: pipelining replication: peer="{Voter server-151b625a-041b-161b-394b-e1fee9bd3717 151b625a-041b-161b-394b-e1fee9bd3717}" 2024-06-11T14:41:39.726Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:39.62038382 +0000 UTC m=+106.203462772 (2 nodes), 51.849449ms from start of poll, 52.639741ms from cluster start. Timeout at 2024-06-11 14:41:39.726126125 +0000 UTC m=+106.309205037, 105.742265ms after stability 2024-06-11T14:41:39.726Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:39.832Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:39.726317198 +0000 UTC m=+106.309396110 (1 nodes), 2µs from start of poll, 158.573079ms from cluster start. Timeout at 2024-06-11 14:41:39.83238357 +0000 UTC m=+106.415462522, 106.066412ms after stability 2024-06-11T14:41:39.832Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: aborting pipeline replication: peer="{Voter server-151b625a-041b-161b-394b-e1fee9bd3717 151b625a-041b-161b-394b-e1fee9bd3717}" 2024-06-11T14:41:39.832Z [INFO] server-9518d936-bed6-ad1c-71b7-fef4c180430d: aborting pipeline replication: peer="{Voter server-16c0ed60-5139-8a05-d8e6-55b470dda081 16c0ed60-5139-8a05-d8e6-55b470dda081}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer 2024-06-11T14:41:39.835Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bfc051c-4c5c-8c24-42ca-d09a4b592602 Address:5bfc051c-4c5c-8c24-42ca-d09a4b592602} {Suffrage:Voter ID:server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 Address:9c74938e-f5c4-37b5-b3ea-9287cdd892f5} {Suffrage:Voter ID:server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8 Address:3ed8fc0c-7af6-10e1-a947-545aa4372fb8}]" 2024-06-11T14:41:39.835Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: entering follower state: follower="Node at 5bfc051c-4c5c-8c24-42ca-d09a4b592602 [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.835Z [INFO] server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bfc051c-4c5c-8c24-42ca-d09a4b592602 Address:5bfc051c-4c5c-8c24-42ca-d09a4b592602} {Suffrage:Voter ID:server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 Address:9c74938e-f5c4-37b5-b3ea-9287cdd892f5} {Suffrage:Voter ID:server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8 Address:3ed8fc0c-7af6-10e1-a947-545aa4372fb8}]" 2024-06-11T14:41:39.835Z [INFO] server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5: entering follower state: follower="Node at 9c74938e-f5c4-37b5-b3ea-9287cdd892f5 [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.835Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bfc051c-4c5c-8c24-42ca-d09a4b592602 Address:5bfc051c-4c5c-8c24-42ca-d09a4b592602} {Suffrage:Voter ID:server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 Address:9c74938e-f5c4-37b5-b3ea-9287cdd892f5} {Suffrage:Voter ID:server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8 Address:3ed8fc0c-7af6-10e1-a947-545aa4372fb8}]" 2024-06-11T14:41:39.835Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:39.835Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: entering follower state: follower="Node at 3ed8fc0c-7af6-10e1-a947-545aa4372fb8 [Follower]" leader-address= leader-id= 2024-06-11T14:41:39.888Z [WARN] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:39.888Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: entering candidate state: node="Node at 3ed8fc0c-7af6-10e1-a947-545aa4372fb8 [Candidate]" term=2 2024-06-11T14:41:39.889Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: election won: term=2 tally=2 2024-06-11T14:41:39.889Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: entering leader state: leader="Node at 3ed8fc0c-7af6-10e1-a947-545aa4372fb8 [Leader]" 2024-06-11T14:41:39.889Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: added peer, starting replication: peer=server-5bfc051c-4c5c-8c24-42ca-d09a4b592602 2024-06-11T14:41:39.889Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: added peer, starting replication: peer=server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 2024-06-11T14:41:39.889Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: pipelining replication: peer="{Voter server-5bfc051c-4c5c-8c24-42ca-d09a4b592602 5bfc051c-4c5c-8c24-42ca-d09a4b592602}" 2024-06-11T14:41:39.889Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: pipelining replication: peer="{Voter server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 9c74938e-f5c4-37b5-b3ea-9287cdd892f5}" 2024-06-11T14:41:40.003Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:39.88921652 +0000 UTC m=+106.472295432 (1 nodes), 53.300037ms from start of poll, 54.184085ms from cluster start. Timeout at 2024-06-11 14:41:40.001427191 +0000 UTC m=+106.584506103, 112.210671ms after stability 2024-06-11T14:41:40.003Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:40.109Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:40.00366267 +0000 UTC m=+106.586741582 (2 nodes), 1.96µs from start of poll, 168.630235ms from cluster start. Timeout at 2024-06-11 14:41:40.10898919 +0000 UTC m=+106.692068142, 105.32656ms after stability 2024-06-11T14:41:40.225Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: entering candidate state: node="Node at 5bfc051c-4c5c-8c24-42ca-d09a4b592602 [Candidate]" term=3 2024-06-11T14:41:40.225Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: entering follower state: follower="Node at 3ed8fc0c-7af6-10e1-a947-545aa4372fb8 [Follower]" leader-address= leader-id= 2024-06-11T14:41:40.225Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:40.225Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: election won: term=3 tally=2 2024-06-11T14:41:40.225Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: entering leader state: leader="Node at 5bfc051c-4c5c-8c24-42ca-d09a4b592602 [Leader]" 2024-06-11T14:41:40.225Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: added peer, starting replication: peer=server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 2024-06-11T14:41:40.226Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: added peer, starting replication: peer=server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8 2024-06-11T14:41:40.226Z [WARN] server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5: failed to get previous log: previous-index=1002 last-index=951 error="log not found" 2024-06-11T14:41:40.226Z [WARN] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: appendEntries rejected, sending older logs: peer="{Voter server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 9c74938e-f5c4-37b5-b3ea-9287cdd892f5}" next=952 2024-06-11T14:41:40.226Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: pipelining replication: peer="{Voter server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8 3ed8fc0c-7af6-10e1-a947-545aa4372fb8}" 2024-06-11T14:41:40.226Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: aborting pipeline replication: peer="{Voter server-5bfc051c-4c5c-8c24-42ca-d09a4b592602 5bfc051c-4c5c-8c24-42ca-d09a4b592602}" 2024-06-11T14:41:40.226Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: pipelining replication: peer="{Voter server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 9c74938e-f5c4-37b5-b3ea-9287cdd892f5}" 2024-06-11T14:41:40.227Z [INFO] server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8: aborting pipeline replication: peer="{Voter server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 9c74938e-f5c4-37b5-b3ea-9287cdd892f5}" 2024-06-11T14:41:40.332Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:40.226195161 +0000 UTC m=+106.809274113 (1 nodes), 323.269µs from start of poll, 391.162766ms from cluster start. Timeout at 2024-06-11 14:41:40.332134858 +0000 UTC m=+106.915213810, 105.939697ms after stability 2024-06-11T14:41:40.332Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: aborting pipeline replication: peer="{Voter server-3ed8fc0c-7af6-10e1-a947-545aa4372fb8 3ed8fc0c-7af6-10e1-a947-545aa4372fb8}" 2024-06-11T14:41:40.332Z [INFO] server-5bfc051c-4c5c-8c24-42ca-d09a4b592602: aborting pipeline replication: peer="{Voter server-9c74938e-f5c4-37b5-b3ea-9287cdd892f5 9c74938e-f5c4-37b5-b3ea-9287cdd892f5}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.50s) === RUN TestRaft_LeadershipTransferToItself 2024-06-11T14:41:40.335Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5d00b9c-12a4-2a04-8960-d055199380ed Address:d5d00b9c-12a4-2a04-8960-d055199380ed} {Suffrage:Voter ID:server-cdb06027-43ba-34d3-96b7-2ea214a566df Address:cdb06027-43ba-34d3-96b7-2ea214a566df} {Suffrage:Voter ID:server-1d9279dc-859c-618f-ebe2-70abdfb48d50 Address:1d9279dc-859c-618f-ebe2-70abdfb48d50}]" 2024-06-11T14:41:40.335Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: entering follower state: follower="Node at d5d00b9c-12a4-2a04-8960-d055199380ed [Follower]" leader-address= leader-id= 2024-06-11T14:41:40.335Z [INFO] server-cdb06027-43ba-34d3-96b7-2ea214a566df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5d00b9c-12a4-2a04-8960-d055199380ed Address:d5d00b9c-12a4-2a04-8960-d055199380ed} {Suffrage:Voter ID:server-cdb06027-43ba-34d3-96b7-2ea214a566df Address:cdb06027-43ba-34d3-96b7-2ea214a566df} {Suffrage:Voter ID:server-1d9279dc-859c-618f-ebe2-70abdfb48d50 Address:1d9279dc-859c-618f-ebe2-70abdfb48d50}]" 2024-06-11T14:41:40.336Z [INFO] server-1d9279dc-859c-618f-ebe2-70abdfb48d50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5d00b9c-12a4-2a04-8960-d055199380ed Address:d5d00b9c-12a4-2a04-8960-d055199380ed} {Suffrage:Voter ID:server-cdb06027-43ba-34d3-96b7-2ea214a566df Address:cdb06027-43ba-34d3-96b7-2ea214a566df} {Suffrage:Voter ID:server-1d9279dc-859c-618f-ebe2-70abdfb48d50 Address:1d9279dc-859c-618f-ebe2-70abdfb48d50}]" 2024-06-11T14:41:40.336Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:40.336Z [INFO] server-cdb06027-43ba-34d3-96b7-2ea214a566df: entering follower state: follower="Node at cdb06027-43ba-34d3-96b7-2ea214a566df [Follower]" leader-address= leader-id= 2024-06-11T14:41:40.336Z [INFO] server-1d9279dc-859c-618f-ebe2-70abdfb48d50: entering follower state: follower="Node at 1d9279dc-859c-618f-ebe2-70abdfb48d50 [Follower]" leader-address= leader-id= 2024-06-11T14:41:40.385Z [WARN] server-d5d00b9c-12a4-2a04-8960-d055199380ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:40.385Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: entering candidate state: node="Node at d5d00b9c-12a4-2a04-8960-d055199380ed [Candidate]" term=2 2024-06-11T14:41:40.386Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: election won: term=2 tally=2 2024-06-11T14:41:40.386Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: entering leader state: leader="Node at d5d00b9c-12a4-2a04-8960-d055199380ed [Leader]" 2024-06-11T14:41:40.386Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: added peer, starting replication: peer=server-cdb06027-43ba-34d3-96b7-2ea214a566df 2024-06-11T14:41:40.386Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: added peer, starting replication: peer=server-1d9279dc-859c-618f-ebe2-70abdfb48d50 2024-06-11T14:41:40.386Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: pipelining replication: peer="{Voter server-cdb06027-43ba-34d3-96b7-2ea214a566df cdb06027-43ba-34d3-96b7-2ea214a566df}" 2024-06-11T14:41:40.386Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: pipelining replication: peer="{Voter server-1d9279dc-859c-618f-ebe2-70abdfb48d50 1d9279dc-859c-618f-ebe2-70abdfb48d50}" 2024-06-11T14:41:40.503Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:40.386193468 +0000 UTC m=+106.969272420 (1 nodes), 49.984917ms from start of poll, 51.105676ms from cluster start. Timeout at 2024-06-11 14:41:40.498252704 +0000 UTC m=+107.081331656, 112.059236ms after stability 2024-06-11T14:41:40.503Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: cannot transfer leadership to itself 2024-06-11T14:41:40.503Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: aborting pipeline replication: peer="{Voter server-1d9279dc-859c-618f-ebe2-70abdfb48d50 1d9279dc-859c-618f-ebe2-70abdfb48d50}" 2024-06-11T14:41:40.504Z [INFO] server-d5d00b9c-12a4-2a04-8960-d055199380ed: aborting pipeline replication: peer="{Voter server-cdb06027-43ba-34d3-96b7-2ea214a566df cdb06027-43ba-34d3-96b7-2ea214a566df}" 2024-06-11T14:41:41.004Z [ERROR] server-d5d00b9c-12a4-2a04-8960-d055199380ed: failed to heartbeat to: peer=1d9279dc-859c-618f-ebe2-70abdfb48d50 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferToItself (0.67s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2024-06-11T14:41:41.006Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a Address:658bebb0-81b2-0b0c-9d7c-b5d6202a737a} {Suffrage:Voter ID:server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309 Address:fd57bc65-ef53-4d9f-2a47-ce3e938a0309} {Suffrage:Voter ID:server-2dba6e06-7ca7-7166-4d07-44caa094eb29 Address:2dba6e06-7ca7-7166-4d07-44caa094eb29}]" 2024-06-11T14:41:41.007Z [INFO] server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a Address:658bebb0-81b2-0b0c-9d7c-b5d6202a737a} {Suffrage:Voter ID:server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309 Address:fd57bc65-ef53-4d9f-2a47-ce3e938a0309} {Suffrage:Voter ID:server-2dba6e06-7ca7-7166-4d07-44caa094eb29 Address:2dba6e06-7ca7-7166-4d07-44caa094eb29}]" 2024-06-11T14:41:41.007Z [INFO] server-2dba6e06-7ca7-7166-4d07-44caa094eb29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a Address:658bebb0-81b2-0b0c-9d7c-b5d6202a737a} {Suffrage:Voter ID:server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309 Address:fd57bc65-ef53-4d9f-2a47-ce3e938a0309} {Suffrage:Voter ID:server-2dba6e06-7ca7-7166-4d07-44caa094eb29 Address:2dba6e06-7ca7-7166-4d07-44caa094eb29}]" 2024-06-11T14:41:41.007Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:41.007Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: entering follower state: follower="Node at 658bebb0-81b2-0b0c-9d7c-b5d6202a737a [Follower]" leader-address= leader-id= 2024-06-11T14:41:41.007Z [INFO] server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309: entering follower state: follower="Node at fd57bc65-ef53-4d9f-2a47-ce3e938a0309 [Follower]" leader-address= leader-id= 2024-06-11T14:41:41.007Z [INFO] server-2dba6e06-7ca7-7166-4d07-44caa094eb29: entering follower state: follower="Node at 2dba6e06-7ca7-7166-4d07-44caa094eb29 [Follower]" leader-address= leader-id= 2024-06-11T14:41:41.064Z [WARN] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:41.064Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: entering candidate state: node="Node at 658bebb0-81b2-0b0c-9d7c-b5d6202a737a [Candidate]" term=2 2024-06-11T14:41:41.064Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: election won: term=2 tally=2 2024-06-11T14:41:41.064Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: entering leader state: leader="Node at 658bebb0-81b2-0b0c-9d7c-b5d6202a737a [Leader]" 2024-06-11T14:41:41.064Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: added peer, starting replication: peer=server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309 2024-06-11T14:41:41.064Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: added peer, starting replication: peer=server-2dba6e06-7ca7-7166-4d07-44caa094eb29 2024-06-11T14:41:41.064Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: pipelining replication: peer="{Voter server-2dba6e06-7ca7-7166-4d07-44caa094eb29 2dba6e06-7ca7-7166-4d07-44caa094eb29}" 2024-06-11T14:41:41.065Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: pipelining replication: peer="{Voter server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309 fd57bc65-ef53-4d9f-2a47-ce3e938a0309}" 2024-06-11T14:41:41.170Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:41.065581994 +0000 UTC m=+107.648660906 (1 nodes), 58.304497ms from start of poll, 58.962153ms from cluster start. Timeout at 2024-06-11 14:41:41.170920713 +0000 UTC m=+107.753999625, 105.338719ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2024-06-11T14:41:41.171Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: aborting pipeline replication: peer="{Voter server-2dba6e06-7ca7-7166-4d07-44caa094eb29 2dba6e06-7ca7-7166-4d07-44caa094eb29}" 2024-06-11T14:41:41.171Z [INFO] server-658bebb0-81b2-0b0c-9d7c-b5d6202a737a: aborting pipeline replication: peer="{Voter server-fd57bc65-ef53-4d9f-2a47-ce3e938a0309 fd57bc65-ef53-4d9f-2a47-ce3e938a0309}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2024-06-11T14:41:41.174Z [INFO] server-dec7d583-6925-2c2f-2390-bde74ba109d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dec7d583-6925-2c2f-2390-bde74ba109d1 Address:dec7d583-6925-2c2f-2390-bde74ba109d1} {Suffrage:Voter ID:server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e Address:955dd75a-b85c-1b27-8cbb-bc3edfd9753e} {Suffrage:Voter ID:server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44 Address:6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44}]" 2024-06-11T14:41:41.174Z [INFO] server-dec7d583-6925-2c2f-2390-bde74ba109d1: entering follower state: follower="Node at dec7d583-6925-2c2f-2390-bde74ba109d1 [Follower]" leader-address= leader-id= 2024-06-11T14:41:41.174Z [INFO] server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dec7d583-6925-2c2f-2390-bde74ba109d1 Address:dec7d583-6925-2c2f-2390-bde74ba109d1} {Suffrage:Voter ID:server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e Address:955dd75a-b85c-1b27-8cbb-bc3edfd9753e} {Suffrage:Voter ID:server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44 Address:6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44}]" 2024-06-11T14:41:41.174Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dec7d583-6925-2c2f-2390-bde74ba109d1 Address:dec7d583-6925-2c2f-2390-bde74ba109d1} {Suffrage:Voter ID:server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e Address:955dd75a-b85c-1b27-8cbb-bc3edfd9753e} {Suffrage:Voter ID:server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44 Address:6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44}]" 2024-06-11T14:41:41.174Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:41.175Z [INFO] server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e: entering follower state: follower="Node at 955dd75a-b85c-1b27-8cbb-bc3edfd9753e [Follower]" leader-address= leader-id= 2024-06-11T14:41:41.175Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: entering follower state: follower="Node at 6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44 [Follower]" leader-address= leader-id= 2024-06-11T14:41:41.226Z [WARN] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:41.226Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: entering candidate state: node="Node at 6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44 [Candidate]" term=2 2024-06-11T14:41:41.227Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: election won: term=2 tally=2 2024-06-11T14:41:41.227Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: entering leader state: leader="Node at 6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44 [Leader]" 2024-06-11T14:41:41.227Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: added peer, starting replication: peer=server-dec7d583-6925-2c2f-2390-bde74ba109d1 2024-06-11T14:41:41.227Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: added peer, starting replication: peer=server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e 2024-06-11T14:41:41.227Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: pipelining replication: peer="{Voter server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e 955dd75a-b85c-1b27-8cbb-bc3edfd9753e}" 2024-06-11T14:41:41.227Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: pipelining replication: peer="{Voter server-dec7d583-6925-2c2f-2390-bde74ba109d1 dec7d583-6925-2c2f-2390-bde74ba109d1}" 2024-06-11T14:41:41.333Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:41.227713264 +0000 UTC m=+107.810792215 (1 nodes), 52.728297ms from start of poll, 53.901735ms from cluster start. Timeout at 2024-06-11 14:41:41.333021344 +0000 UTC m=+107.916100296, 105.308081ms after stability 2024-06-11T14:41:41.333Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:41.439Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:41.333183498 +0000 UTC m=+107.916262450 (2 nodes), 2.84µs from start of poll, 159.37197ms from cluster start. Timeout at 2024-06-11 14:41:41.439313908 +0000 UTC m=+108.022392860, 106.13041ms after stability 2024-06-11T14:41:42.703Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: aborting pipeline replication: peer="{Voter server-dec7d583-6925-2c2f-2390-bde74ba109d1 dec7d583-6925-2c2f-2390-bde74ba109d1}" 2024-06-11T14:41:43.203Z [ERROR] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: failed to pipeline appendEntries: peer="{Voter server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e 955dd75a-b85c-1b27-8cbb-bc3edfd9753e}" error="command enqueue timeout" 2024-06-11T14:41:43.203Z [INFO] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: aborting pipeline replication: peer="{Voter server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e 955dd75a-b85c-1b27-8cbb-bc3edfd9753e}" 2024-06-11T14:41:43.203Z [ERROR] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: failed to heartbeat to: peer=dec7d583-6925-2c2f-2390-bde74ba109d1 backoff time=10ms error="command timed out" 2024-06-11T14:41:43.203Z [ERROR] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: failed to heartbeat to: peer=955dd75a-b85c-1b27-8cbb-bc3edfd9753e backoff time=10ms error="send timed out" 2024-06-11T14:41:43.203Z [ERROR] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: failed to appendEntries to: peer="{Voter server-dec7d583-6925-2c2f-2390-bde74ba109d1 dec7d583-6925-2c2f-2390-bde74ba109d1}" error="command timed out" 2024-06-11T14:41:43.705Z [ERROR] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: failed to appendEntries to: peer="{Voter server-955dd75a-b85c-1b27-8cbb-bc3edfd9753e 955dd75a-b85c-1b27-8cbb-bc3edfd9753e}" error="send timed out" 2024-06-11T14:41:43.724Z [ERROR] server-6f2ecc9e-e8f3-abf4-5393-f4a54c54bd44: failed to heartbeat to: peer=955dd75a-b85c-1b27-8cbb-bc3edfd9753e backoff time=10ms error="send timed out" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (2.55s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2024-06-11T14:41:43.726Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc Address:aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc} {Suffrage:Voter ID:server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8 Address:ffcc12fe-0802-a1f2-5660-d02fc1823ff8}]" 2024-06-11T14:41:43.726Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: entering follower state: follower="Node at aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc [Follower]" leader-address= leader-id= 2024-06-11T14:41:43.726Z [INFO] server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc Address:aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc} {Suffrage:Voter ID:server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8 Address:ffcc12fe-0802-a1f2-5660-d02fc1823ff8}]" 2024-06-11T14:41:43.726Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:43.727Z [INFO] server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8: entering follower state: follower="Node at ffcc12fe-0802-a1f2-5660-d02fc1823ff8 [Follower]" leader-address= leader-id= 2024-06-11T14:41:43.780Z [WARN] server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:43.780Z [INFO] server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8: entering candidate state: node="Node at ffcc12fe-0802-a1f2-5660-d02fc1823ff8 [Candidate]" term=2 2024-06-11T14:41:43.781Z [WARN] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:43.781Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: entering candidate state: node="Node at aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc [Candidate]" term=2 2024-06-11T14:41:43.781Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: duplicate requestVote for same term: term=2 2024-06-11T14:41:43.781Z [INFO] server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8: duplicate requestVote for same term: term=2 2024-06-11T14:41:43.838Z [WARN] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: Election timeout reached, restarting election 2024-06-11T14:41:43.838Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: entering candidate state: node="Node at aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc [Candidate]" term=3 2024-06-11T14:41:43.841Z [INFO] server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8: entering follower state: follower="Node at ffcc12fe-0802-a1f2-5660-d02fc1823ff8 [Follower]" leader-address= leader-id= 2024-06-11T14:41:43.841Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: election won: term=3 tally=2 2024-06-11T14:41:43.841Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: entering leader state: leader="Node at aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc [Leader]" 2024-06-11T14:41:43.841Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: added peer, starting replication: peer=server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8 2024-06-11T14:41:43.841Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: pipelining replication: peer="{Voter server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8 ffcc12fe-0802-a1f2-5660-d02fc1823ff8}" 2024-06-11T14:41:43.947Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:43.841444554 +0000 UTC m=+110.424523506 (1 nodes), 114.47255ms from start of poll, 114.966572ms from cluster start. Timeout at 2024-06-11 14:41:43.946890869 +0000 UTC m=+110.529969781, 105.446275ms after stability 2024-06-11T14:41:43.947Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:44.052Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:43.9471355 +0000 UTC m=+110.530214452 (1 nodes), 1.72µs from start of poll, 220.657518ms from cluster start. Timeout at 2024-06-11 14:41:44.05215855 +0000 UTC m=+110.635237502, 105.02305ms after stability 2024-06-11T14:41:44.052Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: updating configuration: command=DemoteVoter server-id=server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8 server-addr= servers="[{Suffrage:Voter ID:server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc Address:aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc} {Suffrage:Nonvoter ID:server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8 Address:ffcc12fe-0802-a1f2-5660-d02fc1823ff8}]" 2024-06-11T14:41:44.052Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:44.157Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:44.052693211 +0000 UTC m=+110.635772123 (1 nodes), 1.56µs from start of poll, 326.215189ms from cluster start. Timeout at 2024-06-11 14:41:44.157900175 +0000 UTC m=+110.740979087, 105.206964ms after stability 2024-06-11T14:41:44.158Z [INFO] server-aff0d8ea-8167-1c6f-b0d8-6fead3e0f5dc: aborting pipeline replication: peer="{Voter server-ffcc12fe-0802-a1f2-5660-d02fc1823ff8 ffcc12fe-0802-a1f2-5660-d02fc1823ff8}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.43s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2024-06-11T14:41:44.160Z [INFO] server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec Address:49bea988-951f-ea9f-2d2a-bc641f9ed4ec} {Suffrage:Voter ID:server-fe174f54-dc40-329a-2cb4-be309f317f9d Address:fe174f54-dc40-329a-2cb4-be309f317f9d}]" 2024-06-11T14:41:44.160Z [INFO] server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec: entering follower state: follower="Node at 49bea988-951f-ea9f-2d2a-bc641f9ed4ec [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.161Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec Address:49bea988-951f-ea9f-2d2a-bc641f9ed4ec} {Suffrage:Voter ID:server-fe174f54-dc40-329a-2cb4-be309f317f9d Address:fe174f54-dc40-329a-2cb4-be309f317f9d}]" 2024-06-11T14:41:44.161Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:44.161Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: entering follower state: follower="Node at fe174f54-dc40-329a-2cb4-be309f317f9d [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.214Z [WARN] server-fe174f54-dc40-329a-2cb4-be309f317f9d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:44.214Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: entering candidate state: node="Node at fe174f54-dc40-329a-2cb4-be309f317f9d [Candidate]" term=2 2024-06-11T14:41:44.214Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: election won: term=2 tally=2 2024-06-11T14:41:44.214Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: entering leader state: leader="Node at fe174f54-dc40-329a-2cb4-be309f317f9d [Leader]" 2024-06-11T14:41:44.214Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: added peer, starting replication: peer=server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec 2024-06-11T14:41:44.215Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: pipelining replication: peer="{Voter server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec 49bea988-951f-ea9f-2d2a-bc641f9ed4ec}" 2024-06-11T14:41:44.325Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:44.215048993 +0000 UTC m=+110.798127945 (1 nodes), 53.678543ms from start of poll, 55.087652ms from cluster start. Timeout at 2024-06-11 14:41:44.322358761 +0000 UTC m=+110.905437713, 107.309768ms after stability 2024-06-11T14:41:44.325Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:44.430Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:44.325744759 +0000 UTC m=+110.908823671 (1 nodes), 1.36µs from start of poll, 165.783378ms from cluster start. Timeout at 2024-06-11 14:41:44.430878365 +0000 UTC m=+111.013957317, 105.133646ms after stability 2024-06-11T14:41:44.441Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec Address:49bea988-951f-ea9f-2d2a-bc641f9ed4ec} {Suffrage:Voter ID:server-fe174f54-dc40-329a-2cb4-be309f317f9d Address:fe174f54-dc40-329a-2cb4-be309f317f9d}]" 2024-06-11T14:41:44.442Z [INFO] server-fe174f54-dc40-329a-2cb4-be309f317f9d: aborting pipeline replication: peer="{Voter server-49bea988-951f-ea9f-2d2a-bc641f9ed4ec 49bea988-951f-ea9f-2d2a-bc641f9ed4ec}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) === RUN TestRaft_LogStoreIsMonotonic 2024-06-11T14:41:44.443Z [INFO] server-e17e46e4-ba4c-0168-2b66-6ba9248e74e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e17e46e4-ba4c-0168-2b66-6ba9248e74e4 Address:e17e46e4-ba4c-0168-2b66-6ba9248e74e4}]" 2024-06-11T14:41:44.443Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:44.443Z [INFO] server-e17e46e4-ba4c-0168-2b66-6ba9248e74e4: entering follower state: follower="Node at e17e46e4-ba4c-0168-2b66-6ba9248e74e4 [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.535Z [WARN] server-e17e46e4-ba4c-0168-2b66-6ba9248e74e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:44.535Z [INFO] server-e17e46e4-ba4c-0168-2b66-6ba9248e74e4: entering candidate state: node="Node at e17e46e4-ba4c-0168-2b66-6ba9248e74e4 [Candidate]" term=2 2024-06-11T14:41:44.535Z [INFO] server-e17e46e4-ba4c-0168-2b66-6ba9248e74e4: election won: term=2 tally=1 2024-06-11T14:41:44.535Z [INFO] server-e17e46e4-ba4c-0168-2b66-6ba9248e74e4: entering leader state: leader="Node at e17e46e4-ba4c-0168-2b66-6ba9248e74e4 [Leader]" 2024-06-11T14:41:44.641Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:44.535422633 +0000 UTC m=+111.118501585 (1 nodes), 91.861965ms from start of poll, 92.158675ms from cluster start. Timeout at 2024-06-11 14:41:44.641208416 +0000 UTC m=+111.224287368, 105.785783ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.20s) === RUN TestRaft_CacheLogWithStoreError 2024-06-11T14:41:44.642Z [INFO] server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 Address:bc12ea0b-9d02-fbe2-938e-644cec9c8b32} {Suffrage:Voter ID:server-f075f6c5-df3c-a844-e45a-8c305df27a08 Address:f075f6c5-df3c-a844-e45a-8c305df27a08}]" 2024-06-11T14:41:44.643Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 Address:bc12ea0b-9d02-fbe2-938e-644cec9c8b32} {Suffrage:Voter ID:server-f075f6c5-df3c-a844-e45a-8c305df27a08 Address:f075f6c5-df3c-a844-e45a-8c305df27a08}]" 2024-06-11T14:41:44.643Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:44.643Z [INFO] server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32: entering follower state: follower="Node at bc12ea0b-9d02-fbe2-938e-644cec9c8b32 [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.643Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: entering follower state: follower="Node at f075f6c5-df3c-a844-e45a-8c305df27a08 [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.699Z [WARN] server-f075f6c5-df3c-a844-e45a-8c305df27a08: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:44.699Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: entering candidate state: node="Node at f075f6c5-df3c-a844-e45a-8c305df27a08 [Candidate]" term=2 2024-06-11T14:41:44.699Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: election won: term=2 tally=2 2024-06-11T14:41:44.699Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: entering leader state: leader="Node at f075f6c5-df3c-a844-e45a-8c305df27a08 [Leader]" 2024-06-11T14:41:44.699Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: added peer, starting replication: peer=server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 2024-06-11T14:41:44.699Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: pipelining replication: peer="{Voter server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 bc12ea0b-9d02-fbe2-938e-644cec9c8b32}" 2024-06-11T14:41:44.805Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:44.699645987 +0000 UTC m=+111.282724939 (1 nodes), 56.411724ms from start of poll, 56.869748ms from cluster start. Timeout at 2024-06-11 14:41:44.805271736 +0000 UTC m=+111.388350688, 105.625749ms after stability 2024-06-11T14:41:44.805Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:44.911Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:44.80543445 +0000 UTC m=+111.388513402 (1 nodes), 1.64µs from start of poll, 162.658211ms from cluster start. Timeout at 2024-06-11 14:41:44.911661137 +0000 UTC m=+111.494740089, 106.226687ms after stability 2024-06-11T14:41:44.913Z [ERROR] server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2024-06-11T14:41:44.913Z [WARN] server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2024-06-11T14:41:44.913Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: aborting pipeline replication: peer="{Voter server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 bc12ea0b-9d02-fbe2-938e-644cec9c8b32}" 2024-06-11T14:41:44.920Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: pipelining replication: peer="{Voter server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 bc12ea0b-9d02-fbe2-938e-644cec9c8b32}" 2024-06-11T14:41:44.921Z [INFO] server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 Address:bc12ea0b-9d02-fbe2-938e-644cec9c8b32} {Suffrage:Voter ID:server-f075f6c5-df3c-a844-e45a-8c305df27a08 Address:f075f6c5-df3c-a844-e45a-8c305df27a08}]" 2024-06-11T14:41:44.921Z [INFO] server-f075f6c5-df3c-a844-e45a-8c305df27a08: aborting pipeline replication: peer="{Voter server-bc12ea0b-9d02-fbe2-938e-644cec9c8b32 bc12ea0b-9d02-fbe2-938e-644cec9c8b32}" --- PASS: TestRaft_CacheLogWithStoreError (0.28s) === RUN TestRaft_ReloadConfig 2024-06-11T14:41:44.922Z [INFO] server-84ed6916-688b-f4d1-7cf3-8d17066dff65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84ed6916-688b-f4d1-7cf3-8d17066dff65 Address:84ed6916-688b-f4d1-7cf3-8d17066dff65}]" --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2024-06-11T14:41:44.923Z [INFO] server-ba336610-5d27-dde3-e031-adc314f4562b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba336610-5d27-dde3-e031-adc314f4562b Address:ba336610-5d27-dde3-e031-adc314f4562b}]" --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2024-06-11T14:41:44.924Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2024-06-11T14:41:44.925Z [INFO] server-44cfcbec-4959-cc19-ef01-f585e2df8a1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44cfcbec-4959-cc19-ef01-f585e2df8a1a Address:44cfcbec-4959-cc19-ef01-f585e2df8a1a} {Suffrage:Voter ID:server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e Address:43d54a78-dbee-1c6a-d31c-829e19e2fe6e} {Suffrage:Voter ID:server-ce8a7d42-3954-43b5-831b-b7ad4a07f081 Address:ce8a7d42-3954-43b5-831b-b7ad4a07f081}]" 2024-06-11T14:41:44.926Z [INFO] server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44cfcbec-4959-cc19-ef01-f585e2df8a1a Address:44cfcbec-4959-cc19-ef01-f585e2df8a1a} {Suffrage:Voter ID:server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e Address:43d54a78-dbee-1c6a-d31c-829e19e2fe6e} {Suffrage:Voter ID:server-ce8a7d42-3954-43b5-831b-b7ad4a07f081 Address:ce8a7d42-3954-43b5-831b-b7ad4a07f081}]" 2024-06-11T14:41:44.926Z [INFO] server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e: entering follower state: follower="Node at 43d54a78-dbee-1c6a-d31c-829e19e2fe6e [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.926Z [INFO] server-44cfcbec-4959-cc19-ef01-f585e2df8a1a: entering follower state: follower="Node at 44cfcbec-4959-cc19-ef01-f585e2df8a1a [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.929Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44cfcbec-4959-cc19-ef01-f585e2df8a1a Address:44cfcbec-4959-cc19-ef01-f585e2df8a1a} {Suffrage:Voter ID:server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e Address:43d54a78-dbee-1c6a-d31c-829e19e2fe6e} {Suffrage:Voter ID:server-ce8a7d42-3954-43b5-831b-b7ad4a07f081 Address:ce8a7d42-3954-43b5-831b-b7ad4a07f081}]" 2024-06-11T14:41:44.929Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:44.930Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: entering follower state: follower="Node at ce8a7d42-3954-43b5-831b-b7ad4a07f081 [Follower]" leader-address= leader-id= 2024-06-11T14:41:44.987Z [WARN] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:44.987Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: entering candidate state: node="Node at ce8a7d42-3954-43b5-831b-b7ad4a07f081 [Candidate]" term=2 2024-06-11T14:41:44.987Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: election won: term=2 tally=2 2024-06-11T14:41:44.987Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: entering leader state: leader="Node at ce8a7d42-3954-43b5-831b-b7ad4a07f081 [Leader]" 2024-06-11T14:41:44.987Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: added peer, starting replication: peer=server-44cfcbec-4959-cc19-ef01-f585e2df8a1a 2024-06-11T14:41:44.987Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: added peer, starting replication: peer=server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e 2024-06-11T14:41:44.987Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: pipelining replication: peer="{Voter server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e 43d54a78-dbee-1c6a-d31c-829e19e2fe6e}" 2024-06-11T14:41:44.988Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: pipelining replication: peer="{Voter server-44cfcbec-4959-cc19-ef01-f585e2df8a1a 44cfcbec-4959-cc19-ef01-f585e2df8a1a}" 2024-06-11T14:41:45.093Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:44.988323011 +0000 UTC m=+111.571401923 (1 nodes), 59.125586ms from start of poll, 62.698297ms from cluster start. Timeout at 2024-06-11 14:41:45.093546095 +0000 UTC m=+111.676625007, 105.223084ms after stability 2024-06-11T14:41:45.099Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:45.205Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:45.099196571 +0000 UTC m=+111.682275483 (2 nodes), 1.88µs from start of poll, 173.571857ms from cluster start. Timeout at 2024-06-11 14:41:45.205127748 +0000 UTC m=+111.788206700, 105.931217ms after stability 2024-06-11T14:41:45.205Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: updating configuration: command=RemoveServer server-id=server-44cfcbec-4959-cc19-ef01-f585e2df8a1a server-addr= servers="[{Suffrage:Voter ID:server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e Address:43d54a78-dbee-1c6a-d31c-829e19e2fe6e} {Suffrage:Voter ID:server-ce8a7d42-3954-43b5-831b-b7ad4a07f081 Address:ce8a7d42-3954-43b5-831b-b7ad4a07f081}]" 2024-06-11T14:41:45.205Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: removed peer, stopping replication: peer=server-44cfcbec-4959-cc19-ef01-f585e2df8a1a last-index=3 2024-06-11T14:41:45.205Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: aborting pipeline replication: peer="{Voter server-44cfcbec-4959-cc19-ef01-f585e2df8a1a 44cfcbec-4959-cc19-ef01-f585e2df8a1a}" 2024-06-11T14:41:45.306Z [WARN] server-44cfcbec-4959-cc19-ef01-f585e2df8a1a: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2024-06-11T14:41:45.315Z [ERROR] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to heartbeat to: peer=43d54a78-dbee-1c6a-d31c-829e19e2fe6e backoff time=10ms error="failed to connect to peer: 43d54a78-dbee-1c6a-d31c-829e19e2fe6e" 2024-06-11T14:41:45.318Z [ERROR] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to pipeline appendEntries: peer="{Voter server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e 43d54a78-dbee-1c6a-d31c-829e19e2fe6e}" error="append pipeline closed" 2024-06-11T14:41:45.318Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: aborting pipeline replication: peer="{Voter server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e 43d54a78-dbee-1c6a-d31c-829e19e2fe6e}" 2024-06-11T14:41:45.328Z [ERROR] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to appendEntries to: peer="{Voter server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e 43d54a78-dbee-1c6a-d31c-829e19e2fe6e}" error="failed to connect to peer: 43d54a78-dbee-1c6a-d31c-829e19e2fe6e" 2024-06-11T14:41:45.335Z [ERROR] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to heartbeat to: peer=43d54a78-dbee-1c6a-d31c-829e19e2fe6e backoff time=10ms error="failed to connect to peer: 43d54a78-dbee-1c6a-d31c-829e19e2fe6e" 2024-06-11T14:41:45.349Z [ERROR] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to appendEntries to: peer="{Voter server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e 43d54a78-dbee-1c6a-d31c-829e19e2fe6e}" error="failed to connect to peer: 43d54a78-dbee-1c6a-d31c-829e19e2fe6e" 2024-06-11T14:41:45.351Z [ERROR] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to heartbeat to: peer=43d54a78-dbee-1c6a-d31c-829e19e2fe6e backoff time=10ms error="failed to connect to peer: 43d54a78-dbee-1c6a-d31c-829e19e2fe6e" 2024-06-11T14:41:45.361Z [WARN] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to contact: server-id=server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e time=50.960361ms 2024-06-11T14:41:45.361Z [WARN] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to contact quorum of nodes, stepping down 2024-06-11T14:41:45.361Z [INFO] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: entering follower state: follower="Node at ce8a7d42-3954-43b5-831b-b7ad4a07f081 [Follower]" leader-address= leader-id= 2024-06-11T14:41:45.366Z [ERROR] server-ce8a7d42-3954-43b5-831b-b7ad4a07f081: failed to appendEntries to: peer="{Voter server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e 43d54a78-dbee-1c6a-d31c-829e19e2fe6e}" error="failed to connect to peer: 43d54a78-dbee-1c6a-d31c-829e19e2fe6e" 2024-06-11T14:41:45.373Z [WARN] server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e: heartbeat timeout reached, starting election: last-leader-addr=ce8a7d42-3954-43b5-831b-b7ad4a07f081 last-leader-id=server-ce8a7d42-3954-43b5-831b-b7ad4a07f081 2024-06-11T14:41:45.373Z [INFO] server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e: entering candidate state: node="Node at 43d54a78-dbee-1c6a-d31c-829e19e2fe6e [Candidate]" term=3 2024-06-11T14:41:45.373Z [ERROR] server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e: failed to make requestVote RPC: target="{Voter server-ce8a7d42-3954-43b5-831b-b7ad4a07f081 ce8a7d42-3954-43b5-831b-b7ad4a07f081}" error="failed to connect to peer: ce8a7d42-3954-43b5-831b-b7ad4a07f081" term=3 2024-06-11T14:41:45.417Z [WARN] server-43d54a78-dbee-1c6a-d31c-829e19e2fe6e: rejecting vote request since node is not in configuration: from=44cfcbec-4959-cc19-ef01-f585e2df8a1a --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.49s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2024-06-11T14:41:45.420Z [INFO] server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 Address:513d8ab2-1e5c-9e48-c831-5e7ce6765a97} {Suffrage:Voter ID:server-a045bb67-9c05-29e5-a174-bba3e0028b6e Address:a045bb67-9c05-29e5-a174-bba3e0028b6e} {Suffrage:Voter ID:server-bde4e140-66ce-3230-f547-de184f270711 Address:bde4e140-66ce-3230-f547-de184f270711}]" 2024-06-11T14:41:45.420Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 Address:513d8ab2-1e5c-9e48-c831-5e7ce6765a97} {Suffrage:Voter ID:server-a045bb67-9c05-29e5-a174-bba3e0028b6e Address:a045bb67-9c05-29e5-a174-bba3e0028b6e} {Suffrage:Voter ID:server-bde4e140-66ce-3230-f547-de184f270711 Address:bde4e140-66ce-3230-f547-de184f270711}]" 2024-06-11T14:41:45.420Z [INFO] server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97: entering follower state: follower="Node at 513d8ab2-1e5c-9e48-c831-5e7ce6765a97 [Follower]" leader-address= leader-id= 2024-06-11T14:41:45.420Z [INFO] server-bde4e140-66ce-3230-f547-de184f270711: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 Address:513d8ab2-1e5c-9e48-c831-5e7ce6765a97} {Suffrage:Voter ID:server-a045bb67-9c05-29e5-a174-bba3e0028b6e Address:a045bb67-9c05-29e5-a174-bba3e0028b6e} {Suffrage:Voter ID:server-bde4e140-66ce-3230-f547-de184f270711 Address:bde4e140-66ce-3230-f547-de184f270711}]" 2024-06-11T14:41:45.420Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: entering follower state: follower="Node at a045bb67-9c05-29e5-a174-bba3e0028b6e [Follower]" leader-address= leader-id= 2024-06-11T14:41:45.420Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:45.420Z [INFO] server-bde4e140-66ce-3230-f547-de184f270711: entering follower state: follower="Node at bde4e140-66ce-3230-f547-de184f270711 [Follower]" leader-address= leader-id= 2024-06-11T14:41:45.471Z [WARN] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:45.472Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: entering candidate state: node="Node at a045bb67-9c05-29e5-a174-bba3e0028b6e [Candidate]" term=2 2024-06-11T14:41:45.472Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: election won: term=2 tally=2 2024-06-11T14:41:45.472Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: entering leader state: leader="Node at a045bb67-9c05-29e5-a174-bba3e0028b6e [Leader]" 2024-06-11T14:41:45.472Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: added peer, starting replication: peer=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 2024-06-11T14:41:45.472Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: added peer, starting replication: peer=server-bde4e140-66ce-3230-f547-de184f270711 2024-06-11T14:41:45.472Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: pipelining replication: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:45.472Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: pipelining replication: peer="{Voter server-bde4e140-66ce-3230-f547-de184f270711 bde4e140-66ce-3230-f547-de184f270711}" 2024-06-11T14:41:45.578Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:45.472347426 +0000 UTC m=+112.055426378 (1 nodes), 51.603658ms from start of poll, 52.349511ms from cluster start. Timeout at 2024-06-11 14:41:45.578244685 +0000 UTC m=+112.161323637, 105.897259ms after stability 2024-06-11T14:41:45.583Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:45.698Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:45.583837244 +0000 UTC m=+112.166916196 (2 nodes), 2.16µs from start of poll, 163.839329ms from cluster start. Timeout at 2024-06-11 14:41:45.694699443 +0000 UTC m=+112.277778395, 110.862199ms after stability 2024-06-11T14:41:45.701Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to pipeline appendEntries: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" error="append pipeline closed" 2024-06-11T14:41:45.701Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: aborting pipeline replication: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:45.708Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to heartbeat to: peer=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 backoff time=10ms error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.711Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to appendEntries to: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.725Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to heartbeat to: peer=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 backoff time=10ms error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.732Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to appendEntries to: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.746Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to heartbeat to: peer=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 backoff time=10ms error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.749Z [WARN] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to contact: server-id=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 time=51.114675ms 2024-06-11T14:41:45.750Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to appendEntries to: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.764Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to heartbeat to: peer=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 backoff time=20ms error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.777Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to appendEntries to: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.790Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to heartbeat to: peer=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 backoff time=25ms error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.798Z [WARN] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to contact: server-id=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 time=100.3071ms 2024-06-11T14:41:45.804Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: updating configuration: command=RemoveServer server-id=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 server-addr= servers="[{Suffrage:Voter ID:server-a045bb67-9c05-29e5-a174-bba3e0028b6e Address:a045bb67-9c05-29e5-a174-bba3e0028b6e} {Suffrage:Voter ID:server-bde4e140-66ce-3230-f547-de184f270711 Address:bde4e140-66ce-3230-f547-de184f270711}]" 2024-06-11T14:41:45.804Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: removed peer, stopping replication: peer=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 last-index=3 2024-06-11T14:41:45.804Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: updating configuration: command=AddNonvoter server-id=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 server-addr=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 servers="[{Suffrage:Voter ID:server-a045bb67-9c05-29e5-a174-bba3e0028b6e Address:a045bb67-9c05-29e5-a174-bba3e0028b6e} {Suffrage:Voter ID:server-bde4e140-66ce-3230-f547-de184f270711 Address:bde4e140-66ce-3230-f547-de184f270711} {Suffrage:Nonvoter ID:server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 Address:513d8ab2-1e5c-9e48-c831-5e7ce6765a97}]" 2024-06-11T14:41:45.805Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: added peer, starting replication: peer=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 2024-06-11T14:41:45.805Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to appendEntries to: peer="{Nonvoter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" error="failed to connect to peer: 513d8ab2-1e5c-9e48-c831-5e7ce6765a97" 2024-06-11T14:41:45.824Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: peer has newer term, stopping replication: peer="{Nonvoter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:45.825Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: entering follower state: follower="Node at a045bb67-9c05-29e5-a174-bba3e0028b6e [Follower]" leader-address= leader-id= 2024-06-11T14:41:45.825Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: aborting pipeline replication: peer="{Voter server-bde4e140-66ce-3230-f547-de184f270711 bde4e140-66ce-3230-f547-de184f270711}" 2024-06-11T14:41:45.827Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: peer has newer term, stopping replication: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:45.830Z [WARN] server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97: heartbeat timeout reached, starting election: last-leader-addr=a045bb67-9c05-29e5-a174-bba3e0028b6e last-leader-id=server-a045bb67-9c05-29e5-a174-bba3e0028b6e 2024-06-11T14:41:45.830Z [INFO] server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97: entering candidate state: node="Node at 513d8ab2-1e5c-9e48-c831-5e7ce6765a97 [Candidate]" term=23 2024-06-11T14:41:45.830Z [WARN] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: rejecting vote request since node is not a voter: from=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 2024-06-11T14:41:45.830Z [WARN] server-bde4e140-66ce-3230-f547-de184f270711: rejecting vote request since we have a leader: from=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 leader=a045bb67-9c05-29e5-a174-bba3e0028b6e leader-id=server-a045bb67-9c05-29e5-a174-bba3e0028b6e 2024-06-11T14:41:45.905Z [WARN] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:45.905Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: entering candidate state: node="Node at a045bb67-9c05-29e5-a174-bba3e0028b6e [Candidate]" term=24 2024-06-11T14:41:45.905Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: election won: term=24 tally=2 2024-06-11T14:41:45.905Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: entering leader state: leader="Node at a045bb67-9c05-29e5-a174-bba3e0028b6e [Leader]" 2024-06-11T14:41:45.905Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: added peer, starting replication: peer=server-bde4e140-66ce-3230-f547-de184f270711 2024-06-11T14:41:45.905Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: added peer, starting replication: peer=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 2024-06-11T14:41:45.905Z [WARN] server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97: failed to get previous log: previous-index=4 last-index=2 error="log not found" 2024-06-11T14:41:45.905Z [INFO] server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97: entering follower state: follower="Node at 513d8ab2-1e5c-9e48-c831-5e7ce6765a97 [Follower]" leader-address=a045bb67-9c05-29e5-a174-bba3e0028b6e leader-id=server-a045bb67-9c05-29e5-a174-bba3e0028b6e 2024-06-11T14:41:45.906Z [WARN] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: appendEntries rejected, sending older logs: peer="{Nonvoter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" next=3 2024-06-11T14:41:45.906Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: pipelining replication: peer="{Nonvoter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:45.906Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: pipelining replication: peer="{Voter server-bde4e140-66ce-3230-f547-de184f270711 bde4e140-66ce-3230-f547-de184f270711}" 2024-06-11T14:41:45.911Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:46.016Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:45.911210271 +0000 UTC m=+112.494289223 (1 nodes), 84.717µs from start of poll, 491.212356ms from cluster start. Timeout at 2024-06-11 14:41:46.016715944 +0000 UTC m=+112.599794856, 105.505633ms after stability 2024-06-11T14:41:46.070Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: updating configuration: command=RemoveServer server-id=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 server-addr= servers="[{Suffrage:Voter ID:server-a045bb67-9c05-29e5-a174-bba3e0028b6e Address:a045bb67-9c05-29e5-a174-bba3e0028b6e} {Suffrage:Voter ID:server-bde4e140-66ce-3230-f547-de184f270711 Address:bde4e140-66ce-3230-f547-de184f270711}]" 2024-06-11T14:41:46.071Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: removed peer, stopping replication: peer=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 last-index=106 2024-06-11T14:41:46.071Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: aborting pipeline replication: peer="{Nonvoter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:46.071Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: updating configuration: command=AddVoter server-id=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 server-addr=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 servers="[{Suffrage:Voter ID:server-a045bb67-9c05-29e5-a174-bba3e0028b6e Address:a045bb67-9c05-29e5-a174-bba3e0028b6e} {Suffrage:Voter ID:server-bde4e140-66ce-3230-f547-de184f270711 Address:bde4e140-66ce-3230-f547-de184f270711} {Suffrage:Voter ID:server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 Address:513d8ab2-1e5c-9e48-c831-5e7ce6765a97}]" 2024-06-11T14:41:46.071Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: added peer, starting replication: peer=server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 2024-06-11T14:41:46.071Z [WARN] server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97: failed to get previous log: previous-index=107 last-index=106 error="log not found" 2024-06-11T14:41:46.072Z [WARN] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: appendEntries rejected, sending older logs: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" next=107 2024-06-11T14:41:46.072Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: pipelining replication: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:47.166Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:47.271Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:47.16641514 +0000 UTC m=+113.749494092 (1 nodes), 1.64µs from start of poll, 1.746417225s from cluster start. Timeout at 2024-06-11 14:41:47.271795858 +0000 UTC m=+113.854874770, 105.380678ms after stability 2024-06-11T14:41:47.272Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: aborting pipeline replication: peer="{Voter server-513d8ab2-1e5c-9e48-c831-5e7ce6765a97 513d8ab2-1e5c-9e48-c831-5e7ce6765a97}" 2024-06-11T14:41:47.272Z [INFO] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: aborting pipeline replication: peer="{Voter server-bde4e140-66ce-3230-f547-de184f270711 bde4e140-66ce-3230-f547-de184f270711}" 2024-06-11T14:41:47.772Z [ERROR] server-a045bb67-9c05-29e5-a174-bba3e0028b6e: failed to heartbeat to: peer=513d8ab2-1e5c-9e48-c831-5e7ce6765a97 backoff time=10ms error="command timed out" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (2.36s) === RUN TestRaft_FollowerRemovalNoElection 2024-06-11T14:41:47.775Z [INFO] server-1045014b-e8dc-97de-2df4-ebefe9b78d9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1045014b-e8dc-97de-2df4-ebefe9b78d9b Address:1045014b-e8dc-97de-2df4-ebefe9b78d9b} {Suffrage:Voter ID:server-161a8333-47c9-57ee-14fd-64b15b0092fb Address:161a8333-47c9-57ee-14fd-64b15b0092fb} {Suffrage:Voter ID:server-a300e903-a649-40ba-dbfb-e4f415e8be9e Address:a300e903-a649-40ba-dbfb-e4f415e8be9e}]" 2024-06-11T14:41:47.775Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1045014b-e8dc-97de-2df4-ebefe9b78d9b Address:1045014b-e8dc-97de-2df4-ebefe9b78d9b} {Suffrage:Voter ID:server-161a8333-47c9-57ee-14fd-64b15b0092fb Address:161a8333-47c9-57ee-14fd-64b15b0092fb} {Suffrage:Voter ID:server-a300e903-a649-40ba-dbfb-e4f415e8be9e Address:a300e903-a649-40ba-dbfb-e4f415e8be9e}]" 2024-06-11T14:41:47.775Z [INFO] server-1045014b-e8dc-97de-2df4-ebefe9b78d9b: entering follower state: follower="Node at 1045014b-e8dc-97de-2df4-ebefe9b78d9b [Follower]" leader-address= leader-id= 2024-06-11T14:41:47.775Z [INFO] server-a300e903-a649-40ba-dbfb-e4f415e8be9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1045014b-e8dc-97de-2df4-ebefe9b78d9b Address:1045014b-e8dc-97de-2df4-ebefe9b78d9b} {Suffrage:Voter ID:server-161a8333-47c9-57ee-14fd-64b15b0092fb Address:161a8333-47c9-57ee-14fd-64b15b0092fb} {Suffrage:Voter ID:server-a300e903-a649-40ba-dbfb-e4f415e8be9e Address:a300e903-a649-40ba-dbfb-e4f415e8be9e}]" 2024-06-11T14:41:47.775Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: entering follower state: follower="Node at 161a8333-47c9-57ee-14fd-64b15b0092fb [Follower]" leader-address= leader-id= 2024-06-11T14:41:47.775Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:47.776Z [INFO] server-a300e903-a649-40ba-dbfb-e4f415e8be9e: entering follower state: follower="Node at a300e903-a649-40ba-dbfb-e4f415e8be9e [Follower]" leader-address= leader-id= 2024-06-11T14:41:47.878Z [WARN] server-161a8333-47c9-57ee-14fd-64b15b0092fb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:47.878Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: entering candidate state: node="Node at 161a8333-47c9-57ee-14fd-64b15b0092fb [Candidate]" term=2 2024-06-11T14:41:47.878Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: election won: term=2 tally=2 2024-06-11T14:41:47.878Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: entering leader state: leader="Node at 161a8333-47c9-57ee-14fd-64b15b0092fb [Leader]" 2024-06-11T14:41:47.878Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: added peer, starting replication: peer=server-1045014b-e8dc-97de-2df4-ebefe9b78d9b 2024-06-11T14:41:47.878Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: added peer, starting replication: peer=server-a300e903-a649-40ba-dbfb-e4f415e8be9e 2024-06-11T14:41:47.878Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: pipelining replication: peer="{Voter server-a300e903-a649-40ba-dbfb-e4f415e8be9e a300e903-a649-40ba-dbfb-e4f415e8be9e}" 2024-06-11T14:41:47.878Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: pipelining replication: peer="{Voter server-1045014b-e8dc-97de-2df4-ebefe9b78d9b 1045014b-e8dc-97de-2df4-ebefe9b78d9b}" 2024-06-11T14:41:48.084Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:47.878851874 +0000 UTC m=+114.461930786 (1 nodes), 102.875208ms from start of poll, 103.558624ms from cluster start. Timeout at 2024-06-11 14:41:48.084440336 +0000 UTC m=+114.667519288, 205.588502ms after stability 2024-06-11T14:41:48.084Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:48.290Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:48.084685007 +0000 UTC m=+114.667763959 (1 nodes), 2.6µs from start of poll, 309.391797ms from cluster start. Timeout at 2024-06-11 14:41:48.290069436 +0000 UTC m=+114.873148348, 205.384389ms after stability 2024-06-11T14:41:48.295Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:48.500Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:48.295310687 +0000 UTC m=+114.878389639 (2 nodes), 2.32µs from start of poll, 520.017477ms from cluster start. Timeout at 2024-06-11 14:41:48.500378728 +0000 UTC m=+115.083457640, 205.068001ms after stability 2024-06-11T14:41:48.500Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:48.706Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:48.500538162 +0000 UTC m=+115.083617114 (1 nodes), 1.56µs from start of poll, 725.244952ms from cluster start. Timeout at 2024-06-11 14:41:48.706569368 +0000 UTC m=+115.289648280, 206.031166ms after stability raft_test.go:2961: [INFO] restarting Node at 1045014b-e8dc-97de-2df4-ebefe9b78d9b [Follower] 2024-06-11T14:41:48.707Z [INFO] server-1045014b-e8dc-97de-2df4-ebefe9b78d9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1045014b-e8dc-97de-2df4-ebefe9b78d9b Address:1045014b-e8dc-97de-2df4-ebefe9b78d9b} {Suffrage:Voter ID:server-161a8333-47c9-57ee-14fd-64b15b0092fb Address:161a8333-47c9-57ee-14fd-64b15b0092fb} {Suffrage:Voter ID:server-a300e903-a649-40ba-dbfb-e4f415e8be9e Address:a300e903-a649-40ba-dbfb-e4f415e8be9e}]" 2024-06-11T14:41:48.707Z [INFO] server-1045014b-e8dc-97de-2df4-ebefe9b78d9b: entering follower state: follower="Node at 1045014b-e8dc-97de-2df4-ebefe9b78d9b [Follower]" leader-address= leader-id= 2024-06-11T14:41:48.958Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: aborting pipeline replication: peer="{Voter server-a300e903-a649-40ba-dbfb-e4f415e8be9e a300e903-a649-40ba-dbfb-e4f415e8be9e}" 2024-06-11T14:41:49.339Z [ERROR] server-161a8333-47c9-57ee-14fd-64b15b0092fb: failed to pipeline appendEntries: peer="{Voter server-1045014b-e8dc-97de-2df4-ebefe9b78d9b 1045014b-e8dc-97de-2df4-ebefe9b78d9b}" error="command enqueue timeout" 2024-06-11T14:41:49.339Z [INFO] server-161a8333-47c9-57ee-14fd-64b15b0092fb: aborting pipeline replication: peer="{Voter server-1045014b-e8dc-97de-2df4-ebefe9b78d9b 1045014b-e8dc-97de-2df4-ebefe9b78d9b}" 2024-06-11T14:41:49.474Z [ERROR] server-161a8333-47c9-57ee-14fd-64b15b0092fb: failed to heartbeat to: peer=1045014b-e8dc-97de-2df4-ebefe9b78d9b backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.70s) === RUN TestRaft_VoteWithNoIDNoAddr 2024-06-11T14:41:49.477Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf Address:bfbd3367-a9df-8fe4-3e94-db4d9db85ecf} {Suffrage:Voter ID:server-bc52a834-10eb-32b6-7563-6e51c2b4939d Address:bc52a834-10eb-32b6-7563-6e51c2b4939d} {Suffrage:Voter ID:server-d37df15c-2879-fa0f-efce-0f7a11a0d197 Address:d37df15c-2879-fa0f-efce-0f7a11a0d197}]" 2024-06-11T14:41:49.477Z [INFO] server-bc52a834-10eb-32b6-7563-6e51c2b4939d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf Address:bfbd3367-a9df-8fe4-3e94-db4d9db85ecf} {Suffrage:Voter ID:server-bc52a834-10eb-32b6-7563-6e51c2b4939d Address:bc52a834-10eb-32b6-7563-6e51c2b4939d} {Suffrage:Voter ID:server-d37df15c-2879-fa0f-efce-0f7a11a0d197 Address:d37df15c-2879-fa0f-efce-0f7a11a0d197}]" 2024-06-11T14:41:49.477Z [INFO] server-d37df15c-2879-fa0f-efce-0f7a11a0d197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf Address:bfbd3367-a9df-8fe4-3e94-db4d9db85ecf} {Suffrage:Voter ID:server-bc52a834-10eb-32b6-7563-6e51c2b4939d Address:bc52a834-10eb-32b6-7563-6e51c2b4939d} {Suffrage:Voter ID:server-d37df15c-2879-fa0f-efce-0f7a11a0d197 Address:d37df15c-2879-fa0f-efce-0f7a11a0d197}]" 2024-06-11T14:41:49.477Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: entering follower state: follower="Node at bfbd3367-a9df-8fe4-3e94-db4d9db85ecf [Follower]" leader-address= leader-id= 2024-06-11T14:41:49.477Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:49.477Z [INFO] server-bc52a834-10eb-32b6-7563-6e51c2b4939d: entering follower state: follower="Node at bc52a834-10eb-32b6-7563-6e51c2b4939d [Follower]" leader-address= leader-id= 2024-06-11T14:41:49.477Z [INFO] server-d37df15c-2879-fa0f-efce-0f7a11a0d197: entering follower state: follower="Node at d37df15c-2879-fa0f-efce-0f7a11a0d197 [Follower]" leader-address= leader-id= 2024-06-11T14:41:49.527Z [WARN] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-11T14:41:49.527Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: entering candidate state: node="Node at bfbd3367-a9df-8fe4-3e94-db4d9db85ecf [Candidate]" term=2 2024-06-11T14:41:49.527Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: election won: term=2 tally=2 2024-06-11T14:41:49.527Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: entering leader state: leader="Node at bfbd3367-a9df-8fe4-3e94-db4d9db85ecf [Leader]" 2024-06-11T14:41:49.527Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: added peer, starting replication: peer=server-bc52a834-10eb-32b6-7563-6e51c2b4939d 2024-06-11T14:41:49.527Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: added peer, starting replication: peer=server-d37df15c-2879-fa0f-efce-0f7a11a0d197 2024-06-11T14:41:49.528Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: pipelining replication: peer="{Voter server-d37df15c-2879-fa0f-efce-0f7a11a0d197 d37df15c-2879-fa0f-efce-0f7a11a0d197}" 2024-06-11T14:41:49.528Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: pipelining replication: peer="{Voter server-bc52a834-10eb-32b6-7563-6e51c2b4939d bc52a834-10eb-32b6-7563-6e51c2b4939d}" 2024-06-11T14:41:49.633Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:49.52796501 +0000 UTC m=+116.111043922 (1 nodes), 50.465139ms from start of poll, 51.106596ms from cluster start. Timeout at 2024-06-11 14:41:49.633375326 +0000 UTC m=+116.216454238, 105.410316ms after stability 2024-06-11T14:41:49.633Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-11T14:41:49.739Z [INFO] cluster: stable state for Leader reached at 2024-06-11 14:41:49.633528121 +0000 UTC m=+116.216607073 (1 nodes), 1.68µs from start of poll, 156.669747ms from cluster start. Timeout at 2024-06-11 14:41:49.739314704 +0000 UTC m=+116.322393656, 105.786583ms after stability 2024-06-11T14:41:49.744Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-11T14:41:49.850Z [INFO] cluster: stable state for Follower reached at 2024-06-11 14:41:49.744888623 +0000 UTC m=+116.327967535 (2 nodes), 1.84µs from start of poll, 268.030209ms from cluster start. Timeout at 2024-06-11 14:41:49.850764602 +0000 UTC m=+116.433843554, 105.876019ms after stability 2024-06-11T14:41:49.855Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to heartbeat to: peer=d37df15c-2879-fa0f-efce-0f7a11a0d197 backoff time=10ms error="failed to connect to peer: d37df15c-2879-fa0f-efce-0f7a11a0d197" 2024-06-11T14:41:49.857Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to pipeline appendEntries: peer="{Voter server-d37df15c-2879-fa0f-efce-0f7a11a0d197 d37df15c-2879-fa0f-efce-0f7a11a0d197}" error="append pipeline closed" 2024-06-11T14:41:49.857Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: aborting pipeline replication: peer="{Voter server-d37df15c-2879-fa0f-efce-0f7a11a0d197 d37df15c-2879-fa0f-efce-0f7a11a0d197}" 2024-06-11T14:41:49.858Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to pipeline appendEntries: peer="{Voter server-bc52a834-10eb-32b6-7563-6e51c2b4939d bc52a834-10eb-32b6-7563-6e51c2b4939d}" error="append pipeline closed" 2024-06-11T14:41:49.858Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: aborting pipeline replication: peer="{Voter server-bc52a834-10eb-32b6-7563-6e51c2b4939d bc52a834-10eb-32b6-7563-6e51c2b4939d}" 2024-06-11T14:41:49.860Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to heartbeat to: peer=bc52a834-10eb-32b6-7563-6e51c2b4939d backoff time=10ms error="failed to connect to peer: bc52a834-10eb-32b6-7563-6e51c2b4939d" 2024-06-11T14:41:49.864Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to appendEntries to: peer="{Voter server-d37df15c-2879-fa0f-efce-0f7a11a0d197 d37df15c-2879-fa0f-efce-0f7a11a0d197}" error="failed to connect to peer: d37df15c-2879-fa0f-efce-0f7a11a0d197" 2024-06-11T14:41:49.868Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to appendEntries to: peer="{Voter server-bc52a834-10eb-32b6-7563-6e51c2b4939d bc52a834-10eb-32b6-7563-6e51c2b4939d}" error="failed to connect to peer: bc52a834-10eb-32b6-7563-6e51c2b4939d" 2024-06-11T14:41:49.872Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to heartbeat to: peer=d37df15c-2879-fa0f-efce-0f7a11a0d197 backoff time=10ms error="failed to connect to peer: d37df15c-2879-fa0f-efce-0f7a11a0d197" 2024-06-11T14:41:49.879Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to heartbeat to: peer=bc52a834-10eb-32b6-7563-6e51c2b4939d backoff time=10ms error="failed to connect to peer: bc52a834-10eb-32b6-7563-6e51c2b4939d" 2024-06-11T14:41:49.884Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to appendEntries to: peer="{Voter server-bc52a834-10eb-32b6-7563-6e51c2b4939d bc52a834-10eb-32b6-7563-6e51c2b4939d}" error="failed to connect to peer: bc52a834-10eb-32b6-7563-6e51c2b4939d" 2024-06-11T14:41:49.885Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to appendEntries to: peer="{Voter server-d37df15c-2879-fa0f-efce-0f7a11a0d197 d37df15c-2879-fa0f-efce-0f7a11a0d197}" error="failed to connect to peer: d37df15c-2879-fa0f-efce-0f7a11a0d197" 2024-06-11T14:41:49.893Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to heartbeat to: peer=d37df15c-2879-fa0f-efce-0f7a11a0d197 backoff time=10ms error="failed to connect to peer: d37df15c-2879-fa0f-efce-0f7a11a0d197" 2024-06-11T14:41:49.897Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to heartbeat to: peer=bc52a834-10eb-32b6-7563-6e51c2b4939d backoff time=10ms error="failed to connect to peer: bc52a834-10eb-32b6-7563-6e51c2b4939d" 2024-06-11T14:41:49.900Z [WARN] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to contact: server-id=server-bc52a834-10eb-32b6-7563-6e51c2b4939d time=50.005116ms 2024-06-11T14:41:49.900Z [WARN] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to contact: server-id=server-d37df15c-2879-fa0f-efce-0f7a11a0d197 time=50.44902ms 2024-06-11T14:41:49.900Z [WARN] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to contact quorum of nodes, stepping down 2024-06-11T14:41:49.900Z [INFO] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: entering follower state: follower="Node at bfbd3367-a9df-8fe4-3e94-db4d9db85ecf [Follower]" leader-address= leader-id= 2024-06-11T14:41:49.905Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to appendEntries to: peer="{Voter server-bc52a834-10eb-32b6-7563-6e51c2b4939d bc52a834-10eb-32b6-7563-6e51c2b4939d}" error="failed to connect to peer: bc52a834-10eb-32b6-7563-6e51c2b4939d" 2024-06-11T14:41:49.905Z [ERROR] server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf: failed to appendEntries to: peer="{Voter server-d37df15c-2879-fa0f-efce-0f7a11a0d197 d37df15c-2879-fa0f-efce-0f7a11a0d197}" error="failed to connect to peer: d37df15c-2879-fa0f-efce-0f7a11a0d197" 2024-06-11T14:41:49.919Z [WARN] server-d37df15c-2879-fa0f-efce-0f7a11a0d197: heartbeat timeout reached, starting election: last-leader-addr=bfbd3367-a9df-8fe4-3e94-db4d9db85ecf last-leader-id=server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf 2024-06-11T14:41:49.919Z [INFO] server-d37df15c-2879-fa0f-efce-0f7a11a0d197: entering candidate state: node="Node at d37df15c-2879-fa0f-efce-0f7a11a0d197 [Candidate]" term=3 2024-06-11T14:41:49.919Z [WARN] server-bc52a834-10eb-32b6-7563-6e51c2b4939d: rejecting vote request since we have a leader: from=d37df15c-2879-fa0f-efce-0f7a11a0d197 leader=bfbd3367-a9df-8fe4-3e94-db4d9db85ecf leader-id=server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf 2024-06-11T14:41:49.919Z [ERROR] server-d37df15c-2879-fa0f-efce-0f7a11a0d197: failed to make requestVote RPC: target="{Voter server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf bfbd3367-a9df-8fe4-3e94-db4d9db85ecf}" error="failed to connect to peer: bfbd3367-a9df-8fe4-3e94-db4d9db85ecf" term=3 2024-06-11T14:41:49.990Z [WARN] server-bc52a834-10eb-32b6-7563-6e51c2b4939d: heartbeat timeout reached, starting election: last-leader-addr=bfbd3367-a9df-8fe4-3e94-db4d9db85ecf last-leader-id=server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf 2024-06-11T14:41:49.990Z [INFO] server-bc52a834-10eb-32b6-7563-6e51c2b4939d: entering candidate state: node="Node at bc52a834-10eb-32b6-7563-6e51c2b4939d [Candidate]" term=3 2024-06-11T14:41:49.990Z [INFO] server-d37df15c-2879-fa0f-efce-0f7a11a0d197: duplicate requestVote for same term: term=3 2024-06-11T14:41:49.990Z [ERROR] server-bc52a834-10eb-32b6-7563-6e51c2b4939d: failed to make requestVote RPC: target="{Voter server-bfbd3367-a9df-8fe4-3e94-db4d9db85ecf bfbd3367-a9df-8fe4-3e94-db4d9db85ecf}" error="failed to connect to peer: bfbd3367-a9df-8fe4-3e94-db4d9db85ecf" term=3 2024-06-11T14:41:49.991Z [INFO] server-bc52a834-10eb-32b6-7563-6e51c2b4939d: entering follower state: follower="Node at bc52a834-10eb-32b6-7563-6e51c2b4939d [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.52s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2024-06-11T14:41:49.993Z [INFO] 127.0.0.1:46579 :: starting node: addr=127.0.0.1:46579 2024-06-11T14:41:49.993Z [INFO] 127.0.0.1:46579 :: initial configuration: index=0 servers=[] 2024-06-11T14:41:49.993Z [INFO] 127.0.0.1:46579 :: entering follower state: follower="Node at 127.0.0.1:46579 [Follower]" leader-address= leader-id= 2024-06-11T14:41:50.070Z [WARN] 127.0.0.1:46579 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2024-06-11T14:41:50.144Z [INFO] 127.0.0.1:37993 :: starting node: addr=127.0.0.1:37993 2024-06-11T14:41:50.144Z [INFO] 127.0.0.1:37993 :: initial configuration: index=0 servers=[] 2024-06-11T14:41:50.145Z [INFO] 127.0.0.1:37993 :: entering follower state: follower="Node at 127.0.0.1:37993 [Follower]" leader-address= leader-id= 2024-06-11T14:41:50.239Z [WARN] 127.0.0.1:37993 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2024-06-11T14:41:50.296Z [INFO] 127.0.0.1:46417 :: starting node: addr=127.0.0.1:46417 2024-06-11T14:41:50.296Z [INFO] 127.0.0.1:46417 :: initial configuration: index=0 servers=[] 2024-06-11T14:41:50.296Z [INFO] 127.0.0.1:46417 :: entering follower state: follower="Node at 127.0.0.1:46417 [Follower]" leader-address= leader-id= 2024-06-11T14:41:50.354Z [WARN] 127.0.0.1:46417 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.45s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2024-06-11T14:41:50.448Z [INFO] 127.0.0.1:38527 :: starting node: addr=127.0.0.1:38527 2024-06-11T14:41:50.448Z [INFO] 127.0.0.1:38527 :: initial configuration: index=0 servers=[] 2024-06-11T14:41:50.448Z [INFO] 127.0.0.1:38527 :: entering follower state: follower="Node at 127.0.0.1:38527 [Follower]" leader-address= leader-id= 2024-06-11T14:41:50.448Z [WARN] 127.0.0.1:38527 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS ok github.com/hashicorp/raft 117.206s make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/21988 and its subdirectories I: Current time: Tue Jun 11 02:42:02 -12 2024 I: pbuilder-time-stamp: 1718116922 Tue Jun 11 14:42:10 UTC 2024 I: 1st build successful. Starting 2nd build on remote node virt64c-armhf-rb.debian.net. Tue Jun 11 14:42:10 UTC 2024 I: Preparing to do remote build '2' on virt64c-armhf-rb.debian.net. Tue Jun 11 14:47:50 UTC 2024 I: Deleting $TMPDIR on virt64c-armhf-rb.debian.net.