Sat Jun 1 18:39:26 UTC 2024 I: starting to build golang-github-hashicorp-raft/unstable/amd64 on jenkins on '2024-06-01 18:39' Sat Jun 1 18:39:26 UTC 2024 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/amd64_37/14321/console.log Sat Jun 1 18:39:26 UTC 2024 I: Downloading source for unstable/golang-github-hashicorp-raft=1.5.0-1 --2024-06-01 18:39:26-- http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc Connecting to 46.16.76.132:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2480 (2.4K) [text/prs.lines.tag] Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ 0K .. 100% 374M=0s 2024-06-01 18:39:26 (374 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480] Sat Jun 1 18:39:26 UTC 2024 I: golang-github-hashicorp-raft_1.5.0-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.5.0-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.6.2 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev Package-List: golang-github-hashicorp-raft-dev deb golang optional arch=all Checksums-Sha1: cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Checksums-Sha256: f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Files: 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg +DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z 14FQMVvbK31veGXJjA== =okTI -----END PGP SIGNATURE----- Sat Jun 1 18:39:26 UTC 2024 I: Checking whether the package is not for us Sat Jun 1 18:39:26 UTC 2024 I: Starting 1st build on remote node infom01-amd64.debian.net. Sat Jun 1 18:39:26 UTC 2024 I: Preparing to do remote build '1' on infom01-amd64.debian.net. Sat Jun 1 18:43:25 UTC 2024 I: Deleting $TMPDIR on infom01-amd64.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Sat Jun 1 06:39:28 -12 2024 I: pbuilder-time-stamp: 1717267168 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source gpgv: Signature made Thu Jul 27 15:15:46 2023 gpgv: using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29 gpgv: issuer "dswarbrick@debian.org" gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/3061969/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='amd64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 ' DISTRIBUTION='unstable' HOME='/root' HOST_ARCH='amd64' IFS=' ' INVOCATION_ID='bce9907d2b5c45628ca71bfc5ac1ff8a' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='3061969' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/pbuilderrc_D68v --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='109' SUDO_UID='104' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' I: uname -a Linux infom01-amd64 6.1.0-21-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 May 28 07:42 /bin -> usr/bin I: user script /srv/workspace/pbuilder/3061969/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: amd64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19719 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.22-go{a} golang-1.22-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 102 newly installed, 0 to remove and 0 not upgraded. Need to get 90.2 MB of archives. After unpacking 627 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian unstable/main amd64 sensible-utils all 0.0.22 [22.4 kB] Get: 2 http://deb.debian.org/debian unstable/main amd64 libmagic-mgc amd64 1:5.45-3 [314 kB] Get: 3 http://deb.debian.org/debian unstable/main amd64 libmagic1t64 amd64 1:5.45-3 [105 kB] Get: 4 http://deb.debian.org/debian unstable/main amd64 file amd64 1:5.45-3 [42.9 kB] Get: 5 http://deb.debian.org/debian unstable/main amd64 gettext-base amd64 0.21-14+b1 [161 kB] Get: 6 http://deb.debian.org/debian unstable/main amd64 libuchardet0 amd64 0.0.8-1+b1 [68.8 kB] Get: 7 http://deb.debian.org/debian unstable/main amd64 groff-base amd64 1.23.0-4 [1180 kB] Get: 8 http://deb.debian.org/debian unstable/main amd64 bsdextrautils amd64 2.40.1-4 [95.7 kB] Get: 9 http://deb.debian.org/debian unstable/main amd64 libpipeline1 amd64 1.5.7-2 [38.0 kB] Get: 10 http://deb.debian.org/debian unstable/main amd64 man-db amd64 2.12.1-1 [1411 kB] Get: 11 http://deb.debian.org/debian unstable/main amd64 m4 amd64 1.4.19-4 [287 kB] Get: 12 http://deb.debian.org/debian unstable/main amd64 autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian unstable/main amd64 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian unstable/main amd64 automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian unstable/main amd64 autopoint all 0.21-14 [496 kB] Get: 16 http://deb.debian.org/debian unstable/main amd64 libdebhelper-perl all 13.15.3 [88.0 kB] Get: 17 http://deb.debian.org/debian unstable/main amd64 libtool all 2.4.7-7 [517 kB] Get: 18 http://deb.debian.org/debian unstable/main amd64 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian unstable/main amd64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian unstable/main amd64 libfile-stripnondeterminism-perl all 1.14.0-1 [19.5 kB] Get: 21 http://deb.debian.org/debian unstable/main amd64 dh-strip-nondeterminism all 1.14.0-1 [8448 B] Get: 22 http://deb.debian.org/debian unstable/main amd64 libelf1t64 amd64 0.191-1+b1 [189 kB] Get: 23 http://deb.debian.org/debian unstable/main amd64 dwz amd64 0.15-1+b1 [110 kB] Get: 24 http://deb.debian.org/debian unstable/main amd64 libicu72 amd64 72.1-4+b1 [9395 kB] Get: 25 http://deb.debian.org/debian unstable/main amd64 libxml2 amd64 2.12.7+dfsg-2 [670 kB] Get: 26 http://deb.debian.org/debian unstable/main amd64 gettext amd64 0.21-14+b1 [1301 kB] Get: 27 http://deb.debian.org/debian unstable/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 28 http://deb.debian.org/debian unstable/main amd64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 29 http://deb.debian.org/debian unstable/main amd64 debhelper all 13.15.3 [901 kB] Get: 30 http://deb.debian.org/debian unstable/main amd64 dh-golang all 1.62 [24.7 kB] Get: 31 http://deb.debian.org/debian unstable/main amd64 golang-1.22-src all 1.22.3-1 [18.8 MB] Get: 32 http://deb.debian.org/debian unstable/main amd64 golang-1.22-go amd64 1.22.3-1 [24.7 MB] Get: 33 http://deb.debian.org/debian unstable/main amd64 golang-src all 2:1.22~3 [5020 B] Get: 34 http://deb.debian.org/debian unstable/main amd64 golang-go amd64 2:1.22~3 [44.2 kB] Get: 35 http://deb.debian.org/debian unstable/main amd64 golang-any amd64 2:1.22~3 [5104 B] Get: 36 http://deb.debian.org/debian unstable/main amd64 golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 37 http://deb.debian.org/debian unstable/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 38 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 39 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-sys-dev all 0.20.0-1 [430 kB] Get: 40 http://deb.debian.org/debian unstable/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 41 http://deb.debian.org/debian unstable/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 42 http://deb.debian.org/debian unstable/main amd64 golang-github-fatih-color-dev all 1.16.0-1 [13.8 kB] Get: 43 http://deb.debian.org/debian unstable/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 44 http://deb.debian.org/debian unstable/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 45 http://deb.debian.org/debian unstable/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 46 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 47 http://deb.debian.org/debian unstable/main amd64 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB] Get: 48 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 49 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 50 http://deb.debian.org/debian unstable/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 51 http://deb.debian.org/debian unstable/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 52 http://deb.debian.org/debian unstable/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 53 http://deb.debian.org/debian unstable/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 54 http://deb.debian.org/debian unstable/main amd64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 55 http://deb.debian.org/debian unstable/main amd64 golang-google-protobuf-dev all 1.33.0-1 [625 kB] Get: 56 http://deb.debian.org/debian unstable/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 57 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 58 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 60 http://deb.debian.org/debian unstable/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 61 http://deb.debian.org/debian unstable/main amd64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 62 http://deb.debian.org/debian unstable/main amd64 golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 63 http://deb.debian.org/debian unstable/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian unstable/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 65 http://deb.debian.org/debian unstable/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 66 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 67 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 68 http://deb.debian.org/debian unstable/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 69 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-term-dev all 0.20.0-1 [14.8 kB] Get: 70 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-text-dev all 0.15.0-1 [4006 kB] Get: 71 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-crypto-dev all 1:0.23.0-1 [1689 kB] Get: 72 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-net-dev all 1:0.25.0+dfsg-1 [895 kB] Get: 73 http://deb.debian.org/debian unstable/main amd64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 74 http://deb.debian.org/debian unstable/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 75 http://deb.debian.org/debian unstable/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 76 http://deb.debian.org/debian unstable/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 77 http://deb.debian.org/debian unstable/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 78 http://deb.debian.org/debian unstable/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 79 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB] Get: 80 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 81 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-common-dev all 0.53.0-1 [114 kB] Get: 82 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 83 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB] Get: 84 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-client-golang-dev all 1.19.0-1 [179 kB] Get: 85 http://deb.debian.org/debian unstable/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 86 http://deb.debian.org/debian unstable/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 87 http://deb.debian.org/debian unstable/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 88 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 89 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 90 http://deb.debian.org/debian unstable/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-6 [19.5 kB] Get: 91 http://deb.debian.org/debian unstable/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-6 [56.9 kB] Get: 92 http://deb.debian.org/debian unstable/main amd64 libssl-dev amd64 3.2.1-3 [2699 kB] Get: 93 http://deb.debian.org/debian unstable/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-6 [231 kB] Get: 94 http://deb.debian.org/debian unstable/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 95 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-mod-dev all 0.17.0-1 [103 kB] Get: 96 http://deb.debian.org/debian unstable/main amd64 golang-github-yuin-goldmark-dev all 1.7.1-1 [113 kB] Get: 97 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-tools-dev all 1:0.20.0+ds-1 [1571 kB] Get: 98 http://deb.debian.org/debian unstable/main amd64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 99 http://deb.debian.org/debian unstable/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 100 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 101 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 102 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 90.2 MB in 2s (43.0 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19719 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.22_all.deb ... Unpacking sensible-utils (0.0.22) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3_amd64.deb ... Unpacking libmagic-mgc (1:5.45-3) ... Selecting previously unselected package libmagic1t64:amd64. Preparing to unpack .../002-libmagic1t64_1%3a5.45-3_amd64.deb ... Unpacking libmagic1t64:amd64 (1:5.45-3) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.45-3_amd64.deb ... Unpacking file (1:5.45-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.21-14+b1_amd64.deb ... Unpacking gettext-base (0.21-14+b1) ... Selecting previously unselected package libuchardet0:amd64. Preparing to unpack .../005-libuchardet0_0.0.8-1+b1_amd64.deb ... Unpacking libuchardet0:amd64 (0.0.8-1+b1) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-4_amd64.deb ... Unpacking groff-base (1.23.0-4) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.40.1-4_amd64.deb ... Unpacking bsdextrautils (2.40.1-4) ... Selecting previously unselected package libpipeline1:amd64. Preparing to unpack .../008-libpipeline1_1.5.7-2_amd64.deb ... Unpacking libpipeline1:amd64 (1.5.7-2) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.12.1-1_amd64.deb ... Unpacking man-db (2.12.1-1) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-4_amd64.deb ... Unpacking m4 (1.4.19-4) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.21-14_all.deb ... Unpacking autopoint (0.21-14) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../015-libdebhelper-perl_13.15.3_all.deb ... Unpacking libdebhelper-perl (13.15.3) ... Selecting previously unselected package libtool. Preparing to unpack .../016-libtool_2.4.7-7_all.deb ... Unpacking libtool (2.4.7-7) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../017-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../020-dh-strip-nondeterminism_1.14.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.14.0-1) ... Selecting previously unselected package libelf1t64:amd64. Preparing to unpack .../021-libelf1t64_0.191-1+b1_amd64.deb ... Unpacking libelf1t64:amd64 (0.191-1+b1) ... Selecting previously unselected package dwz. Preparing to unpack .../022-dwz_0.15-1+b1_amd64.deb ... Unpacking dwz (0.15-1+b1) ... Selecting previously unselected package libicu72:amd64. Preparing to unpack .../023-libicu72_72.1-4+b1_amd64.deb ... Unpacking libicu72:amd64 (72.1-4+b1) ... Selecting previously unselected package libxml2:amd64. Preparing to unpack .../024-libxml2_2.12.7+dfsg-2_amd64.deb ... Unpacking libxml2:amd64 (2.12.7+dfsg-2) ... Selecting previously unselected package gettext. Preparing to unpack .../025-gettext_0.21-14+b1_amd64.deb ... Unpacking gettext (0.21-14+b1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../028-debhelper_13.15.3_all.deb ... Unpacking debhelper (13.15.3) ... Selecting previously unselected package dh-golang. Preparing to unpack .../029-dh-golang_1.62_all.deb ... Unpacking dh-golang (1.62) ... Selecting previously unselected package golang-1.22-src. Preparing to unpack .../030-golang-1.22-src_1.22.3-1_all.deb ... Unpacking golang-1.22-src (1.22.3-1) ... Selecting previously unselected package golang-1.22-go. Preparing to unpack .../031-golang-1.22-go_1.22.3-1_amd64.deb ... Unpacking golang-1.22-go (1.22.3-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../032-golang-src_2%3a1.22~3_all.deb ... Unpacking golang-src (2:1.22~3) ... Selecting previously unselected package golang-go:amd64. Preparing to unpack .../033-golang-go_2%3a1.22~3_amd64.deb ... Unpacking golang-go:amd64 (2:1.22~3) ... Selecting previously unselected package golang-any:amd64. Preparing to unpack .../034-golang-any_2%3a1.22~3_amd64.deb ... Unpacking golang-any:amd64 (2:1.22~3) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../038-golang-golang-x-sys-dev_0.20.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.20.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../041-golang-github-fatih-color-dev_1.16.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.16.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../054-golang-google-protobuf-dev_1.33.0-1_all.deb ... Unpacking golang-google-protobuf-dev (1.33.0-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../065-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../066-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../067-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../068-golang-golang-x-term-dev_0.20.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.20.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../069-golang-golang-x-text-dev_0.15.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.15.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../070-golang-golang-x-crypto-dev_1%3a0.23.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.23.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../071-golang-golang-x-net-dev_1%3a0.25.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.25.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../072-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../073-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../074-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../075-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../076-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../078-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../079-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../080-golang-github-prometheus-common-dev_0.53.0-1_all.deb ... Unpacking golang-github-prometheus-common-dev (0.53.0-1) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../081-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../082-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../083-golang-github-prometheus-client-golang-dev_1.19.0-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.19.0-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../084-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../085-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../086-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../087-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../088-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:amd64. Preparing to unpack .../089-libsasl2-modules-db_2.1.28+dfsg1-6_amd64.deb ... Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-6) ... Selecting previously unselected package libsasl2-2:amd64. Preparing to unpack .../090-libsasl2-2_2.1.28+dfsg1-6_amd64.deb ... Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-6) ... Selecting previously unselected package libssl-dev:amd64. Preparing to unpack .../091-libssl-dev_3.2.1-3_amd64.deb ... Unpacking libssl-dev:amd64 (3.2.1-3) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../092-libsasl2-dev_2.1.28+dfsg1-6_amd64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-6) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../093-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../094-golang-golang-x-mod-dev_0.17.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.17.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../095-golang-github-yuin-goldmark-dev_1.7.1-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.1-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../096-golang-golang-x-tools-dev_1%3a0.20.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.20.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../097-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../098-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../099-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../100-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../101-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:amd64 (1.5.7-2) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up libicu72:amd64 (72.1-4+b1) ... Setting up bsdextrautils (2.40.1-4) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.45-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.15.3) ... Setting up golang-golang-x-sys-dev (0.20.0-1) ... Setting up libmagic1t64:amd64 (1:5.45-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-14+b1) ... Setting up m4 (1.4.19-4) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-golang-x-term-dev (0.20.0-1) ... Setting up file (1:5.45-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up libelf1t64:amd64 (0.191-1+b1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-6) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.33.0-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.9.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.1-1) ... Setting up golang-1.22-src (1.22.3-1) ... Setting up libssl-dev:amd64 (3.2.1-3) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.17.0-1) ... Setting up autopoint (0.21-14) ... Setting up libsasl2-2:amd64 (2.1.28+dfsg1-6) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b1) ... Setting up sensible-utils (0.0.22) ... Setting up golang-golang-x-text-dev (0.15.0-1) ... Setting up libuchardet0:amd64 (0.0.8-1+b1) ... Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.23.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.15.0-1) ... Setting up golang-src (2:1.22~3) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:amd64 (2.12.7+dfsg-2) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.0-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.21-14+b1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-7) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-1.22-go (1.22.3-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-6) ... Setting up golang-golang-x-net-dev (1:0.25.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.14.0-1) ... Setting up groff-base (1.23.0-4) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.16.0-1) ... Setting up golang-go:amd64 (2:1.22~3) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.20.0+ds-1) ... Setting up man-db (2.12.1-1) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-any:amd64 (2:1.22~3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.15.3) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.62) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.53.0-1) ... Setting up golang-github-prometheus-client-golang-dev (1.19.0-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.38-11) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture amd64 debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-x86_64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/goarch internal/coverage/rtcov internal/godebugs internal/unsafeheader internal/goos unicode internal/race internal/goexperiment internal/abi internal/cpu unicode/utf8 runtime/internal/math internal/chacha8rand sync/atomic runtime/internal/atomic runtime/internal/syscall runtime/internal/sys container/list internal/itoa math/bits crypto/internal/alias crypto/subtle crypto/internal/boring/sig cmp encoding unicode/utf16 log/internal slices internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/alias internal/bytealg math runtime internal/reflectlite sync internal/testlog internal/singleflight internal/bisect runtime/cgo errors sort io internal/godebug github.com/hashicorp/golang-lru/simplelru strconv internal/safefilepath internal/oserror vendor/golang.org/x/net/dns/dnsmessage path crypto/internal/nistec/fiat syscall math/rand internal/intern hash bytes strings crypto/internal/randutil internal/saferio hash/crc32 hash/crc64 crypto reflect net/netip crypto/rc4 vendor/golang.org/x/text/transform bufio net/http/internal/ascii github.com/hashicorp/go-immutable-radix regexp/syntax html regexp internal/syscall/execenv time internal/syscall/unix context internal/poll io/fs encoding/binary internal/fmtsort embed os encoding/base64 crypto/cipher crypto/internal/edwards25519/field crypto/md5 vendor/golang.org/x/crypto/internal/poly1305 crypto/internal/nistec golang.org/x/sys/unix encoding/pem crypto/internal/edwards25519 crypto/internal/boring crypto/des vendor/golang.org/x/crypto/chacha20 crypto/aes crypto/sha1 crypto/sha256 crypto/sha512 crypto/hmac vendor/golang.org/x/crypto/hkdf fmt path/filepath vendor/golang.org/x/sys/cpu internal/sysinfo io/ioutil os/signal net crypto/ecdh vendor/golang.org/x/crypto/chacha20poly1305 log encoding/hex math/big encoding/json compress/flate vendor/golang.org/x/net/http2/hpack net/url vendor/golang.org/x/text/unicode/norm mime/quotedprintable mime net/http/internal encoding/gob go/token vendor/golang.org/x/text/unicode/bidi text/template/parse flag runtime/debug runtime/trace compress/gzip vendor/golang.org/x/text/secure/bidirule testing vendor/golang.org/x/net/idna github.com/mattn/go-isatty text/template github.com/mattn/go-colorable github.com/fatih/color crypto/rand crypto/internal/boring/bbig crypto/elliptic crypto/dsa crypto/internal/bigmod encoding/asn1 github.com/hashicorp/go-hclog crypto/ed25519 crypto/rsa vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix html/template crypto/ecdsa vendor/golang.org/x/net/http/httpproxy crypto/x509 net/textproto vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http github.com/armon/go-metrics net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-x86_64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench ? github.com/hashicorp/raft/bench [no test files] === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2024-06-01T18:41:24.503Z [INFO] creating new snapshot: path=/tmp/raft3000994605/raft2880898894/snapshots/3-10-1717267284503.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot 2024-06-01T18:41:24.509Z [INFO] creating new snapshot: path=/tmp/raft2480000965/snapshots/3-10-1717267284509.tmp 2024-06-01T18:41:24.512Z [WARN] found temporary snapshot: name=3-10-1717267284509.tmp --- PASS: TestFileSS_CreateSnapshot (0.01s) === RUN TestFileSS_CancelSnapshot 2024-06-01T18:41:24.520Z [INFO] creating new snapshot: path=/tmp/raft2251222730/snapshots/3-10-1717267284520.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention 2024-06-01T18:41:24.524Z [INFO] creating new snapshot: path=/tmp/raft2537024007/snapshots/3-10-1717267284524.tmp 2024-06-01T18:41:24.533Z [INFO] creating new snapshot: path=/tmp/raft2537024007/snapshots/3-11-1717267284533.tmp 2024-06-01T18:41:24.543Z [INFO] creating new snapshot: path=/tmp/raft2537024007/snapshots/3-12-1717267284543.tmp 2024-06-01T18:41:24.552Z [INFO] reaping snapshot: path=/tmp/raft2537024007/snapshots/3-10-1717267284524 2024-06-01T18:41:24.552Z [INFO] creating new snapshot: path=/tmp/raft2537024007/snapshots/3-13-1717267284552.tmp 2024-06-01T18:41:24.564Z [INFO] reaping snapshot: path=/tmp/raft2537024007/snapshots/3-11-1717267284533 2024-06-01T18:41:24.564Z [INFO] creating new snapshot: path=/tmp/raft2537024007/snapshots/3-14-1717267284564.tmp 2024-06-01T18:41:24.574Z [INFO] reaping snapshot: path=/tmp/raft2537024007/snapshots/3-12-1717267284543 --- PASS: TestFileSS_Retention (0.05s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2024-06-01T18:41:24.575Z [INFO] creating new snapshot: path=/tmp/raft74151803/snapshots/5-130350-1717267284575.tmp 2024-06-01T18:41:24.584Z [INFO] creating new snapshot: path=/tmp/raft74151803/snapshots/36-204917-1717267284584.tmp --- PASS: TestFileSS_Ordering (0.02s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.01s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 2024-06-01T18:41:24.685Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 2024-06-01T18:41:24.738Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.15s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) 2024-06-01T18:41:24.789Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2024-06-01T18:41:24.794Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:24.799Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:24.809Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:24.830Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:24.871Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:24.952Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:25.112Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:25.433Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:26.074Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:27.076Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2024-06-01T18:41:28.078Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2024-06-01T18:41:28.796Z [INFO] server-8f826b7d-11c1-db1a-99e8-14aba6c34a44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f826b7d-11c1-db1a-99e8-14aba6c34a44 Address:8f826b7d-11c1-db1a-99e8-14aba6c34a44}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2024-06-01T18:41:28.796Z [INFO] server-ef136fe7-f9f5-8d20-cd50-9c07f35df002: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef136fe7-f9f5-8d20-cd50-9c07f35df002 Address:ef136fe7-f9f5-8d20-cd50-9c07f35df002}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2024-06-01T18:41:28.797Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: initial configuration: index=0 servers=[] 2024-06-01T18:41:28.797Z [INFO] server-b8a9a789-3324-b90b-293b-b829976d53d9: initial configuration: index=0 servers=[] 2024-06-01T18:41:28.797Z [INFO] server-4eefb283-a2e9-5259-d980-e72d85cb5b1e: initial configuration: index=0 servers=[] 2024-06-01T18:41:28.797Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: entering follower state: follower="Node at fbc2bc52-b739-5703-3521-5d373477e181 [Follower]" leader-address= leader-id= 2024-06-01T18:41:28.797Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:28.797Z [INFO] server-b8a9a789-3324-b90b-293b-b829976d53d9: entering follower state: follower="Node at b8a9a789-3324-b90b-293b-b829976d53d9 [Follower]" leader-address= leader-id= 2024-06-01T18:41:28.797Z [INFO] server-4eefb283-a2e9-5259-d980-e72d85cb5b1e: entering follower state: follower="Node at 4eefb283-a2e9-5259-d980-e72d85cb5b1e [Follower]" leader-address= leader-id= 2024-06-01T18:41:28.867Z [WARN] server-fbc2bc52-b739-5703-3521-5d373477e181: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:28.867Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: entering candidate state: node="Node at fbc2bc52-b739-5703-3521-5d373477e181 [Candidate]" term=2 2024-06-01T18:41:28.867Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: election won: term=2 tally=2 2024-06-01T18:41:28.867Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: entering leader state: leader="Node at fbc2bc52-b739-5703-3521-5d373477e181 [Leader]" 2024-06-01T18:41:28.867Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: added peer, starting replication: peer=server-b8a9a789-3324-b90b-293b-b829976d53d9 2024-06-01T18:41:28.867Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: added peer, starting replication: peer=server-4eefb283-a2e9-5259-d980-e72d85cb5b1e 2024-06-01T18:41:28.867Z [WARN] server-4eefb283-a2e9-5259-d980-e72d85cb5b1e: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2024-06-01T18:41:28.867Z [WARN] server-fbc2bc52-b739-5703-3521-5d373477e181: appendEntries rejected, sending older logs: peer="{Voter server-4eefb283-a2e9-5259-d980-e72d85cb5b1e 4eefb283-a2e9-5259-d980-e72d85cb5b1e}" next=1 2024-06-01T18:41:28.868Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: pipelining replication: peer="{Voter server-4eefb283-a2e9-5259-d980-e72d85cb5b1e 4eefb283-a2e9-5259-d980-e72d85cb5b1e}" 2024-06-01T18:41:28.868Z [WARN] server-b8a9a789-3324-b90b-293b-b829976d53d9: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2024-06-01T18:41:28.868Z [WARN] server-fbc2bc52-b739-5703-3521-5d373477e181: appendEntries rejected, sending older logs: peer="{Voter server-b8a9a789-3324-b90b-293b-b829976d53d9 b8a9a789-3324-b90b-293b-b829976d53d9}" next=1 2024-06-01T18:41:28.868Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: pipelining replication: peer="{Voter server-b8a9a789-3324-b90b-293b-b829976d53d9 b8a9a789-3324-b90b-293b-b829976d53d9}" 2024-06-01T18:41:28.973Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:28.868170845 +0000 UTC m=+4.366360297 (2 nodes), 70.913118ms from start of poll, 71.095187ms from cluster start. Timeout at 2024-06-01 18:41:28.973875899 +0000 UTC m=+4.472065341, 105.705044ms after stability 2024-06-01T18:41:28.974Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:29.079Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:28.974013545 +0000 UTC m=+4.472202998 (1 nodes), 952ns from start of poll, 176.937888ms from cluster start. Timeout at 2024-06-01 18:41:29.079836721 +0000 UTC m=+4.578026173, 105.823175ms after stability 2024-06-01T18:41:29.091Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: aborting pipeline replication: peer="{Voter server-4eefb283-a2e9-5259-d980-e72d85cb5b1e 4eefb283-a2e9-5259-d980-e72d85cb5b1e}" 2024-06-01T18:41:29.091Z [INFO] server-fbc2bc52-b739-5703-3521-5d373477e181: aborting pipeline replication: peer="{Voter server-b8a9a789-3324-b90b-293b-b829976d53d9 b8a9a789-3324-b90b-293b-b829976d53d9}" --- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2024-06-01T18:41:29.092Z [INFO] server-595959db-1a44-9b00-3379-105dd18ef719: initial configuration: index=0 servers=[] 2024-06-01T18:41:29.092Z [INFO] server-0430a0b9-3198-ac79-cc69-e05c35603864: initial configuration: index=0 servers=[] 2024-06-01T18:41:29.092Z [INFO] server-595959db-1a44-9b00-3379-105dd18ef719: entering follower state: follower="Node at 595959db-1a44-9b00-3379-105dd18ef719 [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.092Z [INFO] server-0430a0b9-3198-ac79-cc69-e05c35603864: entering follower state: follower="Node at 0430a0b9-3198-ac79-cc69-e05c35603864 [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2024-06-01T18:41:29.093Z [INFO] server-54523468-6b09-ed42-7a05-b0620005d128: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2024-06-01T18:41:29.094Z [INFO] server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a Address:77dd4eab-b7cf-5447-0275-d500bd8c2b5a} {Suffrage:Voter ID:server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 Address:02afdb9f-bde8-e3ad-da8e-f5659743c2e2} {Suffrage:Voter ID:server-8ee06f3c-6e25-e81d-71b9-49df02779f5a Address:8ee06f3c-6e25-e81d-71b9-49df02779f5a}]" 2024-06-01T18:41:29.094Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a Address:77dd4eab-b7cf-5447-0275-d500bd8c2b5a} {Suffrage:Voter ID:server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 Address:02afdb9f-bde8-e3ad-da8e-f5659743c2e2} {Suffrage:Voter ID:server-8ee06f3c-6e25-e81d-71b9-49df02779f5a Address:8ee06f3c-6e25-e81d-71b9-49df02779f5a}]" 2024-06-01T18:41:29.094Z [INFO] server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a: entering follower state: follower="Node at 77dd4eab-b7cf-5447-0275-d500bd8c2b5a [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.094Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a Address:77dd4eab-b7cf-5447-0275-d500bd8c2b5a} {Suffrage:Voter ID:server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 Address:02afdb9f-bde8-e3ad-da8e-f5659743c2e2} {Suffrage:Voter ID:server-8ee06f3c-6e25-e81d-71b9-49df02779f5a Address:8ee06f3c-6e25-e81d-71b9-49df02779f5a}]" 2024-06-01T18:41:29.094Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:29.094Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: entering follower state: follower="Node at 02afdb9f-bde8-e3ad-da8e-f5659743c2e2 [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.094Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: entering follower state: follower="Node at 8ee06f3c-6e25-e81d-71b9-49df02779f5a [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.157Z [WARN] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:29.157Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: entering candidate state: node="Node at 8ee06f3c-6e25-e81d-71b9-49df02779f5a [Candidate]" term=2 2024-06-01T18:41:29.157Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: election won: term=2 tally=2 2024-06-01T18:41:29.157Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: entering leader state: leader="Node at 8ee06f3c-6e25-e81d-71b9-49df02779f5a [Leader]" 2024-06-01T18:41:29.157Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: added peer, starting replication: peer=server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a 2024-06-01T18:41:29.157Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: added peer, starting replication: peer=server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 2024-06-01T18:41:29.157Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: pipelining replication: peer="{Voter server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 02afdb9f-bde8-e3ad-da8e-f5659743c2e2}" 2024-06-01T18:41:29.158Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: pipelining replication: peer="{Voter server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a 77dd4eab-b7cf-5447-0275-d500bd8c2b5a}" 2024-06-01T18:41:29.263Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:29.157961594 +0000 UTC m=+4.656151046 (1 nodes), 63.699079ms from start of poll, 63.967469ms from cluster start. Timeout at 2024-06-01 18:41:29.263481264 +0000 UTC m=+4.761670716, 105.51967ms after stability 2024-06-01T18:41:29.263Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: aborting pipeline replication: peer="{Voter server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 02afdb9f-bde8-e3ad-da8e-f5659743c2e2}" 2024-06-01T18:41:29.263Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: aborting pipeline replication: peer="{Voter server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a 77dd4eab-b7cf-5447-0275-d500bd8c2b5a}" 2024-06-01T18:41:29.263Z [INFO] creating new snapshot: path=/tmp/raft190693018/snapshots/2-2-1717267289263.tmp 2024-06-01T18:41:29.264Z [INFO] server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a: starting restore from snapshot: id=2-2-1717267289263 last-index=2 last-term=2 size-in-bytes=1 2024-06-01T18:41:29.264Z [INFO] server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a: snapshot restore progress: id=2-2-1717267289263 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2024-06-01T18:41:29.264Z [INFO] server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a: restored from snapshot: id=2-2-1717267289263 last-index=2 last-term=2 size-in-bytes=1 2024-06-01T18:41:29.264Z [INFO] server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a Address:77dd4eab-b7cf-5447-0275-d500bd8c2b5a} {Suffrage:Voter ID:server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 Address:02afdb9f-bde8-e3ad-da8e-f5659743c2e2} {Suffrage:Voter ID:server-8ee06f3c-6e25-e81d-71b9-49df02779f5a Address:8ee06f3c-6e25-e81d-71b9-49df02779f5a}]" 2024-06-01T18:41:29.264Z [INFO] creating new snapshot: path=/tmp/raft3393215562/snapshots/2-2-1717267289264.tmp 2024-06-01T18:41:29.264Z [INFO] server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a: entering follower state: follower="Node at 77dd4eab-b7cf-5447-0275-d500bd8c2b5a [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.264Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: starting restore from snapshot: id=2-2-1717267289264 last-index=2 last-term=2 size-in-bytes=1 2024-06-01T18:41:29.265Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: snapshot restore progress: id=2-2-1717267289264 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2024-06-01T18:41:29.265Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: restored from snapshot: id=2-2-1717267289264 last-index=2 last-term=2 size-in-bytes=1 2024-06-01T18:41:29.265Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a Address:77dd4eab-b7cf-5447-0275-d500bd8c2b5a} {Suffrage:Voter ID:server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 Address:02afdb9f-bde8-e3ad-da8e-f5659743c2e2} {Suffrage:Voter ID:server-8ee06f3c-6e25-e81d-71b9-49df02779f5a Address:8ee06f3c-6e25-e81d-71b9-49df02779f5a}]" 2024-06-01T18:41:29.265Z [INFO] creating new snapshot: path=/tmp/raft1993556579/snapshots/2-2-1717267289265.tmp 2024-06-01T18:41:29.265Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: entering follower state: follower="Node at 02afdb9f-bde8-e3ad-da8e-f5659743c2e2 [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.265Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: starting restore from snapshot: id=2-2-1717267289265 last-index=2 last-term=2 size-in-bytes=1 2024-06-01T18:41:29.265Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: snapshot restore progress: id=2-2-1717267289265 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2024-06-01T18:41:29.265Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: restored from snapshot: id=2-2-1717267289265 last-index=2 last-term=2 size-in-bytes=1 2024-06-01T18:41:29.265Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a Address:77dd4eab-b7cf-5447-0275-d500bd8c2b5a} {Suffrage:Voter ID:server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2 Address:02afdb9f-bde8-e3ad-da8e-f5659743c2e2} {Suffrage:Voter ID:server-8ee06f3c-6e25-e81d-71b9-49df02779f5a Address:8ee06f3c-6e25-e81d-71b9-49df02779f5a}]" 2024-06-01T18:41:29.265Z [INFO] server-8ee06f3c-6e25-e81d-71b9-49df02779f5a: entering follower state: follower="Node at 8ee06f3c-6e25-e81d-71b9-49df02779f5a [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.320Z [WARN] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:29.320Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: entering candidate state: node="Node at 02afdb9f-bde8-e3ad-da8e-f5659743c2e2 [Candidate]" term=3 2024-06-01T18:41:29.320Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: election won: term=3 tally=2 2024-06-01T18:41:29.320Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: entering leader state: leader="Node at 02afdb9f-bde8-e3ad-da8e-f5659743c2e2 [Leader]" 2024-06-01T18:41:29.320Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: added peer, starting replication: peer=server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a 2024-06-01T18:41:29.320Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: added peer, starting replication: peer=server-8ee06f3c-6e25-e81d-71b9-49df02779f5a 2024-06-01T18:41:29.321Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: pipelining replication: peer="{Voter server-8ee06f3c-6e25-e81d-71b9-49df02779f5a 8ee06f3c-6e25-e81d-71b9-49df02779f5a}" 2024-06-01T18:41:29.321Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: pipelining replication: peer="{Voter server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a 77dd4eab-b7cf-5447-0275-d500bd8c2b5a}" 2024-06-01T18:41:29.581Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:29.687Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:29.58140303 +0000 UTC m=+5.079592492 (1 nodes), 17.793µs from start of poll, 487.408915ms from cluster start. Timeout at 2024-06-01 18:41:29.687434813 +0000 UTC m=+5.185624265, 106.031773ms after stability 2024-06-01T18:41:29.687Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: aborting pipeline replication: peer="{Voter server-8ee06f3c-6e25-e81d-71b9-49df02779f5a 8ee06f3c-6e25-e81d-71b9-49df02779f5a}" 2024-06-01T18:41:29.687Z [INFO] server-02afdb9f-bde8-e3ad-da8e-f5659743c2e2: aborting pipeline replication: peer="{Voter server-77dd4eab-b7cf-5447-0275-d500bd8c2b5a 77dd4eab-b7cf-5447-0275-d500bd8c2b5a}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2024-06-01T18:41:29.691Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3322817-7f44-eb00-8ff7-e345db3b37d3 Address:c3322817-7f44-eb00-8ff7-e345db3b37d3} {Suffrage:Voter ID:server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 Address:bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2} {Suffrage:Voter ID:server-a49c7011-69e0-effe-884f-a696b336291e Address:a49c7011-69e0-effe-884f-a696b336291e}]" 2024-06-01T18:41:29.691Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3322817-7f44-eb00-8ff7-e345db3b37d3 Address:c3322817-7f44-eb00-8ff7-e345db3b37d3} {Suffrage:Voter ID:server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 Address:bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2} {Suffrage:Voter ID:server-a49c7011-69e0-effe-884f-a696b336291e Address:a49c7011-69e0-effe-884f-a696b336291e}]" 2024-06-01T18:41:29.691Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: entering follower state: follower="Node at c3322817-7f44-eb00-8ff7-e345db3b37d3 [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.691Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: entering follower state: follower="Node at bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.691Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3322817-7f44-eb00-8ff7-e345db3b37d3 Address:c3322817-7f44-eb00-8ff7-e345db3b37d3} {Suffrage:Voter ID:server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 Address:bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2} {Suffrage:Voter ID:server-a49c7011-69e0-effe-884f-a696b336291e Address:a49c7011-69e0-effe-884f-a696b336291e}]" 2024-06-01T18:41:29.691Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:29.691Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: entering follower state: follower="Node at a49c7011-69e0-effe-884f-a696b336291e [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.770Z [WARN] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:29.770Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: entering candidate state: node="Node at bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 [Candidate]" term=2 2024-06-01T18:41:29.770Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: election won: term=2 tally=2 2024-06-01T18:41:29.770Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: entering leader state: leader="Node at bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 [Leader]" 2024-06-01T18:41:29.770Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: added peer, starting replication: peer=server-c3322817-7f44-eb00-8ff7-e345db3b37d3 2024-06-01T18:41:29.770Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: added peer, starting replication: peer=server-a49c7011-69e0-effe-884f-a696b336291e 2024-06-01T18:41:29.770Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: pipelining replication: peer="{Voter server-a49c7011-69e0-effe-884f-a696b336291e a49c7011-69e0-effe-884f-a696b336291e}" 2024-06-01T18:41:29.770Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: pipelining replication: peer="{Voter server-c3322817-7f44-eb00-8ff7-e345db3b37d3 c3322817-7f44-eb00-8ff7-e345db3b37d3}" 2024-06-01T18:41:29.875Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:29.770367326 +0000 UTC m=+5.268556778 (1 nodes), 79.137939ms from start of poll, 79.389628ms from cluster start. Timeout at 2024-06-01 18:41:29.875778042 +0000 UTC m=+5.373967494, 105.410716ms after stability 2024-06-01T18:41:29.876Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: aborting pipeline replication: peer="{Voter server-a49c7011-69e0-effe-884f-a696b336291e a49c7011-69e0-effe-884f-a696b336291e}" 2024-06-01T18:41:29.876Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: aborting pipeline replication: peer="{Voter server-c3322817-7f44-eb00-8ff7-e345db3b37d3 c3322817-7f44-eb00-8ff7-e345db3b37d3}" 2024-06-01T18:41:29.876Z [INFO] creating new snapshot: path=/tmp/raft9410879/snapshots/2-6-1717267289876.tmp 2024-06-01T18:41:29.876Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: starting restore from snapshot: id=2-6-1717267289876 last-index=6 last-term=2 size-in-bytes=25 2024-06-01T18:41:29.877Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: snapshot restore progress: id=2-6-1717267289876 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2024-06-01T18:41:29.877Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: restored from snapshot: id=2-6-1717267289876 last-index=6 last-term=2 size-in-bytes=25 2024-06-01T18:41:29.877Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3322817-7f44-eb00-8ff7-e345db3b37d3 Address:c3322817-7f44-eb00-8ff7-e345db3b37d3} {Suffrage:Voter ID:server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 Address:bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2} {Suffrage:Voter ID:server-a49c7011-69e0-effe-884f-a696b336291e Address:a49c7011-69e0-effe-884f-a696b336291e}]" 2024-06-01T18:41:29.877Z [INFO] creating new snapshot: path=/tmp/raft2349773711/snapshots/2-6-1717267289877.tmp 2024-06-01T18:41:29.877Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: entering follower state: follower="Node at c3322817-7f44-eb00-8ff7-e345db3b37d3 [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.877Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: starting restore from snapshot: id=2-6-1717267289877 last-index=6 last-term=2 size-in-bytes=25 2024-06-01T18:41:29.878Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: snapshot restore progress: id=2-6-1717267289877 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2024-06-01T18:41:29.878Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: restored from snapshot: id=2-6-1717267289877 last-index=6 last-term=2 size-in-bytes=25 2024-06-01T18:41:29.878Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3322817-7f44-eb00-8ff7-e345db3b37d3 Address:c3322817-7f44-eb00-8ff7-e345db3b37d3} {Suffrage:Voter ID:server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 Address:bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2} {Suffrage:Voter ID:server-a49c7011-69e0-effe-884f-a696b336291e Address:a49c7011-69e0-effe-884f-a696b336291e}]" 2024-06-01T18:41:29.878Z [INFO] creating new snapshot: path=/tmp/raft783718271/snapshots/2-6-1717267289878.tmp 2024-06-01T18:41:29.878Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: entering follower state: follower="Node at bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.878Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: starting restore from snapshot: id=2-6-1717267289878 last-index=6 last-term=2 size-in-bytes=25 2024-06-01T18:41:29.879Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: snapshot restore progress: id=2-6-1717267289878 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2024-06-01T18:41:29.879Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: restored from snapshot: id=2-6-1717267289878 last-index=6 last-term=2 size-in-bytes=25 2024-06-01T18:41:29.879Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3322817-7f44-eb00-8ff7-e345db3b37d3 Address:c3322817-7f44-eb00-8ff7-e345db3b37d3} {Suffrage:Voter ID:server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 Address:bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2} {Suffrage:Voter ID:server-a49c7011-69e0-effe-884f-a696b336291e Address:a49c7011-69e0-effe-884f-a696b336291e}]" 2024-06-01T18:41:29.879Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: entering follower state: follower="Node at a49c7011-69e0-effe-884f-a696b336291e [Follower]" leader-address= leader-id= 2024-06-01T18:41:29.943Z [WARN] server-a49c7011-69e0-effe-884f-a696b336291e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:29.943Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: entering candidate state: node="Node at a49c7011-69e0-effe-884f-a696b336291e [Candidate]" term=3 2024-06-01T18:41:29.943Z [WARN] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:29.943Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: entering candidate state: node="Node at c3322817-7f44-eb00-8ff7-e345db3b37d3 [Candidate]" term=3 2024-06-01T18:41:29.943Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: election won: term=3 tally=2 2024-06-01T18:41:29.943Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: entering leader state: leader="Node at a49c7011-69e0-effe-884f-a696b336291e [Leader]" 2024-06-01T18:41:29.943Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: added peer, starting replication: peer=server-c3322817-7f44-eb00-8ff7-e345db3b37d3 2024-06-01T18:41:29.943Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: added peer, starting replication: peer=server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 2024-06-01T18:41:29.943Z [INFO] server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2: duplicate requestVote for same term: term=3 2024-06-01T18:41:29.943Z [WARN] server-a49c7011-69e0-effe-884f-a696b336291e: rejecting vote request since we have a leader: from=c3322817-7f44-eb00-8ff7-e345db3b37d3 leader=a49c7011-69e0-effe-884f-a696b336291e leader-id=server-a49c7011-69e0-effe-884f-a696b336291e 2024-06-01T18:41:29.943Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: duplicate requestVote for same term: term=3 2024-06-01T18:41:29.943Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: pipelining replication: peer="{Voter server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2}" 2024-06-01T18:41:29.943Z [INFO] server-c3322817-7f44-eb00-8ff7-e345db3b37d3: entering follower state: follower="Node at c3322817-7f44-eb00-8ff7-e345db3b37d3 [Follower]" leader-address=a49c7011-69e0-effe-884f-a696b336291e leader-id=server-a49c7011-69e0-effe-884f-a696b336291e 2024-06-01T18:41:29.943Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: pipelining replication: peer="{Voter server-c3322817-7f44-eb00-8ff7-e345db3b37d3 c3322817-7f44-eb00-8ff7-e345db3b37d3}" 2024-06-01T18:41:30.194Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:30.300Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:30.194791792 +0000 UTC m=+5.692981244 (1 nodes), 17.583µs from start of poll, 503.814094ms from cluster start. Timeout at 2024-06-01 18:41:30.30052568 +0000 UTC m=+5.798715132, 105.733888ms after stability 2024-06-01T18:41:30.300Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: aborting pipeline replication: peer="{Voter server-bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2 bcb6fb8d-3ceb-a2c9-cc12-9b2719abe0c2}" 2024-06-01T18:41:30.300Z [INFO] server-a49c7011-69e0-effe-884f-a696b336291e: aborting pipeline replication: peer="{Voter server-c3322817-7f44-eb00-8ff7-e345db3b37d3 c3322817-7f44-eb00-8ff7-e345db3b37d3}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2024-06-01T18:41:30.301Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb786a09-8730-30fa-a9f2-84370559d1c6 Address:bb786a09-8730-30fa-a9f2-84370559d1c6} {Suffrage:Voter ID:server-b34f3c2d-f571-ad44-5938-e4cfee65b396 Address:b34f3c2d-f571-ad44-5938-e4cfee65b396} {Suffrage:Voter ID:server-7248b046-fa7d-40e8-a945-9fb7bf715692 Address:7248b046-fa7d-40e8-a945-9fb7bf715692}]" 2024-06-01T18:41:30.301Z [INFO] server-b34f3c2d-f571-ad44-5938-e4cfee65b396: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb786a09-8730-30fa-a9f2-84370559d1c6 Address:bb786a09-8730-30fa-a9f2-84370559d1c6} {Suffrage:Voter ID:server-b34f3c2d-f571-ad44-5938-e4cfee65b396 Address:b34f3c2d-f571-ad44-5938-e4cfee65b396} {Suffrage:Voter ID:server-7248b046-fa7d-40e8-a945-9fb7bf715692 Address:7248b046-fa7d-40e8-a945-9fb7bf715692}]" 2024-06-01T18:41:30.301Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: entering follower state: follower="Node at bb786a09-8730-30fa-a9f2-84370559d1c6 [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.301Z [INFO] server-b34f3c2d-f571-ad44-5938-e4cfee65b396: entering follower state: follower="Node at b34f3c2d-f571-ad44-5938-e4cfee65b396 [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.301Z [INFO] server-7248b046-fa7d-40e8-a945-9fb7bf715692: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb786a09-8730-30fa-a9f2-84370559d1c6 Address:bb786a09-8730-30fa-a9f2-84370559d1c6} {Suffrage:Voter ID:server-b34f3c2d-f571-ad44-5938-e4cfee65b396 Address:b34f3c2d-f571-ad44-5938-e4cfee65b396} {Suffrage:Voter ID:server-7248b046-fa7d-40e8-a945-9fb7bf715692 Address:7248b046-fa7d-40e8-a945-9fb7bf715692}]" 2024-06-01T18:41:30.301Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:30.301Z [INFO] server-7248b046-fa7d-40e8-a945-9fb7bf715692: entering follower state: follower="Node at 7248b046-fa7d-40e8-a945-9fb7bf715692 [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.358Z [WARN] server-bb786a09-8730-30fa-a9f2-84370559d1c6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:30.358Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: entering candidate state: node="Node at bb786a09-8730-30fa-a9f2-84370559d1c6 [Candidate]" term=2 2024-06-01T18:41:30.358Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: election won: term=2 tally=2 2024-06-01T18:41:30.358Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: entering leader state: leader="Node at bb786a09-8730-30fa-a9f2-84370559d1c6 [Leader]" 2024-06-01T18:41:30.358Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: added peer, starting replication: peer=server-b34f3c2d-f571-ad44-5938-e4cfee65b396 2024-06-01T18:41:30.358Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: added peer, starting replication: peer=server-7248b046-fa7d-40e8-a945-9fb7bf715692 2024-06-01T18:41:30.358Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: pipelining replication: peer="{Voter server-7248b046-fa7d-40e8-a945-9fb7bf715692 7248b046-fa7d-40e8-a945-9fb7bf715692}" 2024-06-01T18:41:30.358Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: pipelining replication: peer="{Voter server-b34f3c2d-f571-ad44-5938-e4cfee65b396 b34f3c2d-f571-ad44-5938-e4cfee65b396}" 2024-06-01T18:41:30.464Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:30.358360042 +0000 UTC m=+5.856549494 (1 nodes), 56.457799ms from start of poll, 56.681435ms from cluster start. Timeout at 2024-06-01 18:41:30.464278804 +0000 UTC m=+5.962468256, 105.918762ms after stability 2024-06-01T18:41:30.466Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: aborting pipeline replication: peer="{Voter server-7248b046-fa7d-40e8-a945-9fb7bf715692 7248b046-fa7d-40e8-a945-9fb7bf715692}" 2024-06-01T18:41:30.466Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: aborting pipeline replication: peer="{Voter server-b34f3c2d-f571-ad44-5938-e4cfee65b396 b34f3c2d-f571-ad44-5938-e4cfee65b396}" 2024-06-01T18:41:30.466Z [INFO] creating new snapshot: path=/tmp/raft2749328015/snapshots/2-27-1717267290466.tmp 2024-06-01T18:41:30.466Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: starting restore from snapshot: id=2-27-1717267290466 last-index=27 last-term=2 size-in-bytes=168 2024-06-01T18:41:30.466Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: snapshot restore progress: id=2-27-1717267290466 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2024-06-01T18:41:30.466Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: restored from snapshot: id=2-27-1717267290466 last-index=27 last-term=2 size-in-bytes=168 2024-06-01T18:41:30.466Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb786a09-8730-30fa-a9f2-84370559d1c6 Address:bb786a09-8730-30fa-a9f2-84370559d1c6} {Suffrage:Voter ID:server-b34f3c2d-f571-ad44-5938-e4cfee65b396 Address:b34f3c2d-f571-ad44-5938-e4cfee65b396} {Suffrage:Voter ID:server-7248b046-fa7d-40e8-a945-9fb7bf715692 Address:7248b046-fa7d-40e8-a945-9fb7bf715692}]" 2024-06-01T18:41:30.466Z [INFO] creating new snapshot: path=/tmp/raft678495430/snapshots/2-27-1717267290466.tmp 2024-06-01T18:41:30.466Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: entering follower state: follower="Node at bb786a09-8730-30fa-a9f2-84370559d1c6 [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.467Z [INFO] server-b34f3c2d-f571-ad44-5938-e4cfee65b396: starting restore from snapshot: id=2-27-1717267290466 last-index=27 last-term=2 size-in-bytes=168 2024-06-01T18:41:30.467Z [INFO] server-b34f3c2d-f571-ad44-5938-e4cfee65b396: snapshot restore progress: id=2-27-1717267290466 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2024-06-01T18:41:30.467Z [INFO] server-b34f3c2d-f571-ad44-5938-e4cfee65b396: restored from snapshot: id=2-27-1717267290466 last-index=27 last-term=2 size-in-bytes=168 2024-06-01T18:41:30.467Z [INFO] server-b34f3c2d-f571-ad44-5938-e4cfee65b396: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb786a09-8730-30fa-a9f2-84370559d1c6 Address:bb786a09-8730-30fa-a9f2-84370559d1c6} {Suffrage:Voter ID:server-b34f3c2d-f571-ad44-5938-e4cfee65b396 Address:b34f3c2d-f571-ad44-5938-e4cfee65b396} {Suffrage:Voter ID:server-7248b046-fa7d-40e8-a945-9fb7bf715692 Address:7248b046-fa7d-40e8-a945-9fb7bf715692}]" 2024-06-01T18:41:30.467Z [INFO] server-b34f3c2d-f571-ad44-5938-e4cfee65b396: entering follower state: follower="Node at b34f3c2d-f571-ad44-5938-e4cfee65b396 [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.467Z [INFO] creating new snapshot: path=/tmp/raft3884974810/snapshots/2-27-1717267290467.tmp 2024-06-01T18:41:30.467Z [INFO] server-7248b046-fa7d-40e8-a945-9fb7bf715692: starting restore from snapshot: id=2-27-1717267290467 last-index=27 last-term=2 size-in-bytes=168 2024-06-01T18:41:30.467Z [INFO] server-7248b046-fa7d-40e8-a945-9fb7bf715692: snapshot restore progress: id=2-27-1717267290467 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2024-06-01T18:41:30.467Z [INFO] server-7248b046-fa7d-40e8-a945-9fb7bf715692: restored from snapshot: id=2-27-1717267290467 last-index=27 last-term=2 size-in-bytes=168 2024-06-01T18:41:30.467Z [INFO] server-7248b046-fa7d-40e8-a945-9fb7bf715692: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb786a09-8730-30fa-a9f2-84370559d1c6 Address:bb786a09-8730-30fa-a9f2-84370559d1c6} {Suffrage:Voter ID:server-b34f3c2d-f571-ad44-5938-e4cfee65b396 Address:b34f3c2d-f571-ad44-5938-e4cfee65b396} {Suffrage:Voter ID:server-7248b046-fa7d-40e8-a945-9fb7bf715692 Address:7248b046-fa7d-40e8-a945-9fb7bf715692}]" 2024-06-01T18:41:30.467Z [INFO] server-7248b046-fa7d-40e8-a945-9fb7bf715692: entering follower state: follower="Node at 7248b046-fa7d-40e8-a945-9fb7bf715692 [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.518Z [WARN] server-bb786a09-8730-30fa-a9f2-84370559d1c6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:30.518Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: entering candidate state: node="Node at bb786a09-8730-30fa-a9f2-84370559d1c6 [Candidate]" term=3 2024-06-01T18:41:30.518Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: election won: term=3 tally=2 2024-06-01T18:41:30.518Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: entering leader state: leader="Node at bb786a09-8730-30fa-a9f2-84370559d1c6 [Leader]" 2024-06-01T18:41:30.518Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: added peer, starting replication: peer=server-b34f3c2d-f571-ad44-5938-e4cfee65b396 2024-06-01T18:41:30.518Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: added peer, starting replication: peer=server-7248b046-fa7d-40e8-a945-9fb7bf715692 2024-06-01T18:41:30.518Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: pipelining replication: peer="{Voter server-b34f3c2d-f571-ad44-5938-e4cfee65b396 b34f3c2d-f571-ad44-5938-e4cfee65b396}" 2024-06-01T18:41:30.518Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: pipelining replication: peer="{Voter server-7248b046-fa7d-40e8-a945-9fb7bf715692 7248b046-fa7d-40e8-a945-9fb7bf715692}" 2024-06-01T18:41:30.784Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:30.890Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:30.784116537 +0000 UTC m=+6.282305989 (1 nodes), 34.254µs from start of poll, 482.43793ms from cluster start. Timeout at 2024-06-01 18:41:30.890171924 +0000 UTC m=+6.388361376, 106.055387ms after stability 2024-06-01T18:41:30.890Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: aborting pipeline replication: peer="{Voter server-7248b046-fa7d-40e8-a945-9fb7bf715692 7248b046-fa7d-40e8-a945-9fb7bf715692}" 2024-06-01T18:41:30.890Z [INFO] server-bb786a09-8730-30fa-a9f2-84370559d1c6: aborting pipeline replication: peer="{Voter server-b34f3c2d-f571-ad44-5938-e4cfee65b396 b34f3c2d-f571-ad44-5938-e4cfee65b396}" --- PASS: TestRaft_RecoverCluster (1.80s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.61s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.59s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2024-06-01T18:41:30.891Z [INFO] server-774f68c7-c409-15ab-fd96-89cf9c33f5ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-774f68c7-c409-15ab-fd96-89cf9c33f5ed Address:774f68c7-c409-15ab-fd96-89cf9c33f5ed}]" 2024-06-01T18:41:30.891Z [INFO] server-774f68c7-c409-15ab-fd96-89cf9c33f5ed: entering follower state: follower="Node at 774f68c7-c409-15ab-fd96-89cf9c33f5ed [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.963Z [WARN] server-774f68c7-c409-15ab-fd96-89cf9c33f5ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:30.963Z [INFO] server-774f68c7-c409-15ab-fd96-89cf9c33f5ed: entering candidate state: node="Node at 774f68c7-c409-15ab-fd96-89cf9c33f5ed [Candidate]" term=2 2024-06-01T18:41:30.963Z [INFO] server-774f68c7-c409-15ab-fd96-89cf9c33f5ed: election won: term=2 tally=1 2024-06-01T18:41:30.963Z [INFO] server-774f68c7-c409-15ab-fd96-89cf9c33f5ed: entering leader state: leader="Node at 774f68c7-c409-15ab-fd96-89cf9c33f5ed [Leader]" --- PASS: TestRaft_SingleNode (0.07s) === RUN TestRaft_TripleNode 2024-06-01T18:41:30.964Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0654227f-2405-b5cc-2205-e47242702edd Address:0654227f-2405-b5cc-2205-e47242702edd} {Suffrage:Voter ID:server-7d426642-11ad-0aac-c948-606493c922a9 Address:7d426642-11ad-0aac-c948-606493c922a9} {Suffrage:Voter ID:server-be6cb367-c3f1-b2f7-2091-c941bf194d80 Address:be6cb367-c3f1-b2f7-2091-c941bf194d80}]" 2024-06-01T18:41:30.964Z [INFO] server-7d426642-11ad-0aac-c948-606493c922a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0654227f-2405-b5cc-2205-e47242702edd Address:0654227f-2405-b5cc-2205-e47242702edd} {Suffrage:Voter ID:server-7d426642-11ad-0aac-c948-606493c922a9 Address:7d426642-11ad-0aac-c948-606493c922a9} {Suffrage:Voter ID:server-be6cb367-c3f1-b2f7-2091-c941bf194d80 Address:be6cb367-c3f1-b2f7-2091-c941bf194d80}]" 2024-06-01T18:41:30.964Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: entering follower state: follower="Node at 0654227f-2405-b5cc-2205-e47242702edd [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.964Z [INFO] server-be6cb367-c3f1-b2f7-2091-c941bf194d80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0654227f-2405-b5cc-2205-e47242702edd Address:0654227f-2405-b5cc-2205-e47242702edd} {Suffrage:Voter ID:server-7d426642-11ad-0aac-c948-606493c922a9 Address:7d426642-11ad-0aac-c948-606493c922a9} {Suffrage:Voter ID:server-be6cb367-c3f1-b2f7-2091-c941bf194d80 Address:be6cb367-c3f1-b2f7-2091-c941bf194d80}]" 2024-06-01T18:41:30.964Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:30.964Z [INFO] server-7d426642-11ad-0aac-c948-606493c922a9: entering follower state: follower="Node at 7d426642-11ad-0aac-c948-606493c922a9 [Follower]" leader-address= leader-id= 2024-06-01T18:41:30.964Z [INFO] server-be6cb367-c3f1-b2f7-2091-c941bf194d80: entering follower state: follower="Node at be6cb367-c3f1-b2f7-2091-c941bf194d80 [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.022Z [WARN] server-0654227f-2405-b5cc-2205-e47242702edd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:31.022Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: entering candidate state: node="Node at 0654227f-2405-b5cc-2205-e47242702edd [Candidate]" term=2 2024-06-01T18:41:31.022Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: election won: term=2 tally=2 2024-06-01T18:41:31.022Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: entering leader state: leader="Node at 0654227f-2405-b5cc-2205-e47242702edd [Leader]" 2024-06-01T18:41:31.022Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: added peer, starting replication: peer=server-7d426642-11ad-0aac-c948-606493c922a9 2024-06-01T18:41:31.022Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: added peer, starting replication: peer=server-be6cb367-c3f1-b2f7-2091-c941bf194d80 2024-06-01T18:41:31.022Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: pipelining replication: peer="{Voter server-be6cb367-c3f1-b2f7-2091-c941bf194d80 be6cb367-c3f1-b2f7-2091-c941bf194d80}" 2024-06-01T18:41:31.022Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: pipelining replication: peer="{Voter server-7d426642-11ad-0aac-c948-606493c922a9 7d426642-11ad-0aac-c948-606493c922a9}" 2024-06-01T18:41:31.129Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:31.022843237 +0000 UTC m=+6.521032689 (2 nodes), 58.263409ms from start of poll, 58.47808ms from cluster start. Timeout at 2024-06-01 18:41:31.129058452 +0000 UTC m=+6.627247904, 106.215215ms after stability 2024-06-01T18:41:31.129Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:31.234Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:31.129169118 +0000 UTC m=+6.627358570 (1 nodes), 1.242µs from start of poll, 164.803961ms from cluster start. Timeout at 2024-06-01 18:41:31.234440736 +0000 UTC m=+6.732630188, 105.271618ms after stability 2024-06-01T18:41:31.245Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: aborting pipeline replication: peer="{Voter server-be6cb367-c3f1-b2f7-2091-c941bf194d80 be6cb367-c3f1-b2f7-2091-c941bf194d80}" 2024-06-01T18:41:31.245Z [INFO] server-0654227f-2405-b5cc-2205-e47242702edd: aborting pipeline replication: peer="{Voter server-7d426642-11ad-0aac-c948-606493c922a9 7d426642-11ad-0aac-c948-606493c922a9}" --- PASS: TestRaft_TripleNode (0.28s) === RUN TestRaft_LeaderFail 2024-06-01T18:41:31.246Z [INFO] server-7a247857-fdae-09a8-9e84-37e25a8ebc99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a247857-fdae-09a8-9e84-37e25a8ebc99 Address:7a247857-fdae-09a8-9e84-37e25a8ebc99} {Suffrage:Voter ID:server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 Address:fe70d031-a6e7-3fba-825e-edaa0748f9c3} {Suffrage:Voter ID:server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 Address:dc843e96-37c9-cb32-aad8-dd3489b9c4a2}]" 2024-06-01T18:41:31.246Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a247857-fdae-09a8-9e84-37e25a8ebc99 Address:7a247857-fdae-09a8-9e84-37e25a8ebc99} {Suffrage:Voter ID:server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 Address:fe70d031-a6e7-3fba-825e-edaa0748f9c3} {Suffrage:Voter ID:server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 Address:dc843e96-37c9-cb32-aad8-dd3489b9c4a2}]" 2024-06-01T18:41:31.246Z [INFO] server-7a247857-fdae-09a8-9e84-37e25a8ebc99: entering follower state: follower="Node at 7a247857-fdae-09a8-9e84-37e25a8ebc99 [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.246Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a247857-fdae-09a8-9e84-37e25a8ebc99 Address:7a247857-fdae-09a8-9e84-37e25a8ebc99} {Suffrage:Voter ID:server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 Address:fe70d031-a6e7-3fba-825e-edaa0748f9c3} {Suffrage:Voter ID:server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 Address:dc843e96-37c9-cb32-aad8-dd3489b9c4a2}]" 2024-06-01T18:41:31.246Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:31.246Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: entering follower state: follower="Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.246Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: entering follower state: follower="Node at dc843e96-37c9-cb32-aad8-dd3489b9c4a2 [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.314Z [WARN] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:31.315Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: entering candidate state: node="Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Candidate]" term=2 2024-06-01T18:41:31.315Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: election won: term=2 tally=2 2024-06-01T18:41:31.315Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: entering leader state: leader="Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Leader]" 2024-06-01T18:41:31.315Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: added peer, starting replication: peer=server-7a247857-fdae-09a8-9e84-37e25a8ebc99 2024-06-01T18:41:31.315Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: added peer, starting replication: peer=server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 2024-06-01T18:41:31.315Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: pipelining replication: peer="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" 2024-06-01T18:41:31.315Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: pipelining replication: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" 2024-06-01T18:41:31.421Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:31.315261498 +0000 UTC m=+6.813450950 (2 nodes), 68.283793ms from start of poll, 68.546823ms from cluster start. Timeout at 2024-06-01 18:41:31.421034749 +0000 UTC m=+6.919224201, 105.773251ms after stability 2024-06-01T18:41:31.421Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:31.526Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:31.421117974 +0000 UTC m=+6.919307426 (1 nodes), 1.653µs from start of poll, 174.403299ms from cluster start. Timeout at 2024-06-01 18:41:31.526892909 +0000 UTC m=+7.025082361, 105.774935ms after stability raft_test.go:378: [INFO] Disconnecting Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Leader] 2024-06-01T18:41:31.543Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to pipeline appendEntries: peer="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" error="append pipeline closed" 2024-06-01T18:41:31.543Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: aborting pipeline replication: peer="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" 2024-06-01T18:41:31.543Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:31.544Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to heartbeat to: peer=7a247857-fdae-09a8-9e84-37e25a8ebc99 backoff time=10ms error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" 2024-06-01T18:41:31.545Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to pipeline appendEntries: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" error="append pipeline closed" 2024-06-01T18:41:31.545Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: aborting pipeline replication: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" 2024-06-01T18:41:31.545Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to heartbeat to: peer=dc843e96-37c9-cb32-aad8-dd3489b9c4a2 backoff time=10ms error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" 2024-06-01T18:41:31.550Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to appendEntries to: peer="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" 2024-06-01T18:41:31.555Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to appendEntries to: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" 2024-06-01T18:41:31.564Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to heartbeat to: peer=7a247857-fdae-09a8-9e84-37e25a8ebc99 backoff time=10ms error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" 2024-06-01T18:41:31.565Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to heartbeat to: peer=dc843e96-37c9-cb32-aad8-dd3489b9c4a2 backoff time=10ms error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" 2024-06-01T18:41:31.571Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to appendEntries to: peer="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" 2024-06-01T18:41:31.572Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to appendEntries to: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" 2024-06-01T18:41:31.583Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to heartbeat to: peer=dc843e96-37c9-cb32-aad8-dd3489b9c4a2 backoff time=10ms error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" 2024-06-01T18:41:31.585Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to heartbeat to: peer=7a247857-fdae-09a8-9e84-37e25a8ebc99 backoff time=10ms error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" 2024-06-01T18:41:31.587Z [WARN] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to contact: server-id=server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 time=50.853366ms 2024-06-01T18:41:31.592Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to appendEntries to: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" 2024-06-01T18:41:31.592Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to appendEntries to: peer="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" 2024-06-01T18:41:31.598Z [WARN] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to contact: server-id=server-7a247857-fdae-09a8-9e84-37e25a8ebc99 time=60.581405ms 2024-06-01T18:41:31.598Z [WARN] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to contact: server-id=server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 time=61.735013ms 2024-06-01T18:41:31.598Z [WARN] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to contact quorum of nodes, stepping down 2024-06-01T18:41:31.598Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: entering follower state: follower="Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.607Z [WARN] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: heartbeat timeout reached, starting election: last-leader-addr=fe70d031-a6e7-3fba-825e-edaa0748f9c3 last-leader-id=server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 2024-06-01T18:41:31.607Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: entering candidate state: node="Node at dc843e96-37c9-cb32-aad8-dd3489b9c4a2 [Candidate]" term=3 2024-06-01T18:41:31.607Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to make requestVote RPC: target="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" term=3 2024-06-01T18:41:31.607Z [WARN] server-7a247857-fdae-09a8-9e84-37e25a8ebc99: rejecting vote request since we have a leader: from=dc843e96-37c9-cb32-aad8-dd3489b9c4a2 leader=fe70d031-a6e7-3fba-825e-edaa0748f9c3 leader-id=server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 2024-06-01T18:41:31.649Z [WARN] server-7a247857-fdae-09a8-9e84-37e25a8ebc99: heartbeat timeout reached, starting election: last-leader-addr=fe70d031-a6e7-3fba-825e-edaa0748f9c3 last-leader-id=server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 2024-06-01T18:41:31.649Z [INFO] server-7a247857-fdae-09a8-9e84-37e25a8ebc99: entering candidate state: node="Node at 7a247857-fdae-09a8-9e84-37e25a8ebc99 [Candidate]" term=3 2024-06-01T18:41:31.649Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: duplicate requestVote for same term: term=3 2024-06-01T18:41:31.649Z [ERROR] server-7a247857-fdae-09a8-9e84-37e25a8ebc99: failed to make requestVote RPC: target="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" term=3 2024-06-01T18:41:31.658Z [WARN] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: Election timeout reached, restarting election 2024-06-01T18:41:31.658Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: entering candidate state: node="Node at dc843e96-37c9-cb32-aad8-dd3489b9c4a2 [Candidate]" term=4 2024-06-01T18:41:31.658Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to make requestVote RPC: target="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" term=4 2024-06-01T18:41:31.658Z [INFO] server-7a247857-fdae-09a8-9e84-37e25a8ebc99: entering follower state: follower="Node at 7a247857-fdae-09a8-9e84-37e25a8ebc99 [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.658Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: election won: term=4 tally=2 2024-06-01T18:41:31.658Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: entering leader state: leader="Node at dc843e96-37c9-cb32-aad8-dd3489b9c4a2 [Leader]" 2024-06-01T18:41:31.658Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: added peer, starting replication: peer=server-7a247857-fdae-09a8-9e84-37e25a8ebc99 2024-06-01T18:41:31.658Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: added peer, starting replication: peer=server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 2024-06-01T18:41:31.658Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to appendEntries to: peer="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.658Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: pipelining replication: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" 2024-06-01T18:41:31.666Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to heartbeat to: peer=fe70d031-a6e7-3fba-825e-edaa0748f9c3 backoff time=10ms error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.677Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to appendEntries to: peer="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.685Z [WARN] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:31.685Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to heartbeat to: peer=fe70d031-a6e7-3fba-825e-edaa0748f9c3 backoff time=10ms error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.685Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: entering candidate state: node="Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Candidate]" term=3 2024-06-01T18:41:31.685Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to make requestVote RPC: target="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" term=3 2024-06-01T18:41:31.685Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to make requestVote RPC: target="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" term=3 2024-06-01T18:41:31.694Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to appendEntries to: peer="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.702Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to heartbeat to: peer=fe70d031-a6e7-3fba-825e-edaa0748f9c3 backoff time=10ms error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.709Z [WARN] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to contact: server-id=server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 time=50.60879ms 2024-06-01T18:41:31.720Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to appendEntries to: peer="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.721Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to heartbeat to: peer=fe70d031-a6e7-3fba-825e-edaa0748f9c3 backoff time=20ms error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.747Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to heartbeat to: peer=fe70d031-a6e7-3fba-825e-edaa0748f9c3 backoff time=25ms error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.750Z [WARN] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: Election timeout reached, restarting election 2024-06-01T18:41:31.750Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: entering candidate state: node="Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Candidate]" term=4 2024-06-01T18:41:31.751Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to make requestVote RPC: target="{Voter server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 dc843e96-37c9-cb32-aad8-dd3489b9c4a2}" error="failed to connect to peer: dc843e96-37c9-cb32-aad8-dd3489b9c4a2" term=4 2024-06-01T18:41:31.751Z [ERROR] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: failed to make requestVote RPC: target="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" error="failed to connect to peer: 7a247857-fdae-09a8-9e84-37e25a8ebc99" term=4 2024-06-01T18:41:31.758Z [WARN] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to contact: server-id=server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 time=99.752233ms 2024-06-01T18:41:31.769Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to appendEntries to: peer="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.781Z [ERROR] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to heartbeat to: peer=fe70d031-a6e7-3fba-825e-edaa0748f9c3 backoff time=25ms error="failed to connect to peer: fe70d031-a6e7-3fba-825e-edaa0748f9c3" 2024-06-01T18:41:31.790Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:31.685298869 +0000 UTC m=+7.183488361 (1 nodes), 141.716451ms from start of poll, 438.584234ms from cluster start. Timeout at 2024-06-01 18:41:31.790939314 +0000 UTC m=+7.289128767, 105.640406ms after stability raft_test.go:413: [INFO] Reconnecting Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Candidate] 2024-06-01T18:41:31.806Z [WARN] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: failed to contact: server-id=server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 time=147.381817ms 2024-06-01T18:41:31.815Z [INFO] server-fe70d031-a6e7-3fba-825e-edaa0748f9c3: entering follower state: follower="Node at fe70d031-a6e7-3fba-825e-edaa0748f9c3 [Follower]" leader-address=dc843e96-37c9-cb32-aad8-dd3489b9c4a2 leader-id=server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2 2024-06-01T18:41:31.859Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: pipelining replication: peer="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" 2024-06-01T18:41:31.865Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: aborting pipeline replication: peer="{Voter server-fe70d031-a6e7-3fba-825e-edaa0748f9c3 fe70d031-a6e7-3fba-825e-edaa0748f9c3}" 2024-06-01T18:41:31.865Z [INFO] server-dc843e96-37c9-cb32-aad8-dd3489b9c4a2: aborting pipeline replication: peer="{Voter server-7a247857-fdae-09a8-9e84-37e25a8ebc99 7a247857-fdae-09a8-9e84-37e25a8ebc99}" --- PASS: TestRaft_LeaderFail (0.62s) === RUN TestRaft_BehindFollower 2024-06-01T18:41:31.866Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f Address:c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f} {Suffrage:Voter ID:server-768bd837-228c-9700-eb27-7eab93188dad Address:768bd837-228c-9700-eb27-7eab93188dad} {Suffrage:Voter ID:server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e Address:d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e}]" 2024-06-01T18:41:31.866Z [INFO] server-768bd837-228c-9700-eb27-7eab93188dad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f Address:c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f} {Suffrage:Voter ID:server-768bd837-228c-9700-eb27-7eab93188dad Address:768bd837-228c-9700-eb27-7eab93188dad} {Suffrage:Voter ID:server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e Address:d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e}]" 2024-06-01T18:41:31.866Z [INFO] server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f Address:c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f} {Suffrage:Voter ID:server-768bd837-228c-9700-eb27-7eab93188dad Address:768bd837-228c-9700-eb27-7eab93188dad} {Suffrage:Voter ID:server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e Address:d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e}]" 2024-06-01T18:41:31.866Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: entering follower state: follower="Node at c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.866Z [INFO] server-768bd837-228c-9700-eb27-7eab93188dad: entering follower state: follower="Node at 768bd837-228c-9700-eb27-7eab93188dad [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.866Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:31.866Z [INFO] server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e: entering follower state: follower="Node at d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e [Follower]" leader-address= leader-id= 2024-06-01T18:41:31.930Z [WARN] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:31.930Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: entering candidate state: node="Node at c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f [Candidate]" term=2 2024-06-01T18:41:31.930Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: election won: term=2 tally=2 2024-06-01T18:41:31.930Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: entering leader state: leader="Node at c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f [Leader]" 2024-06-01T18:41:31.930Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: added peer, starting replication: peer=server-768bd837-228c-9700-eb27-7eab93188dad 2024-06-01T18:41:31.930Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: added peer, starting replication: peer=server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e 2024-06-01T18:41:31.931Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: pipelining replication: peer="{Voter server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e}" 2024-06-01T18:41:31.931Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: pipelining replication: peer="{Voter server-768bd837-228c-9700-eb27-7eab93188dad 768bd837-228c-9700-eb27-7eab93188dad}" 2024-06-01T18:41:32.037Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:31.931166563 +0000 UTC m=+7.429356015 (1 nodes), 64.26974ms from start of poll, 64.587181ms from cluster start. Timeout at 2024-06-01 18:41:32.037184141 +0000 UTC m=+7.535373593, 106.017578ms after stability 2024-06-01T18:41:32.037Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:32.143Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:32.0372644 +0000 UTC m=+7.535453852 (2 nodes), 851ns from start of poll, 170.685018ms from cluster start. Timeout at 2024-06-01 18:41:32.143349261 +0000 UTC m=+7.641538713, 106.084861ms after stability 2024-06-01T18:41:32.143Z [ERROR] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: failed to pipeline appendEntries: peer="{Voter server-768bd837-228c-9700-eb27-7eab93188dad 768bd837-228c-9700-eb27-7eab93188dad}" error="append pipeline closed" 2024-06-01T18:41:32.143Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: aborting pipeline replication: peer="{Voter server-768bd837-228c-9700-eb27-7eab93188dad 768bd837-228c-9700-eb27-7eab93188dad}" 2024-06-01T18:41:32.143Z [ERROR] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: failed to appendEntries to: peer="{Voter server-768bd837-228c-9700-eb27-7eab93188dad 768bd837-228c-9700-eb27-7eab93188dad}" error="failed to connect to peer: 768bd837-228c-9700-eb27-7eab93188dad" 2024-06-01T18:41:32.146Z [ERROR] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: failed to heartbeat to: peer=768bd837-228c-9700-eb27-7eab93188dad backoff time=10ms error="failed to connect to peer: 768bd837-228c-9700-eb27-7eab93188dad" raft_test.go:459: [INFO] Finished apply without behind follower 2024-06-01T18:41:32.153Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: pipelining replication: peer="{Voter server-768bd837-228c-9700-eb27-7eab93188dad 768bd837-228c-9700-eb27-7eab93188dad}" 2024-06-01T18:41:32.162Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:32.268Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:32.163004691 +0000 UTC m=+7.661194143 (1 nodes), 651ns from start of poll, 296.425309ms from cluster start. Timeout at 2024-06-01 18:41:32.268564836 +0000 UTC m=+7.766754288, 105.560145ms after stability 2024-06-01T18:41:32.268Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: aborting pipeline replication: peer="{Voter server-d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e d6ff38da-7f9c-6dc3-b8f9-f4c6a1a6cd9e}" 2024-06-01T18:41:32.268Z [INFO] server-c16e5b3c-a7dc-b76f-4a2b-10c4cc857c5f: aborting pipeline replication: peer="{Voter server-768bd837-228c-9700-eb27-7eab93188dad 768bd837-228c-9700-eb27-7eab93188dad}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader 2024-06-01T18:41:32.270Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-664c240c-e6c0-963b-57da-f82db3e7b50c Address:664c240c-e6c0-963b-57da-f82db3e7b50c} {Suffrage:Voter ID:server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04 Address:f4314dc1-918d-b2b9-3663-b5fca9ed9c04} {Suffrage:Voter ID:server-88862128-4375-1ea6-9bb0-e10d1ce319b2 Address:88862128-4375-1ea6-9bb0-e10d1ce319b2}]" 2024-06-01T18:41:32.270Z [INFO] server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-664c240c-e6c0-963b-57da-f82db3e7b50c Address:664c240c-e6c0-963b-57da-f82db3e7b50c} {Suffrage:Voter ID:server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04 Address:f4314dc1-918d-b2b9-3663-b5fca9ed9c04} {Suffrage:Voter ID:server-88862128-4375-1ea6-9bb0-e10d1ce319b2 Address:88862128-4375-1ea6-9bb0-e10d1ce319b2}]" 2024-06-01T18:41:32.270Z [INFO] server-88862128-4375-1ea6-9bb0-e10d1ce319b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-664c240c-e6c0-963b-57da-f82db3e7b50c Address:664c240c-e6c0-963b-57da-f82db3e7b50c} {Suffrage:Voter ID:server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04 Address:f4314dc1-918d-b2b9-3663-b5fca9ed9c04} {Suffrage:Voter ID:server-88862128-4375-1ea6-9bb0-e10d1ce319b2 Address:88862128-4375-1ea6-9bb0-e10d1ce319b2}]" 2024-06-01T18:41:32.270Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: entering follower state: follower="Node at 664c240c-e6c0-963b-57da-f82db3e7b50c [Follower]" leader-address= leader-id= 2024-06-01T18:41:32.270Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:32.270Z [INFO] server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04: entering follower state: follower="Node at f4314dc1-918d-b2b9-3663-b5fca9ed9c04 [Follower]" leader-address= leader-id= 2024-06-01T18:41:32.270Z [INFO] server-88862128-4375-1ea6-9bb0-e10d1ce319b2: entering follower state: follower="Node at 88862128-4375-1ea6-9bb0-e10d1ce319b2 [Follower]" leader-address= leader-id= 2024-06-01T18:41:32.344Z [WARN] server-664c240c-e6c0-963b-57da-f82db3e7b50c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:32.344Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: entering candidate state: node="Node at 664c240c-e6c0-963b-57da-f82db3e7b50c [Candidate]" term=2 2024-06-01T18:41:32.344Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: election won: term=2 tally=2 2024-06-01T18:41:32.344Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: entering leader state: leader="Node at 664c240c-e6c0-963b-57da-f82db3e7b50c [Leader]" 2024-06-01T18:41:32.344Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: added peer, starting replication: peer=server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04 2024-06-01T18:41:32.344Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: added peer, starting replication: peer=server-88862128-4375-1ea6-9bb0-e10d1ce319b2 2024-06-01T18:41:32.344Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: pipelining replication: peer="{Voter server-88862128-4375-1ea6-9bb0-e10d1ce319b2 88862128-4375-1ea6-9bb0-e10d1ce319b2}" 2024-06-01T18:41:32.344Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: pipelining replication: peer="{Voter server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04 f4314dc1-918d-b2b9-3663-b5fca9ed9c04}" 2024-06-01T18:41:32.449Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:32.34427324 +0000 UTC m=+7.842462692 (1 nodes), 74.007798ms from start of poll, 74.294622ms from cluster start. Timeout at 2024-06-01 18:41:32.449767052 +0000 UTC m=+7.947956504, 105.493812ms after stability 2024-06-01T18:41:32.449Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:32.555Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:32.449839877 +0000 UTC m=+7.948029330 (2 nodes), 1.884µs from start of poll, 179.86126ms from cluster start. Timeout at 2024-06-01 18:41:32.555469663 +0000 UTC m=+8.053659115, 105.629785ms after stability 2024-06-01T18:41:32.555Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: aborting pipeline replication: peer="{Voter server-88862128-4375-1ea6-9bb0-e10d1ce319b2 88862128-4375-1ea6-9bb0-e10d1ce319b2}" 2024-06-01T18:41:32.555Z [INFO] server-664c240c-e6c0-963b-57da-f82db3e7b50c: aborting pipeline replication: peer="{Voter server-f4314dc1-918d-b2b9-3663-b5fca9ed9c04 f4314dc1-918d-b2b9-3663-b5fca9ed9c04}" --- PASS: TestRaft_ApplyNonLeader (0.29s) === RUN TestRaft_ApplyConcurrent 2024-06-01T18:41:32.556Z [INFO] server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a Address:2f34d9bf-6566-dfb2-3749-8a5d47e65c9a} {Suffrage:Voter ID:server-18e15b86-cfb5-1196-cb56-aaad9be54f4b Address:18e15b86-cfb5-1196-cb56-aaad9be54f4b} {Suffrage:Voter ID:server-67bff599-8edb-27ac-d13e-03e246c5c88b Address:67bff599-8edb-27ac-d13e-03e246c5c88b}]" 2024-06-01T18:41:32.556Z [INFO] server-18e15b86-cfb5-1196-cb56-aaad9be54f4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a Address:2f34d9bf-6566-dfb2-3749-8a5d47e65c9a} {Suffrage:Voter ID:server-18e15b86-cfb5-1196-cb56-aaad9be54f4b Address:18e15b86-cfb5-1196-cb56-aaad9be54f4b} {Suffrage:Voter ID:server-67bff599-8edb-27ac-d13e-03e246c5c88b Address:67bff599-8edb-27ac-d13e-03e246c5c88b}]" 2024-06-01T18:41:32.556Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a Address:2f34d9bf-6566-dfb2-3749-8a5d47e65c9a} {Suffrage:Voter ID:server-18e15b86-cfb5-1196-cb56-aaad9be54f4b Address:18e15b86-cfb5-1196-cb56-aaad9be54f4b} {Suffrage:Voter ID:server-67bff599-8edb-27ac-d13e-03e246c5c88b Address:67bff599-8edb-27ac-d13e-03e246c5c88b}]" 2024-06-01T18:41:32.556Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:32.556Z [INFO] server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a: entering follower state: follower="Node at 2f34d9bf-6566-dfb2-3749-8a5d47e65c9a [Follower]" leader-address= leader-id= 2024-06-01T18:41:32.556Z [INFO] server-18e15b86-cfb5-1196-cb56-aaad9be54f4b: entering follower state: follower="Node at 18e15b86-cfb5-1196-cb56-aaad9be54f4b [Follower]" leader-address= leader-id= 2024-06-01T18:41:32.556Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: entering follower state: follower="Node at 67bff599-8edb-27ac-d13e-03e246c5c88b [Follower]" leader-address= leader-id= 2024-06-01T18:41:32.709Z [WARN] server-67bff599-8edb-27ac-d13e-03e246c5c88b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:32.709Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: entering candidate state: node="Node at 67bff599-8edb-27ac-d13e-03e246c5c88b [Candidate]" term=2 2024-06-01T18:41:32.709Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: election won: term=2 tally=2 2024-06-01T18:41:32.709Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: entering leader state: leader="Node at 67bff599-8edb-27ac-d13e-03e246c5c88b [Leader]" 2024-06-01T18:41:32.709Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: added peer, starting replication: peer=server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a 2024-06-01T18:41:32.709Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: added peer, starting replication: peer=server-18e15b86-cfb5-1196-cb56-aaad9be54f4b 2024-06-01T18:41:32.709Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: pipelining replication: peer="{Voter server-18e15b86-cfb5-1196-cb56-aaad9be54f4b 18e15b86-cfb5-1196-cb56-aaad9be54f4b}" 2024-06-01T18:41:32.709Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: pipelining replication: peer="{Voter server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a 2f34d9bf-6566-dfb2-3749-8a5d47e65c9a}" 2024-06-01T18:41:32.914Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:32.709557033 +0000 UTC m=+8.207746495 (1 nodes), 153.022348ms from start of poll, 153.225235ms from cluster start. Timeout at 2024-06-01 18:41:32.914759077 +0000 UTC m=+8.412948529, 205.202034ms after stability 2024-06-01T18:41:32.927Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: aborting pipeline replication: peer="{Voter server-18e15b86-cfb5-1196-cb56-aaad9be54f4b 18e15b86-cfb5-1196-cb56-aaad9be54f4b}" 2024-06-01T18:41:32.927Z [INFO] server-67bff599-8edb-27ac-d13e-03e246c5c88b: aborting pipeline replication: peer="{Voter server-2f34d9bf-6566-dfb2-3749-8a5d47e65c9a 2f34d9bf-6566-dfb2-3749-8a5d47e65c9a}" --- PASS: TestRaft_ApplyConcurrent (0.37s) === RUN TestRaft_ApplyConcurrent_Timeout 2024-06-01T18:41:32.928Z [INFO] server-677f6143-ddc5-c50b-8877-c7b46fa18dfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-677f6143-ddc5-c50b-8877-c7b46fa18dfa Address:677f6143-ddc5-c50b-8877-c7b46fa18dfa}]" 2024-06-01T18:41:32.928Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:32.928Z [INFO] server-677f6143-ddc5-c50b-8877-c7b46fa18dfa: entering follower state: follower="Node at 677f6143-ddc5-c50b-8877-c7b46fa18dfa [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.029Z [WARN] server-677f6143-ddc5-c50b-8877-c7b46fa18dfa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:33.029Z [INFO] server-677f6143-ddc5-c50b-8877-c7b46fa18dfa: entering candidate state: node="Node at 677f6143-ddc5-c50b-8877-c7b46fa18dfa [Candidate]" term=2 2024-06-01T18:41:33.029Z [INFO] server-677f6143-ddc5-c50b-8877-c7b46fa18dfa: election won: term=2 tally=1 2024-06-01T18:41:33.029Z [INFO] server-677f6143-ddc5-c50b-8877-c7b46fa18dfa: entering leader state: leader="Node at 677f6143-ddc5-c50b-8877-c7b46fa18dfa [Leader]" 2024-06-01T18:41:33.231Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:33.029204271 +0000 UTC m=+8.527393723 (1 nodes), 100.67476ms from start of poll, 100.81401ms from cluster start. Timeout at 2024-06-01 18:41:33.231092044 +0000 UTC m=+8.729281496, 201.887773ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.31s) === RUN TestRaft_JoinNode 2024-06-01T18:41:33.233Z [INFO] server-3e869622-3942-00de-73b0-4ca14b68ab82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e869622-3942-00de-73b0-4ca14b68ab82 Address:3e869622-3942-00de-73b0-4ca14b68ab82} {Suffrage:Voter ID:server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0 Address:333811b5-cbaf-9f1b-f95b-b63fe5af8ca0}]" 2024-06-01T18:41:33.233Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e869622-3942-00de-73b0-4ca14b68ab82 Address:3e869622-3942-00de-73b0-4ca14b68ab82} {Suffrage:Voter ID:server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0 Address:333811b5-cbaf-9f1b-f95b-b63fe5af8ca0}]" 2024-06-01T18:41:33.234Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: entering follower state: follower="Node at 333811b5-cbaf-9f1b-f95b-b63fe5af8ca0 [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.234Z [INFO] server-3e869622-3942-00de-73b0-4ca14b68ab82: entering follower state: follower="Node at 3e869622-3942-00de-73b0-4ca14b68ab82 [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.234Z [INFO] server-99f8ea49-b34c-83cd-90c7-c137402aa205: initial configuration: index=0 servers=[] 2024-06-01T18:41:33.234Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:33.234Z [INFO] server-99f8ea49-b34c-83cd-90c7-c137402aa205: entering follower state: follower="Node at 99f8ea49-b34c-83cd-90c7-c137402aa205 [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.328Z [WARN] server-99f8ea49-b34c-83cd-90c7-c137402aa205: no known peers, aborting election 2024-06-01T18:41:33.328Z [WARN] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:33.328Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: entering candidate state: node="Node at 333811b5-cbaf-9f1b-f95b-b63fe5af8ca0 [Candidate]" term=2 2024-06-01T18:41:33.328Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: election won: term=2 tally=2 2024-06-01T18:41:33.328Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: entering leader state: leader="Node at 333811b5-cbaf-9f1b-f95b-b63fe5af8ca0 [Leader]" 2024-06-01T18:41:33.329Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: added peer, starting replication: peer=server-3e869622-3942-00de-73b0-4ca14b68ab82 2024-06-01T18:41:33.329Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: pipelining replication: peer="{Voter server-3e869622-3942-00de-73b0-4ca14b68ab82 3e869622-3942-00de-73b0-4ca14b68ab82}" 2024-06-01T18:41:33.434Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:33.329034349 +0000 UTC m=+8.827223801 (1 nodes), 94.869625ms from start of poll, 95.203816ms from cluster start. Timeout at 2024-06-01 18:41:33.434663592 +0000 UTC m=+8.932853045, 105.629244ms after stability 2024-06-01T18:41:33.434Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: updating configuration: command=AddVoter server-id=server-99f8ea49-b34c-83cd-90c7-c137402aa205 server-addr=99f8ea49-b34c-83cd-90c7-c137402aa205 servers="[{Suffrage:Voter ID:server-3e869622-3942-00de-73b0-4ca14b68ab82 Address:3e869622-3942-00de-73b0-4ca14b68ab82} {Suffrage:Voter ID:server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0 Address:333811b5-cbaf-9f1b-f95b-b63fe5af8ca0} {Suffrage:Voter ID:server-99f8ea49-b34c-83cd-90c7-c137402aa205 Address:99f8ea49-b34c-83cd-90c7-c137402aa205}]" 2024-06-01T18:41:33.434Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: added peer, starting replication: peer=server-99f8ea49-b34c-83cd-90c7-c137402aa205 2024-06-01T18:41:33.434Z [WARN] server-99f8ea49-b34c-83cd-90c7-c137402aa205: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2024-06-01T18:41:33.434Z [WARN] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: appendEntries rejected, sending older logs: peer="{Voter server-99f8ea49-b34c-83cd-90c7-c137402aa205 99f8ea49-b34c-83cd-90c7-c137402aa205}" next=1 2024-06-01T18:41:33.435Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: pipelining replication: peer="{Voter server-99f8ea49-b34c-83cd-90c7-c137402aa205 99f8ea49-b34c-83cd-90c7-c137402aa205}" 2024-06-01T18:41:33.435Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:33.540Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:33.435069117 +0000 UTC m=+8.933258569 (1 nodes), 561ns from start of poll, 201.238584ms from cluster start. Timeout at 2024-06-01 18:41:33.540257109 +0000 UTC m=+9.038446561, 105.187992ms after stability 2024-06-01T18:41:33.540Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: aborting pipeline replication: peer="{Voter server-99f8ea49-b34c-83cd-90c7-c137402aa205 99f8ea49-b34c-83cd-90c7-c137402aa205}" 2024-06-01T18:41:33.540Z [INFO] server-333811b5-cbaf-9f1b-f95b-b63fe5af8ca0: aborting pipeline replication: peer="{Voter server-3e869622-3942-00de-73b0-4ca14b68ab82 3e869622-3942-00de-73b0-4ca14b68ab82}" --- PASS: TestRaft_JoinNode (0.31s) === RUN TestRaft_JoinNode_ConfigStore 2024-06-01T18:41:33.541Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6 Address:621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6}]" 2024-06-01T18:41:33.541Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: entering follower state: follower="Node at 621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6 [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.541Z [INFO] server-f07b2edb-4a36-40e1-d15d-cb87454010ca: initial configuration: index=0 servers=[] 2024-06-01T18:41:33.541Z [INFO] server-f07b2edb-4a36-40e1-d15d-cb87454010ca: entering follower state: follower="Node at f07b2edb-4a36-40e1-d15d-cb87454010ca [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.541Z [INFO] server-336f8be1-d419-ec60-2cb4-a68052146164: initial configuration: index=0 servers=[] 2024-06-01T18:41:33.541Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:33.541Z [INFO] server-336f8be1-d419-ec60-2cb4-a68052146164: entering follower state: follower="Node at 336f8be1-d419-ec60-2cb4-a68052146164 [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.626Z [WARN] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:33.626Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: entering candidate state: node="Node at 621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6 [Candidate]" term=2 2024-06-01T18:41:33.626Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: election won: term=2 tally=1 2024-06-01T18:41:33.626Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: entering leader state: leader="Node at 621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6 [Leader]" 2024-06-01T18:41:33.633Z [WARN] server-f07b2edb-4a36-40e1-d15d-cb87454010ca: no known peers, aborting election 2024-06-01T18:41:33.640Z [WARN] server-336f8be1-d419-ec60-2cb4-a68052146164: no known peers, aborting election 2024-06-01T18:41:33.732Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:33.627022387 +0000 UTC m=+9.125211839 (1 nodes), 85.546429ms from start of poll, 85.99839ms from cluster start. Timeout at 2024-06-01 18:41:33.732640419 +0000 UTC m=+9.230829872, 105.618033ms after stability 2024-06-01T18:41:33.732Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: updating configuration: command=AddVoter server-id=server-f07b2edb-4a36-40e1-d15d-cb87454010ca server-addr=f07b2edb-4a36-40e1-d15d-cb87454010ca servers="[{Suffrage:Voter ID:server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6 Address:621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6} {Suffrage:Voter ID:server-f07b2edb-4a36-40e1-d15d-cb87454010ca Address:f07b2edb-4a36-40e1-d15d-cb87454010ca}]" 2024-06-01T18:41:33.732Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: added peer, starting replication: peer=server-f07b2edb-4a36-40e1-d15d-cb87454010ca 2024-06-01T18:41:33.732Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:33.732Z [WARN] server-f07b2edb-4a36-40e1-d15d-cb87454010ca: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2024-06-01T18:41:33.733Z [WARN] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: appendEntries rejected, sending older logs: peer="{Voter server-f07b2edb-4a36-40e1-d15d-cb87454010ca f07b2edb-4a36-40e1-d15d-cb87454010ca}" next=1 2024-06-01T18:41:33.733Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: pipelining replication: peer="{Voter server-f07b2edb-4a36-40e1-d15d-cb87454010ca f07b2edb-4a36-40e1-d15d-cb87454010ca}" 2024-06-01T18:41:33.838Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:33.732919019 +0000 UTC m=+9.231108471 (1 nodes), 822ns from start of poll, 191.895022ms from cluster start. Timeout at 2024-06-01 18:41:33.838480426 +0000 UTC m=+9.336669868, 105.561397ms after stability 2024-06-01T18:41:33.838Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: updating configuration: command=AddVoter server-id=server-336f8be1-d419-ec60-2cb4-a68052146164 server-addr=336f8be1-d419-ec60-2cb4-a68052146164 servers="[{Suffrage:Voter ID:server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6 Address:621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6} {Suffrage:Voter ID:server-f07b2edb-4a36-40e1-d15d-cb87454010ca Address:f07b2edb-4a36-40e1-d15d-cb87454010ca} {Suffrage:Voter ID:server-336f8be1-d419-ec60-2cb4-a68052146164 Address:336f8be1-d419-ec60-2cb4-a68052146164}]" 2024-06-01T18:41:33.838Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: added peer, starting replication: peer=server-336f8be1-d419-ec60-2cb4-a68052146164 2024-06-01T18:41:33.838Z [WARN] server-336f8be1-d419-ec60-2cb4-a68052146164: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2024-06-01T18:41:33.838Z [WARN] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: appendEntries rejected, sending older logs: peer="{Voter server-336f8be1-d419-ec60-2cb4-a68052146164 336f8be1-d419-ec60-2cb4-a68052146164}" next=1 2024-06-01T18:41:33.838Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: pipelining replication: peer="{Voter server-336f8be1-d419-ec60-2cb4-a68052146164 336f8be1-d419-ec60-2cb4-a68052146164}" 2024-06-01T18:41:33.838Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:33.944Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:33.838842921 +0000 UTC m=+9.337032373 (1 nodes), 1.112µs from start of poll, 297.818924ms from cluster start. Timeout at 2024-06-01 18:41:33.944239491 +0000 UTC m=+9.442428943, 105.39657ms after stability 2024-06-01T18:41:33.944Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: aborting pipeline replication: peer="{Voter server-336f8be1-d419-ec60-2cb4-a68052146164 336f8be1-d419-ec60-2cb4-a68052146164}" 2024-06-01T18:41:33.944Z [INFO] server-621d5a8c-22f3-24bf-4dc1-8ac5f960d4e6: aborting pipeline replication: peer="{Voter server-f07b2edb-4a36-40e1-d15d-cb87454010ca f07b2edb-4a36-40e1-d15d-cb87454010ca}" --- PASS: TestRaft_JoinNode_ConfigStore (0.40s) === RUN TestRaft_RemoveFollower 2024-06-01T18:41:33.945Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6fa45de-69d5-a940-e9c1-1523bdc26588 Address:f6fa45de-69d5-a940-e9c1-1523bdc26588} {Suffrage:Voter ID:server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 Address:d91e987d-a56f-1e14-4162-fb8b58fcaaa7} {Suffrage:Voter ID:server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d Address:b2c6a3f8-d94b-882e-3f17-19afda1cac7d}]" 2024-06-01T18:41:33.945Z [INFO] server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6fa45de-69d5-a940-e9c1-1523bdc26588 Address:f6fa45de-69d5-a940-e9c1-1523bdc26588} {Suffrage:Voter ID:server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 Address:d91e987d-a56f-1e14-4162-fb8b58fcaaa7} {Suffrage:Voter ID:server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d Address:b2c6a3f8-d94b-882e-3f17-19afda1cac7d}]" 2024-06-01T18:41:33.945Z [INFO] server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6fa45de-69d5-a940-e9c1-1523bdc26588 Address:f6fa45de-69d5-a940-e9c1-1523bdc26588} {Suffrage:Voter ID:server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 Address:d91e987d-a56f-1e14-4162-fb8b58fcaaa7} {Suffrage:Voter ID:server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d Address:b2c6a3f8-d94b-882e-3f17-19afda1cac7d}]" 2024-06-01T18:41:33.945Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:33.945Z [INFO] server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7: entering follower state: follower="Node at d91e987d-a56f-1e14-4162-fb8b58fcaaa7 [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.945Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: entering follower state: follower="Node at f6fa45de-69d5-a940-e9c1-1523bdc26588 [Follower]" leader-address= leader-id= 2024-06-01T18:41:33.945Z [INFO] server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d: entering follower state: follower="Node at b2c6a3f8-d94b-882e-3f17-19afda1cac7d [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.002Z [WARN] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:34.002Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: entering candidate state: node="Node at f6fa45de-69d5-a940-e9c1-1523bdc26588 [Candidate]" term=2 2024-06-01T18:41:34.002Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: election won: term=2 tally=2 2024-06-01T18:41:34.002Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: entering leader state: leader="Node at f6fa45de-69d5-a940-e9c1-1523bdc26588 [Leader]" 2024-06-01T18:41:34.002Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: added peer, starting replication: peer=server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 2024-06-01T18:41:34.002Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: added peer, starting replication: peer=server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d 2024-06-01T18:41:34.002Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: pipelining replication: peer="{Voter server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d b2c6a3f8-d94b-882e-3f17-19afda1cac7d}" 2024-06-01T18:41:34.003Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: pipelining replication: peer="{Voter server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 d91e987d-a56f-1e14-4162-fb8b58fcaaa7}" 2024-06-01T18:41:34.108Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:34.003064937 +0000 UTC m=+9.501254389 (1 nodes), 57.649567ms from start of poll, 57.869456ms from cluster start. Timeout at 2024-06-01 18:41:34.108789819 +0000 UTC m=+9.606979271, 105.724882ms after stability 2024-06-01T18:41:34.114Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:34.220Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:34.114549991 +0000 UTC m=+9.612739443 (2 nodes), 1.873µs from start of poll, 169.35451ms from cluster start. Timeout at 2024-06-01 18:41:34.220284151 +0000 UTC m=+9.718473603, 105.73416ms after stability 2024-06-01T18:41:34.220Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: updating configuration: command=RemoveServer server-id=server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 server-addr= servers="[{Suffrage:Voter ID:server-f6fa45de-69d5-a940-e9c1-1523bdc26588 Address:f6fa45de-69d5-a940-e9c1-1523bdc26588} {Suffrage:Voter ID:server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d Address:b2c6a3f8-d94b-882e-3f17-19afda1cac7d}]" 2024-06-01T18:41:34.220Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: removed peer, stopping replication: peer=server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 last-index=3 2024-06-01T18:41:34.220Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: aborting pipeline replication: peer="{Voter server-d91e987d-a56f-1e14-4162-fb8b58fcaaa7 d91e987d-a56f-1e14-4162-fb8b58fcaaa7}" 2024-06-01T18:41:34.325Z [INFO] server-f6fa45de-69d5-a940-e9c1-1523bdc26588: aborting pipeline replication: peer="{Voter server-b2c6a3f8-d94b-882e-3f17-19afda1cac7d b2c6a3f8-d94b-882e-3f17-19afda1cac7d}" --- PASS: TestRaft_RemoveFollower (0.38s) === RUN TestRaft_RemoveLeader 2024-06-01T18:41:34.326Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8b4fd43-8135-a329-eddf-2a9822fa66a0 Address:f8b4fd43-8135-a329-eddf-2a9822fa66a0} {Suffrage:Voter ID:server-09fa6737-3c9e-948e-2f17-6d549016030b Address:09fa6737-3c9e-948e-2f17-6d549016030b} {Suffrage:Voter ID:server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 Address:cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}]" 2024-06-01T18:41:34.326Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8b4fd43-8135-a329-eddf-2a9822fa66a0 Address:f8b4fd43-8135-a329-eddf-2a9822fa66a0} {Suffrage:Voter ID:server-09fa6737-3c9e-948e-2f17-6d549016030b Address:09fa6737-3c9e-948e-2f17-6d549016030b} {Suffrage:Voter ID:server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 Address:cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}]" 2024-06-01T18:41:34.326Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: entering follower state: follower="Node at f8b4fd43-8135-a329-eddf-2a9822fa66a0 [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.326Z [INFO] server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8b4fd43-8135-a329-eddf-2a9822fa66a0 Address:f8b4fd43-8135-a329-eddf-2a9822fa66a0} {Suffrage:Voter ID:server-09fa6737-3c9e-948e-2f17-6d549016030b Address:09fa6737-3c9e-948e-2f17-6d549016030b} {Suffrage:Voter ID:server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 Address:cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}]" 2024-06-01T18:41:34.326Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:34.326Z [INFO] server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34: entering follower state: follower="Node at cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.326Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: entering follower state: follower="Node at 09fa6737-3c9e-948e-2f17-6d549016030b [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.380Z [WARN] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:34.380Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: entering candidate state: node="Node at f8b4fd43-8135-a329-eddf-2a9822fa66a0 [Candidate]" term=2 2024-06-01T18:41:34.380Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: election won: term=2 tally=2 2024-06-01T18:41:34.380Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: entering leader state: leader="Node at f8b4fd43-8135-a329-eddf-2a9822fa66a0 [Leader]" 2024-06-01T18:41:34.380Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: added peer, starting replication: peer=server-09fa6737-3c9e-948e-2f17-6d549016030b 2024-06-01T18:41:34.380Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: added peer, starting replication: peer=server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 2024-06-01T18:41:34.380Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: pipelining replication: peer="{Voter server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}" 2024-06-01T18:41:34.380Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: pipelining replication: peer="{Voter server-09fa6737-3c9e-948e-2f17-6d549016030b 09fa6737-3c9e-948e-2f17-6d549016030b}" 2024-06-01T18:41:34.486Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:34.380353468 +0000 UTC m=+9.878542920 (1 nodes), 53.4338ms from start of poll, 53.674528ms from cluster start. Timeout at 2024-06-01 18:41:34.486022094 +0000 UTC m=+9.984211546, 105.668626ms after stability 2024-06-01T18:41:34.491Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:34.597Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:34.491540808 +0000 UTC m=+9.989730260 (2 nodes), 1.233µs from start of poll, 164.861868ms from cluster start. Timeout at 2024-06-01 18:41:34.597210286 +0000 UTC m=+10.095399739, 105.669479ms after stability 2024-06-01T18:41:34.597Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: updating configuration: command=RemoveServer server-id=server-f8b4fd43-8135-a329-eddf-2a9822fa66a0 server-addr= servers="[{Suffrage:Voter ID:server-09fa6737-3c9e-948e-2f17-6d549016030b Address:09fa6737-3c9e-948e-2f17-6d549016030b} {Suffrage:Voter ID:server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 Address:cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}]" 2024-06-01T18:41:34.597Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: removed ourself, shutting down 2024-06-01T18:41:34.597Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: aborting pipeline replication: peer="{Voter server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}" 2024-06-01T18:41:34.597Z [INFO] server-f8b4fd43-8135-a329-eddf-2a9822fa66a0: aborting pipeline replication: peer="{Voter server-09fa6737-3c9e-948e-2f17-6d549016030b 09fa6737-3c9e-948e-2f17-6d549016030b}" 2024-06-01T18:41:34.657Z [WARN] server-09fa6737-3c9e-948e-2f17-6d549016030b: heartbeat timeout reached, starting election: last-leader-addr=f8b4fd43-8135-a329-eddf-2a9822fa66a0 last-leader-id=server-f8b4fd43-8135-a329-eddf-2a9822fa66a0 2024-06-01T18:41:34.657Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: entering candidate state: node="Node at 09fa6737-3c9e-948e-2f17-6d549016030b [Candidate]" term=3 2024-06-01T18:41:34.657Z [WARN] server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34: rejecting vote request since we have a leader: from=09fa6737-3c9e-948e-2f17-6d549016030b leader=f8b4fd43-8135-a329-eddf-2a9822fa66a0 leader-id=server-f8b4fd43-8135-a329-eddf-2a9822fa66a0 2024-06-01T18:41:34.675Z [WARN] server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34: heartbeat timeout reached, starting election: last-leader-addr=f8b4fd43-8135-a329-eddf-2a9822fa66a0 last-leader-id=server-f8b4fd43-8135-a329-eddf-2a9822fa66a0 2024-06-01T18:41:34.675Z [INFO] server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34: entering candidate state: node="Node at cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 [Candidate]" term=3 2024-06-01T18:41:34.676Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: duplicate requestVote for same term: term=3 2024-06-01T18:41:34.731Z [WARN] server-09fa6737-3c9e-948e-2f17-6d549016030b: Election timeout reached, restarting election 2024-06-01T18:41:34.731Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: entering candidate state: node="Node at 09fa6737-3c9e-948e-2f17-6d549016030b [Candidate]" term=4 2024-06-01T18:41:34.731Z [INFO] server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34: entering follower state: follower="Node at cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.731Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: election won: term=4 tally=2 2024-06-01T18:41:34.731Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: entering leader state: leader="Node at 09fa6737-3c9e-948e-2f17-6d549016030b [Leader]" 2024-06-01T18:41:34.731Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: added peer, starting replication: peer=server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 2024-06-01T18:41:34.731Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: pipelining replication: peer="{Voter server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}" 2024-06-01T18:41:34.808Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:34.913Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:34.808422528 +0000 UTC m=+10.306611980 (1 nodes), 27.171µs from start of poll, 481.743588ms from cluster start. Timeout at 2024-06-01 18:41:34.913683495 +0000 UTC m=+10.411872947, 105.260967ms after stability 2024-06-01T18:41:34.913Z [INFO] server-09fa6737-3c9e-948e-2f17-6d549016030b: aborting pipeline replication: peer="{Voter server-cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34 cc10fb0a-cbd5-4f8f-be1b-0d8d0e6dcc34}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown 2024-06-01T18:41:34.917Z [INFO] server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 Address:da4b25ca-54cb-16a3-0a5e-15a79fa3a187} {Suffrage:Voter ID:server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 Address:89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9} {Suffrage:Voter ID:server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 Address:dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8}]" 2024-06-01T18:41:34.917Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 Address:da4b25ca-54cb-16a3-0a5e-15a79fa3a187} {Suffrage:Voter ID:server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 Address:89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9} {Suffrage:Voter ID:server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 Address:dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8}]" 2024-06-01T18:41:34.917Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 Address:da4b25ca-54cb-16a3-0a5e-15a79fa3a187} {Suffrage:Voter ID:server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 Address:89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9} {Suffrage:Voter ID:server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 Address:dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8}]" 2024-06-01T18:41:34.917Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:34.917Z [INFO] server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187: entering follower state: follower="Node at da4b25ca-54cb-16a3-0a5e-15a79fa3a187 [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.917Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: entering follower state: follower="Node at 89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.917Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: entering follower state: follower="Node at dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 [Follower]" leader-address= leader-id= 2024-06-01T18:41:34.985Z [WARN] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:34.985Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: entering candidate state: node="Node at 89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 [Candidate]" term=2 2024-06-01T18:41:34.985Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: election won: term=2 tally=2 2024-06-01T18:41:34.985Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: entering leader state: leader="Node at 89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 [Leader]" 2024-06-01T18:41:34.985Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: added peer, starting replication: peer=server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 2024-06-01T18:41:34.985Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: added peer, starting replication: peer=server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 2024-06-01T18:41:34.985Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: pipelining replication: peer="{Voter server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8}" 2024-06-01T18:41:34.986Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: pipelining replication: peer="{Voter server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 da4b25ca-54cb-16a3-0a5e-15a79fa3a187}" 2024-06-01T18:41:35.092Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:34.986143864 +0000 UTC m=+10.484333316 (2 nodes), 68.832466ms from start of poll, 69.076449ms from cluster start. Timeout at 2024-06-01 18:41:35.092148205 +0000 UTC m=+10.590337657, 106.004341ms after stability 2024-06-01T18:41:35.092Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:35.197Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:35.092227453 +0000 UTC m=+10.590416905 (1 nodes), 1.713µs from start of poll, 175.160038ms from cluster start. Timeout at 2024-06-01 18:41:35.197528977 +0000 UTC m=+10.695718429, 105.301524ms after stability 2024-06-01T18:41:35.200Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: updating configuration: command=RemoveServer server-id=server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 server-addr= servers="[{Suffrage:Voter ID:server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 Address:da4b25ca-54cb-16a3-0a5e-15a79fa3a187} {Suffrage:Voter ID:server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 Address:dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8}]" 2024-06-01T18:41:35.200Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: removed ourself, transitioning to follower 2024-06-01T18:41:35.200Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: entering follower state: follower="Node at 89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.200Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: aborting pipeline replication: peer="{Voter server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 da4b25ca-54cb-16a3-0a5e-15a79fa3a187}" 2024-06-01T18:41:35.200Z [INFO] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: aborting pipeline replication: peer="{Voter server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8}" 2024-06-01T18:41:35.269Z [WARN] server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187: heartbeat timeout reached, starting election: last-leader-addr=89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 last-leader-id=server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 2024-06-01T18:41:35.269Z [INFO] server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187: entering candidate state: node="Node at da4b25ca-54cb-16a3-0a5e-15a79fa3a187 [Candidate]" term=3 2024-06-01T18:41:35.269Z [WARN] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: rejecting vote request since we have a leader: from=da4b25ca-54cb-16a3-0a5e-15a79fa3a187 leader=89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 leader-id=server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 2024-06-01T18:41:35.276Z [WARN] server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9: not part of stable configuration, aborting election 2024-06-01T18:41:35.277Z [WARN] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: heartbeat timeout reached, starting election: last-leader-addr=89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 last-leader-id=server-89cf4bc1-7cea-9a89-848d-7c7cf7fb43e9 2024-06-01T18:41:35.277Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: entering candidate state: node="Node at dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 [Candidate]" term=3 2024-06-01T18:41:35.277Z [INFO] server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187: duplicate requestVote for same term: term=3 2024-06-01T18:41:35.306Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:35.338Z [WARN] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: Election timeout reached, restarting election 2024-06-01T18:41:35.338Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: entering candidate state: node="Node at dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 [Candidate]" term=4 2024-06-01T18:41:35.338Z [INFO] server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187: entering follower state: follower="Node at da4b25ca-54cb-16a3-0a5e-15a79fa3a187 [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.338Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: election won: term=4 tally=2 2024-06-01T18:41:35.338Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: entering leader state: leader="Node at dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8 [Leader]" 2024-06-01T18:41:35.338Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: added peer, starting replication: peer=server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 2024-06-01T18:41:35.338Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: pipelining replication: peer="{Voter server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 da4b25ca-54cb-16a3-0a5e-15a79fa3a187}" 2024-06-01T18:41:35.444Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:35.338631615 +0000 UTC m=+10.836821067 (1 nodes), 32.50047ms from start of poll, 421.5642ms from cluster start. Timeout at 2024-06-01 18:41:35.444282099 +0000 UTC m=+10.942471551, 105.650484ms after stability 2024-06-01T18:41:35.550Z [INFO] server-dc47ca2d-cf86-7ac3-aef2-97d4fd8cb6e8: aborting pipeline replication: peer="{Voter server-da4b25ca-54cb-16a3-0a5e-15a79fa3a187 da4b25ca-54cb-16a3-0a5e-15a79fa3a187}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s) === RUN TestRaft_RemoveFollower_SplitCluster 2024-06-01T18:41:35.551Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8836a9b9-20ff-7942-1b3c-7109e156c609 Address:8836a9b9-20ff-7942-1b3c-7109e156c609} {Suffrage:Voter ID:server-94cec290-aa3a-be42-27b5-fb71182d4023 Address:94cec290-aa3a-be42-27b5-fb71182d4023} {Suffrage:Voter ID:server-ca4677be-981c-08a5-8027-799da4732c6f Address:ca4677be-981c-08a5-8027-799da4732c6f} {Suffrage:Voter ID:server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 Address:353844a2-1bce-8d9a-bc2b-6b4c51c15b22}]" 2024-06-01T18:41:35.551Z [INFO] server-94cec290-aa3a-be42-27b5-fb71182d4023: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8836a9b9-20ff-7942-1b3c-7109e156c609 Address:8836a9b9-20ff-7942-1b3c-7109e156c609} {Suffrage:Voter ID:server-94cec290-aa3a-be42-27b5-fb71182d4023 Address:94cec290-aa3a-be42-27b5-fb71182d4023} {Suffrage:Voter ID:server-ca4677be-981c-08a5-8027-799da4732c6f Address:ca4677be-981c-08a5-8027-799da4732c6f} {Suffrage:Voter ID:server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 Address:353844a2-1bce-8d9a-bc2b-6b4c51c15b22}]" 2024-06-01T18:41:35.551Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: entering follower state: follower="Node at 8836a9b9-20ff-7942-1b3c-7109e156c609 [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.551Z [INFO] server-ca4677be-981c-08a5-8027-799da4732c6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8836a9b9-20ff-7942-1b3c-7109e156c609 Address:8836a9b9-20ff-7942-1b3c-7109e156c609} {Suffrage:Voter ID:server-94cec290-aa3a-be42-27b5-fb71182d4023 Address:94cec290-aa3a-be42-27b5-fb71182d4023} {Suffrage:Voter ID:server-ca4677be-981c-08a5-8027-799da4732c6f Address:ca4677be-981c-08a5-8027-799da4732c6f} {Suffrage:Voter ID:server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 Address:353844a2-1bce-8d9a-bc2b-6b4c51c15b22}]" 2024-06-01T18:41:35.551Z [INFO] server-94cec290-aa3a-be42-27b5-fb71182d4023: entering follower state: follower="Node at 94cec290-aa3a-be42-27b5-fb71182d4023 [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.551Z [INFO] server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8836a9b9-20ff-7942-1b3c-7109e156c609 Address:8836a9b9-20ff-7942-1b3c-7109e156c609} {Suffrage:Voter ID:server-94cec290-aa3a-be42-27b5-fb71182d4023 Address:94cec290-aa3a-be42-27b5-fb71182d4023} {Suffrage:Voter ID:server-ca4677be-981c-08a5-8027-799da4732c6f Address:ca4677be-981c-08a5-8027-799da4732c6f} {Suffrage:Voter ID:server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 Address:353844a2-1bce-8d9a-bc2b-6b4c51c15b22}]" 2024-06-01T18:41:35.551Z [INFO] server-ca4677be-981c-08a5-8027-799da4732c6f: entering follower state: follower="Node at ca4677be-981c-08a5-8027-799da4732c6f [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.551Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:35.551Z [INFO] server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22: entering follower state: follower="Node at 353844a2-1bce-8d9a-bc2b-6b4c51c15b22 [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.602Z [WARN] server-8836a9b9-20ff-7942-1b3c-7109e156c609: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: entering candidate state: node="Node at 8836a9b9-20ff-7942-1b3c-7109e156c609 [Candidate]" term=2 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: election won: term=2 tally=3 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: entering leader state: leader="Node at 8836a9b9-20ff-7942-1b3c-7109e156c609 [Leader]" 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: added peer, starting replication: peer=server-94cec290-aa3a-be42-27b5-fb71182d4023 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: added peer, starting replication: peer=server-ca4677be-981c-08a5-8027-799da4732c6f 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: added peer, starting replication: peer=server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: pipelining replication: peer="{Voter server-94cec290-aa3a-be42-27b5-fb71182d4023 94cec290-aa3a-be42-27b5-fb71182d4023}" 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: pipelining replication: peer="{Voter server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 353844a2-1bce-8d9a-bc2b-6b4c51c15b22}" 2024-06-01T18:41:35.602Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: pipelining replication: peer="{Voter server-ca4677be-981c-08a5-8027-799da4732c6f ca4677be-981c-08a5-8027-799da4732c6f}" 2024-06-01T18:41:35.707Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:35.602251195 +0000 UTC m=+11.100440647 (1 nodes), 50.830954ms from start of poll, 51.139267ms from cluster start. Timeout at 2024-06-01 18:41:35.707458213 +0000 UTC m=+11.205647665, 105.207018ms after stability 2024-06-01T18:41:35.813Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:35.918Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:35.81361031 +0000 UTC m=+11.311799762 (3 nodes), 2.274µs from start of poll, 262.498382ms from cluster start. Timeout at 2024-06-01 18:41:35.918873542 +0000 UTC m=+11.417062994, 105.263232ms after stability 2024-06-01T18:41:35.918Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: updating configuration: command=RemoveServer server-id=server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 server-addr= servers="[{Suffrage:Voter ID:server-8836a9b9-20ff-7942-1b3c-7109e156c609 Address:8836a9b9-20ff-7942-1b3c-7109e156c609} {Suffrage:Voter ID:server-94cec290-aa3a-be42-27b5-fb71182d4023 Address:94cec290-aa3a-be42-27b5-fb71182d4023} {Suffrage:Voter ID:server-ca4677be-981c-08a5-8027-799da4732c6f Address:ca4677be-981c-08a5-8027-799da4732c6f}]" 2024-06-01T18:41:35.919Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: removed peer, stopping replication: peer=server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 last-index=3 2024-06-01T18:41:35.919Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to pipeline appendEntries: peer="{Voter server-94cec290-aa3a-be42-27b5-fb71182d4023 94cec290-aa3a-be42-27b5-fb71182d4023}" error="append pipeline closed" 2024-06-01T18:41:35.919Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to pipeline appendEntries: peer="{Voter server-ca4677be-981c-08a5-8027-799da4732c6f ca4677be-981c-08a5-8027-799da4732c6f}" error="append pipeline closed" 2024-06-01T18:41:35.919Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: aborting pipeline replication: peer="{Voter server-ca4677be-981c-08a5-8027-799da4732c6f ca4677be-981c-08a5-8027-799da4732c6f}" 2024-06-01T18:41:35.919Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: aborting pipeline replication: peer="{Voter server-353844a2-1bce-8d9a-bc2b-6b4c51c15b22 353844a2-1bce-8d9a-bc2b-6b4c51c15b22}" 2024-06-01T18:41:35.919Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: aborting pipeline replication: peer="{Voter server-94cec290-aa3a-be42-27b5-fb71182d4023 94cec290-aa3a-be42-27b5-fb71182d4023}" 2024-06-01T18:41:35.921Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to heartbeat to: peer=94cec290-aa3a-be42-27b5-fb71182d4023 backoff time=10ms error="failed to connect to peer: 94cec290-aa3a-be42-27b5-fb71182d4023" 2024-06-01T18:41:35.924Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to heartbeat to: peer=ca4677be-981c-08a5-8027-799da4732c6f backoff time=10ms error="failed to connect to peer: ca4677be-981c-08a5-8027-799da4732c6f" 2024-06-01T18:41:35.928Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to appendEntries to: peer="{Voter server-ca4677be-981c-08a5-8027-799da4732c6f ca4677be-981c-08a5-8027-799da4732c6f}" error="failed to connect to peer: ca4677be-981c-08a5-8027-799da4732c6f" 2024-06-01T18:41:35.929Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to appendEntries to: peer="{Voter server-94cec290-aa3a-be42-27b5-fb71182d4023 94cec290-aa3a-be42-27b5-fb71182d4023}" error="failed to connect to peer: 94cec290-aa3a-be42-27b5-fb71182d4023" 2024-06-01T18:41:35.940Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to heartbeat to: peer=94cec290-aa3a-be42-27b5-fb71182d4023 backoff time=10ms error="failed to connect to peer: 94cec290-aa3a-be42-27b5-fb71182d4023" 2024-06-01T18:41:35.943Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to heartbeat to: peer=ca4677be-981c-08a5-8027-799da4732c6f backoff time=10ms error="failed to connect to peer: ca4677be-981c-08a5-8027-799da4732c6f" 2024-06-01T18:41:35.947Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to appendEntries to: peer="{Voter server-ca4677be-981c-08a5-8027-799da4732c6f ca4677be-981c-08a5-8027-799da4732c6f}" error="failed to connect to peer: ca4677be-981c-08a5-8027-799da4732c6f" 2024-06-01T18:41:35.948Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to appendEntries to: peer="{Voter server-94cec290-aa3a-be42-27b5-fb71182d4023 94cec290-aa3a-be42-27b5-fb71182d4023}" error="failed to connect to peer: 94cec290-aa3a-be42-27b5-fb71182d4023" 2024-06-01T18:41:35.961Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to heartbeat to: peer=94cec290-aa3a-be42-27b5-fb71182d4023 backoff time=10ms error="failed to connect to peer: 94cec290-aa3a-be42-27b5-fb71182d4023" 2024-06-01T18:41:35.963Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to appendEntries to: peer="{Voter server-ca4677be-981c-08a5-8027-799da4732c6f ca4677be-981c-08a5-8027-799da4732c6f}" error="failed to connect to peer: ca4677be-981c-08a5-8027-799da4732c6f" 2024-06-01T18:41:35.963Z [WARN] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to contact: server-id=server-94cec290-aa3a-be42-27b5-fb71182d4023 time=50.032689ms 2024-06-01T18:41:35.964Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to heartbeat to: peer=ca4677be-981c-08a5-8027-799da4732c6f backoff time=10ms error="failed to connect to peer: ca4677be-981c-08a5-8027-799da4732c6f" 2024-06-01T18:41:35.966Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to appendEntries to: peer="{Voter server-94cec290-aa3a-be42-27b5-fb71182d4023 94cec290-aa3a-be42-27b5-fb71182d4023}" error="failed to connect to peer: 94cec290-aa3a-be42-27b5-fb71182d4023" 2024-06-01T18:41:35.974Z [WARN] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to contact: server-id=server-94cec290-aa3a-be42-27b5-fb71182d4023 time=60.870634ms 2024-06-01T18:41:35.974Z [WARN] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to contact: server-id=server-ca4677be-981c-08a5-8027-799da4732c6f time=56.453851ms 2024-06-01T18:41:35.974Z [WARN] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to contact quorum of nodes, stepping down 2024-06-01T18:41:35.974Z [INFO] server-8836a9b9-20ff-7942-1b3c-7109e156c609: entering follower state: follower="Node at 8836a9b9-20ff-7942-1b3c-7109e156c609 [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.974Z [ERROR] server-8836a9b9-20ff-7942-1b3c-7109e156c609: failed to appendEntries to: peer="{Voter server-ca4677be-981c-08a5-8027-799da4732c6f ca4677be-981c-08a5-8027-799da4732c6f}" error="failed to connect to peer: ca4677be-981c-08a5-8027-799da4732c6f" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.42s) === RUN TestRaft_AddKnownPeer 2024-06-01T18:41:35.975Z [INFO] server-cefe9104-1387-9d33-862f-5d439e97d460: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cefe9104-1387-9d33-862f-5d439e97d460 Address:cefe9104-1387-9d33-862f-5d439e97d460} {Suffrage:Voter ID:server-055d584c-cacb-3d00-3c39-820b6fe60b5c Address:055d584c-cacb-3d00-3c39-820b6fe60b5c} {Suffrage:Voter ID:server-0acfa2f9-d3b7-06ce-6816-9effff529cbb Address:0acfa2f9-d3b7-06ce-6816-9effff529cbb}]" 2024-06-01T18:41:35.975Z [INFO] server-055d584c-cacb-3d00-3c39-820b6fe60b5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cefe9104-1387-9d33-862f-5d439e97d460 Address:cefe9104-1387-9d33-862f-5d439e97d460} {Suffrage:Voter ID:server-055d584c-cacb-3d00-3c39-820b6fe60b5c Address:055d584c-cacb-3d00-3c39-820b6fe60b5c} {Suffrage:Voter ID:server-0acfa2f9-d3b7-06ce-6816-9effff529cbb Address:0acfa2f9-d3b7-06ce-6816-9effff529cbb}]" 2024-06-01T18:41:35.975Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cefe9104-1387-9d33-862f-5d439e97d460 Address:cefe9104-1387-9d33-862f-5d439e97d460} {Suffrage:Voter ID:server-055d584c-cacb-3d00-3c39-820b6fe60b5c Address:055d584c-cacb-3d00-3c39-820b6fe60b5c} {Suffrage:Voter ID:server-0acfa2f9-d3b7-06ce-6816-9effff529cbb Address:0acfa2f9-d3b7-06ce-6816-9effff529cbb}]" 2024-06-01T18:41:35.975Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:35.975Z [INFO] server-cefe9104-1387-9d33-862f-5d439e97d460: entering follower state: follower="Node at cefe9104-1387-9d33-862f-5d439e97d460 [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.975Z [INFO] server-055d584c-cacb-3d00-3c39-820b6fe60b5c: entering follower state: follower="Node at 055d584c-cacb-3d00-3c39-820b6fe60b5c [Follower]" leader-address= leader-id= 2024-06-01T18:41:35.975Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: entering follower state: follower="Node at 0acfa2f9-d3b7-06ce-6816-9effff529cbb [Follower]" leader-address= leader-id= 2024-06-01T18:41:36.031Z [WARN] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:36.031Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: entering candidate state: node="Node at 0acfa2f9-d3b7-06ce-6816-9effff529cbb [Candidate]" term=2 2024-06-01T18:41:36.031Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: election won: term=2 tally=2 2024-06-01T18:41:36.031Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: entering leader state: leader="Node at 0acfa2f9-d3b7-06ce-6816-9effff529cbb [Leader]" 2024-06-01T18:41:36.031Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: added peer, starting replication: peer=server-cefe9104-1387-9d33-862f-5d439e97d460 2024-06-01T18:41:36.031Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: added peer, starting replication: peer=server-055d584c-cacb-3d00-3c39-820b6fe60b5c 2024-06-01T18:41:36.031Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: pipelining replication: peer="{Voter server-cefe9104-1387-9d33-862f-5d439e97d460 cefe9104-1387-9d33-862f-5d439e97d460}" 2024-06-01T18:41:36.031Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: pipelining replication: peer="{Voter server-055d584c-cacb-3d00-3c39-820b6fe60b5c 055d584c-cacb-3d00-3c39-820b6fe60b5c}" 2024-06-01T18:41:36.137Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:36.031716895 +0000 UTC m=+11.529906347 (1 nodes), 56.140357ms from start of poll, 56.398888ms from cluster start. Timeout at 2024-06-01 18:41:36.13723898 +0000 UTC m=+11.635428432, 105.522085ms after stability 2024-06-01T18:41:36.137Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:36.242Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:36.137284434 +0000 UTC m=+11.635473886 (2 nodes), 812ns from start of poll, 161.966427ms from cluster start. Timeout at 2024-06-01 18:41:36.242910331 +0000 UTC m=+11.741099793, 105.625907ms after stability 2024-06-01T18:41:36.243Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: updating configuration: command=AddVoter server-id=server-cefe9104-1387-9d33-862f-5d439e97d460 server-addr=cefe9104-1387-9d33-862f-5d439e97d460 servers="[{Suffrage:Voter ID:server-cefe9104-1387-9d33-862f-5d439e97d460 Address:cefe9104-1387-9d33-862f-5d439e97d460} {Suffrage:Voter ID:server-055d584c-cacb-3d00-3c39-820b6fe60b5c Address:055d584c-cacb-3d00-3c39-820b6fe60b5c} {Suffrage:Voter ID:server-0acfa2f9-d3b7-06ce-6816-9effff529cbb Address:0acfa2f9-d3b7-06ce-6816-9effff529cbb}]" 2024-06-01T18:41:36.243Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: aborting pipeline replication: peer="{Voter server-055d584c-cacb-3d00-3c39-820b6fe60b5c 055d584c-cacb-3d00-3c39-820b6fe60b5c}" 2024-06-01T18:41:36.243Z [INFO] server-0acfa2f9-d3b7-06ce-6816-9effff529cbb: aborting pipeline replication: peer="{Voter server-cefe9104-1387-9d33-862f-5d439e97d460 cefe9104-1387-9d33-862f-5d439e97d460}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer 2024-06-01T18:41:36.244Z [INFO] server-cbce9701-7024-f4b7-9132-7b52611e05fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbce9701-7024-f4b7-9132-7b52611e05fe Address:cbce9701-7024-f4b7-9132-7b52611e05fe} {Suffrage:Voter ID:server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1 Address:ebb2c59a-efc0-97a2-5bec-31ed085b99f1} {Suffrage:Voter ID:server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6 Address:2c3ae627-9235-fc01-1b2f-17888a6c2ad6}]" 2024-06-01T18:41:36.244Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbce9701-7024-f4b7-9132-7b52611e05fe Address:cbce9701-7024-f4b7-9132-7b52611e05fe} {Suffrage:Voter ID:server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1 Address:ebb2c59a-efc0-97a2-5bec-31ed085b99f1} {Suffrage:Voter ID:server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6 Address:2c3ae627-9235-fc01-1b2f-17888a6c2ad6}]" 2024-06-01T18:41:36.244Z [INFO] server-cbce9701-7024-f4b7-9132-7b52611e05fe: entering follower state: follower="Node at cbce9701-7024-f4b7-9132-7b52611e05fe [Follower]" leader-address= leader-id= 2024-06-01T18:41:36.244Z [INFO] server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbce9701-7024-f4b7-9132-7b52611e05fe Address:cbce9701-7024-f4b7-9132-7b52611e05fe} {Suffrage:Voter ID:server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1 Address:ebb2c59a-efc0-97a2-5bec-31ed085b99f1} {Suffrage:Voter ID:server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6 Address:2c3ae627-9235-fc01-1b2f-17888a6c2ad6}]" 2024-06-01T18:41:36.244Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:36.244Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: entering follower state: follower="Node at ebb2c59a-efc0-97a2-5bec-31ed085b99f1 [Follower]" leader-address= leader-id= 2024-06-01T18:41:36.244Z [INFO] server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6: entering follower state: follower="Node at 2c3ae627-9235-fc01-1b2f-17888a6c2ad6 [Follower]" leader-address= leader-id= 2024-06-01T18:41:36.305Z [WARN] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:36.305Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: entering candidate state: node="Node at ebb2c59a-efc0-97a2-5bec-31ed085b99f1 [Candidate]" term=2 2024-06-01T18:41:36.305Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: election won: term=2 tally=2 2024-06-01T18:41:36.305Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: entering leader state: leader="Node at ebb2c59a-efc0-97a2-5bec-31ed085b99f1 [Leader]" 2024-06-01T18:41:36.305Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: added peer, starting replication: peer=server-cbce9701-7024-f4b7-9132-7b52611e05fe 2024-06-01T18:41:36.305Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: added peer, starting replication: peer=server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6 2024-06-01T18:41:36.305Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: pipelining replication: peer="{Voter server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6 2c3ae627-9235-fc01-1b2f-17888a6c2ad6}" 2024-06-01T18:41:36.305Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: pipelining replication: peer="{Voter server-cbce9701-7024-f4b7-9132-7b52611e05fe cbce9701-7024-f4b7-9132-7b52611e05fe}" 2024-06-01T18:41:36.410Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:36.305288502 +0000 UTC m=+11.803477954 (1 nodes), 61.033387ms from start of poll, 61.255471ms from cluster start. Timeout at 2024-06-01 18:41:36.410643645 +0000 UTC m=+11.908833097, 105.355143ms after stability 2024-06-01T18:41:36.410Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: updating configuration: command=RemoveServer server-id=da5da974-cf48-a65b-08bb-f6ea964a9e4c server-addr= servers="[{Suffrage:Voter ID:server-cbce9701-7024-f4b7-9132-7b52611e05fe Address:cbce9701-7024-f4b7-9132-7b52611e05fe} {Suffrage:Voter ID:server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1 Address:ebb2c59a-efc0-97a2-5bec-31ed085b99f1} {Suffrage:Voter ID:server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6 Address:2c3ae627-9235-fc01-1b2f-17888a6c2ad6}]" 2024-06-01T18:41:36.410Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: aborting pipeline replication: peer="{Voter server-cbce9701-7024-f4b7-9132-7b52611e05fe cbce9701-7024-f4b7-9132-7b52611e05fe}" 2024-06-01T18:41:36.410Z [INFO] server-ebb2c59a-efc0-97a2-5bec-31ed085b99f1: aborting pipeline replication: peer="{Voter server-2c3ae627-9235-fc01-1b2f-17888a6c2ad6 2c3ae627-9235-fc01-1b2f-17888a6c2ad6}" --- PASS: TestRaft_RemoveUnknownPeer (0.17s) === RUN TestRaft_SnapshotRestore 2024-06-01T18:41:36.411Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f67203-0038-ff21-0344-0c439c81b7fd Address:a6f67203-0038-ff21-0344-0c439c81b7fd}]" 2024-06-01T18:41:36.411Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:36.411Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: entering follower state: follower="Node at a6f67203-0038-ff21-0344-0c439c81b7fd [Follower]" leader-address= leader-id= 2024-06-01T18:41:36.503Z [WARN] server-a6f67203-0038-ff21-0344-0c439c81b7fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:36.503Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: entering candidate state: node="Node at a6f67203-0038-ff21-0344-0c439c81b7fd [Candidate]" term=2 2024-06-01T18:41:36.503Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: election won: term=2 tally=1 2024-06-01T18:41:36.503Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: entering leader state: leader="Node at a6f67203-0038-ff21-0344-0c439c81b7fd [Leader]" 2024-06-01T18:41:36.608Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:36.503670678 +0000 UTC m=+12.001860130 (1 nodes), 92.049313ms from start of poll, 92.154589ms from cluster start. Timeout at 2024-06-01 18:41:36.608754667 +0000 UTC m=+12.106944119, 105.083989ms after stability 2024-06-01T18:41:36.610Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: starting snapshot up to: index=102 2024-06-01T18:41:36.610Z [INFO] creating new snapshot: path=/tmp/raft3331614896/snapshots/2-102-1717267296610.tmp 2024-06-01T18:41:36.611Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: compacting logs: from=1 to=92 2024-06-01T18:41:36.611Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: snapshot complete up to: index=102 2024-06-01T18:41:36.611Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: starting restore from snapshot: id=2-102-1717267296610 last-index=102 last-term=2 size-in-bytes=693 2024-06-01T18:41:36.611Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: snapshot restore progress: id=2-102-1717267296610 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-01T18:41:36.611Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: restored from snapshot: id=2-102-1717267296610 last-index=102 last-term=2 size-in-bytes=693 2024-06-01T18:41:36.611Z [INFO] server-a6f67203-0038-ff21-0344-0c439c81b7fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f67203-0038-ff21-0344-0c439c81b7fd Address:a6f67203-0038-ff21-0344-0c439c81b7fd}]" --- PASS: TestRaft_SnapshotRestore (0.20s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2024-06-01T18:41:36.612Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e62bf24-5636-c5ff-f44e-5059b6206e45 Address:1e62bf24-5636-c5ff-f44e-5059b6206e45}]" 2024-06-01T18:41:36.613Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:36.613Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: entering follower state: follower="Node at 1e62bf24-5636-c5ff-f44e-5059b6206e45 [Follower]" leader-address= leader-id= 2024-06-01T18:41:36.670Z [WARN] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:36.670Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: entering candidate state: node="Node at 1e62bf24-5636-c5ff-f44e-5059b6206e45 [Candidate]" term=2 2024-06-01T18:41:36.670Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: election won: term=2 tally=1 2024-06-01T18:41:36.670Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: entering leader state: leader="Node at 1e62bf24-5636-c5ff-f44e-5059b6206e45 [Leader]" 2024-06-01T18:41:36.775Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:36.67035531 +0000 UTC m=+12.168544762 (1 nodes), 57.327829ms from start of poll, 57.439677ms from cluster start. Timeout at 2024-06-01 18:41:36.775890638 +0000 UTC m=+12.274080091, 105.535329ms after stability 2024-06-01T18:41:36.777Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: starting snapshot up to: index=102 2024-06-01T18:41:36.777Z [INFO] creating new snapshot: path=/tmp/raft620791367/snapshots/2-102-1717267296777.tmp 2024-06-01T18:41:36.777Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: compacting logs: from=1 to=92 2024-06-01T18:41:36.777Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: snapshot complete up to: index=102 2024-06-01T18:41:36.777Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: starting restore from snapshot: id=2-102-1717267296777 last-index=102 last-term=2 size-in-bytes=693 2024-06-01T18:41:36.777Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: snapshot restore progress: id=2-102-1717267296777 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-01T18:41:36.777Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: restored from snapshot: id=2-102-1717267296777 last-index=102 last-term=2 size-in-bytes=693 2024-06-01T18:41:36.777Z [INFO] server-1e62bf24-5636-c5ff-f44e-5059b6206e45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e62bf24-5636-c5ff-f44e-5059b6206e45 Address:1e62bf24-5636-c5ff-f44e-5059b6206e45}]" --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.17s) === RUN TestRaft_SnapshotRestore_Progress 2024-06-01T18:41:36.779Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27b21773-b2d7-b081-a5dc-329a01a96168 Address:27b21773-b2d7-b081-a5dc-329a01a96168}]" 2024-06-01T18:41:36.779Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:36.779Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: entering follower state: follower="Node at 27b21773-b2d7-b081-a5dc-329a01a96168 [Follower]" leader-address= leader-id= 2024-06-01T18:41:36.865Z [WARN] server-27b21773-b2d7-b081-a5dc-329a01a96168: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:36.865Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: entering candidate state: node="Node at 27b21773-b2d7-b081-a5dc-329a01a96168 [Candidate]" term=2 2024-06-01T18:41:36.865Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: election won: term=2 tally=1 2024-06-01T18:41:36.865Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: entering leader state: leader="Node at 27b21773-b2d7-b081-a5dc-329a01a96168 [Leader]" 2024-06-01T18:41:36.971Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:36.865814316 +0000 UTC m=+12.364003768 (1 nodes), 86.661906ms from start of poll, 86.763776ms from cluster start. Timeout at 2024-06-01 18:41:36.971327213 +0000 UTC m=+12.469516665, 105.512897ms after stability 2024-06-01T18:41:36.972Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: starting snapshot up to: index=102 2024-06-01T18:41:36.972Z [INFO] creating new snapshot: path=/tmp/raft1952981800/snapshots/2-102-1717267296972.tmp 2024-06-01T18:41:36.973Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: compacting logs: from=1 to=92 2024-06-01T18:41:36.973Z [INFO] server-27b21773-b2d7-b081-a5dc-329a01a96168: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.20s) === RUN TestRaft_NoRestoreOnStart 2024-06-01T18:41:36.974Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d221c79-0417-77cc-2bf4-634c983ec712 Address:9d221c79-0417-77cc-2bf4-634c983ec712}]" 2024-06-01T18:41:36.974Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:36.974Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: entering follower state: follower="Node at 9d221c79-0417-77cc-2bf4-634c983ec712 [Follower]" leader-address= leader-id= 2024-06-01T18:41:37.071Z [WARN] server-9d221c79-0417-77cc-2bf4-634c983ec712: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:37.071Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: entering candidate state: node="Node at 9d221c79-0417-77cc-2bf4-634c983ec712 [Candidate]" term=2 2024-06-01T18:41:37.071Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: election won: term=2 tally=1 2024-06-01T18:41:37.071Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: entering leader state: leader="Node at 9d221c79-0417-77cc-2bf4-634c983ec712 [Leader]" 2024-06-01T18:41:37.177Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:37.071469823 +0000 UTC m=+12.569659275 (1 nodes), 96.764111ms from start of poll, 96.862485ms from cluster start. Timeout at 2024-06-01 18:41:37.177231713 +0000 UTC m=+12.675421165, 105.76189ms after stability 2024-06-01T18:41:37.178Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: starting snapshot up to: index=102 2024-06-01T18:41:37.178Z [INFO] creating new snapshot: path=/tmp/raft4029902676/snapshots/2-102-1717267297178.tmp 2024-06-01T18:41:37.179Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: compacting logs: from=1 to=92 2024-06-01T18:41:37.179Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: snapshot complete up to: index=102 2024-06-01T18:41:37.179Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d221c79-0417-77cc-2bf4-634c983ec712 Address:9d221c79-0417-77cc-2bf4-634c983ec712}]" --- PASS: TestRaft_NoRestoreOnStart (0.21s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1266: Skipping unreliable test 2024-06-01T18:41:37.179Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: entering follower state: follower="Node at 9d221c79-0417-77cc-2bf4-634c983ec712 [Follower]" leader-address= leader-id= --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2024-06-01T18:41:37.179Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5548ade1-e630-4eca-91ac-273dec43e7cd Address:5548ade1-e630-4eca-91ac-273dec43e7cd}]" 2024-06-01T18:41:37.180Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:37.180Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: entering follower state: follower="Node at 5548ade1-e630-4eca-91ac-273dec43e7cd [Follower]" leader-address= leader-id= 2024-06-01T18:41:37.243Z [WARN] server-5548ade1-e630-4eca-91ac-273dec43e7cd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:37.243Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: entering candidate state: node="Node at 5548ade1-e630-4eca-91ac-273dec43e7cd [Candidate]" term=2 2024-06-01T18:41:37.243Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: election won: term=2 tally=1 2024-06-01T18:41:37.243Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: entering leader state: leader="Node at 5548ade1-e630-4eca-91ac-273dec43e7cd [Leader]" 2024-06-01T18:41:37.251Z [WARN] server-9d221c79-0417-77cc-2bf4-634c983ec712: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:37.251Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: entering candidate state: node="Node at 9d221c79-0417-77cc-2bf4-634c983ec712 [Candidate]" term=3 2024-06-01T18:41:37.251Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: election won: term=3 tally=1 2024-06-01T18:41:37.251Z [INFO] server-9d221c79-0417-77cc-2bf4-634c983ec712: entering leader state: leader="Node at 9d221c79-0417-77cc-2bf4-634c983ec712 [Leader]" 2024-06-01T18:41:37.348Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:37.243148449 +0000 UTC m=+12.741337892 (1 nodes), 63.106927ms from start of poll, 63.225667ms from cluster start. Timeout at 2024-06-01 18:41:37.348905922 +0000 UTC m=+12.847095374, 105.757482ms after stability 2024-06-01T18:41:37.363Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: starting snapshot up to: index=102 2024-06-01T18:41:37.363Z [INFO] creating new snapshot: path=/tmp/raft619708934/snapshots/2-102-1717267297363.tmp 2024-06-01T18:41:37.363Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: compacting logs: from=1 to=92 2024-06-01T18:41:37.363Z [INFO] server-5548ade1-e630-4eca-91ac-273dec43e7cd: snapshot complete up to: index=102 --- PASS: TestRaft_AutoSnapshot (0.28s) === RUN TestRaft_UserSnapshot 2024-06-01T18:41:37.456Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09 Address:ec7d65e2-cff4-4e42-fbea-7bf690d5bc09}]" 2024-06-01T18:41:37.456Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:37.456Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: entering follower state: follower="Node at ec7d65e2-cff4-4e42-fbea-7bf690d5bc09 [Follower]" leader-address= leader-id= 2024-06-01T18:41:37.554Z [WARN] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:37.554Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: entering candidate state: node="Node at ec7d65e2-cff4-4e42-fbea-7bf690d5bc09 [Candidate]" term=2 2024-06-01T18:41:37.554Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: election won: term=2 tally=1 2024-06-01T18:41:37.554Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: entering leader state: leader="Node at ec7d65e2-cff4-4e42-fbea-7bf690d5bc09 [Leader]" 2024-06-01T18:41:37.660Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:37.554339826 +0000 UTC m=+13.052529288 (1 nodes), 98.289881ms from start of poll, 98.430142ms from cluster start. Timeout at 2024-06-01 18:41:37.660148464 +0000 UTC m=+13.158337916, 105.808628ms after stability 2024-06-01T18:41:37.660Z [ERROR] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: failed to take snapshot: error="nothing new to snapshot" 2024-06-01T18:41:37.660Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: starting snapshot up to: index=12 2024-06-01T18:41:37.660Z [INFO] creating new snapshot: path=/tmp/raft3612572218/snapshots/2-12-1717267297660.tmp 2024-06-01T18:41:37.661Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: compacting logs: from=1 to=2 2024-06-01T18:41:37.661Z [INFO] server-ec7d65e2-cff4-4e42-fbea-7bf690d5bc09: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.21s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2024-06-01T18:41:37.663Z [INFO] server-96647624-1887-591c-46de-d34c481a4afa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96647624-1887-591c-46de-d34c481a4afa Address:96647624-1887-591c-46de-d34c481a4afa} {Suffrage:Voter ID:server-ab166916-6d15-17a9-2344-b2ca90903ba8 Address:ab166916-6d15-17a9-2344-b2ca90903ba8} {Suffrage:Voter ID:server-e2ccea32-4040-6a37-c93b-2734d28e78c5 Address:e2ccea32-4040-6a37-c93b-2734d28e78c5}]" 2024-06-01T18:41:37.663Z [INFO] server-96647624-1887-591c-46de-d34c481a4afa: entering follower state: follower="Node at 96647624-1887-591c-46de-d34c481a4afa [Follower]" leader-address= leader-id= 2024-06-01T18:41:37.663Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96647624-1887-591c-46de-d34c481a4afa Address:96647624-1887-591c-46de-d34c481a4afa} {Suffrage:Voter ID:server-ab166916-6d15-17a9-2344-b2ca90903ba8 Address:ab166916-6d15-17a9-2344-b2ca90903ba8} {Suffrage:Voter ID:server-e2ccea32-4040-6a37-c93b-2734d28e78c5 Address:e2ccea32-4040-6a37-c93b-2734d28e78c5}]" 2024-06-01T18:41:37.663Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: entering follower state: follower="Node at ab166916-6d15-17a9-2344-b2ca90903ba8 [Follower]" leader-address= leader-id= 2024-06-01T18:41:37.663Z [INFO] server-e2ccea32-4040-6a37-c93b-2734d28e78c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96647624-1887-591c-46de-d34c481a4afa Address:96647624-1887-591c-46de-d34c481a4afa} {Suffrage:Voter ID:server-ab166916-6d15-17a9-2344-b2ca90903ba8 Address:ab166916-6d15-17a9-2344-b2ca90903ba8} {Suffrage:Voter ID:server-e2ccea32-4040-6a37-c93b-2734d28e78c5 Address:e2ccea32-4040-6a37-c93b-2734d28e78c5}]" 2024-06-01T18:41:37.663Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:37.663Z [INFO] server-e2ccea32-4040-6a37-c93b-2734d28e78c5: entering follower state: follower="Node at e2ccea32-4040-6a37-c93b-2734d28e78c5 [Follower]" leader-address= leader-id= 2024-06-01T18:41:38.324Z [WARN] server-ab166916-6d15-17a9-2344-b2ca90903ba8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:38.324Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: entering candidate state: node="Node at ab166916-6d15-17a9-2344-b2ca90903ba8 [Candidate]" term=2 2024-06-01T18:41:38.324Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: election won: term=2 tally=2 2024-06-01T18:41:38.324Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: entering leader state: leader="Node at ab166916-6d15-17a9-2344-b2ca90903ba8 [Leader]" 2024-06-01T18:41:38.324Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: added peer, starting replication: peer=server-96647624-1887-591c-46de-d34c481a4afa 2024-06-01T18:41:38.324Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: added peer, starting replication: peer=server-e2ccea32-4040-6a37-c93b-2734d28e78c5 2024-06-01T18:41:38.324Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: pipelining replication: peer="{Voter server-e2ccea32-4040-6a37-c93b-2734d28e78c5 e2ccea32-4040-6a37-c93b-2734d28e78c5}" 2024-06-01T18:41:38.324Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: pipelining replication: peer="{Voter server-96647624-1887-591c-46de-d34c481a4afa 96647624-1887-591c-46de-d34c481a4afa}" 2024-06-01T18:41:39.330Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:38.324415938 +0000 UTC m=+13.822605390 (1 nodes), 661.140343ms from start of poll, 661.467352ms from cluster start. Timeout at 2024-06-01 18:41:39.33048466 +0000 UTC m=+14.828674112, 1.006068722s after stability 2024-06-01T18:41:39.330Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: starting snapshot up to: index=12 2024-06-01T18:41:39.330Z [INFO] creating new snapshot: path=/tmp/raft3307361910/snapshots/2-12-1717267299330.tmp 2024-06-01T18:41:39.331Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: snapshot complete up to: index=12 2024-06-01T18:41:39.331Z [INFO] creating new snapshot: path=/tmp/raft3307361910/snapshots/2-23-1717267299331.tmp 2024-06-01T18:41:39.332Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: copied to local snapshot: bytes=71 2024-06-01T18:41:39.332Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: snapshot restore progress: id=2-23-1717267299331 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:39.332Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: restored user snapshot: index=23 2024-06-01T18:41:39.332Z [ERROR] server-ab166916-6d15-17a9-2344-b2ca90903ba8: failed to get log: index=23 error="log not found" 2024-06-01T18:41:39.332Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: aborting pipeline replication: peer="{Voter server-96647624-1887-591c-46de-d34c481a4afa 96647624-1887-591c-46de-d34c481a4afa}" 2024-06-01T18:41:39.332Z [ERROR] server-ab166916-6d15-17a9-2344-b2ca90903ba8: failed to get log: index=23 error="log not found" 2024-06-01T18:41:39.332Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: aborting pipeline replication: peer="{Voter server-e2ccea32-4040-6a37-c93b-2734d28e78c5 e2ccea32-4040-6a37-c93b-2734d28e78c5}" 2024-06-01T18:41:39.339Z [ERROR] server-ab166916-6d15-17a9-2344-b2ca90903ba8: failed to get log: index=23 error="log not found" 2024-06-01T18:41:39.339Z [INFO] creating new snapshot: path=/tmp/raft2575902737/snapshots/2-23-1717267299339.tmp 2024-06-01T18:41:39.339Z [INFO] server-96647624-1887-591c-46de-d34c481a4afa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:39.339Z [INFO] server-96647624-1887-591c-46de-d34c481a4afa: copied to local snapshot: bytes=71 2024-06-01T18:41:39.339Z [ERROR] server-ab166916-6d15-17a9-2344-b2ca90903ba8: failed to get log: index=23 error="log not found" 2024-06-01T18:41:39.339Z [INFO] server-96647624-1887-591c-46de-d34c481a4afa: snapshot restore progress: id=2-23-1717267299339 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:39.339Z [INFO] server-96647624-1887-591c-46de-d34c481a4afa: Installed remote snapshot 2024-06-01T18:41:39.339Z [WARN] server-96647624-1887-591c-46de-d34c481a4afa: skipping application of old log: index=22 2024-06-01T18:41:39.340Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: pipelining replication: peer="{Voter server-96647624-1887-591c-46de-d34c481a4afa 96647624-1887-591c-46de-d34c481a4afa}" 2024-06-01T18:41:39.340Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:39.340Z [INFO] creating new snapshot: path=/tmp/raft3077841696/snapshots/2-23-1717267299340.tmp 2024-06-01T18:41:39.340Z [INFO] server-e2ccea32-4040-6a37-c93b-2734d28e78c5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:39.340Z [INFO] server-e2ccea32-4040-6a37-c93b-2734d28e78c5: copied to local snapshot: bytes=71 2024-06-01T18:41:39.340Z [INFO] server-e2ccea32-4040-6a37-c93b-2734d28e78c5: snapshot restore progress: id=2-23-1717267299340 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:39.340Z [INFO] server-e2ccea32-4040-6a37-c93b-2734d28e78c5: Installed remote snapshot 2024-06-01T18:41:39.340Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: pipelining replication: peer="{Voter server-e2ccea32-4040-6a37-c93b-2734d28e78c5 e2ccea32-4040-6a37-c93b-2734d28e78c5}" 2024-06-01T18:41:40.345Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:39.340035239 +0000 UTC m=+14.838224692 (2 nodes), 511ns from start of poll, 1.677086654s from cluster start. Timeout at 2024-06-01 18:41:40.345486993 +0000 UTC m=+15.843676445, 1.005451753s after stability 2024-06-01T18:41:40.356Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: aborting pipeline replication: peer="{Voter server-e2ccea32-4040-6a37-c93b-2734d28e78c5 e2ccea32-4040-6a37-c93b-2734d28e78c5}" 2024-06-01T18:41:40.356Z [INFO] server-ab166916-6d15-17a9-2344-b2ca90903ba8: aborting pipeline replication: peer="{Voter server-96647624-1887-591c-46de-d34c481a4afa 96647624-1887-591c-46de-d34c481a4afa}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2024-06-01T18:41:40.360Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 Address:dfcaa1d7-5cf1-6fa9-5469-3cfff3685023} {Suffrage:Voter ID:server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e Address:381f7ee6-cc38-f347-6e01-4ff5d1b1592e} {Suffrage:Voter ID:server-214f2ef9-e565-e38d-7dee-8a692acf9fc7 Address:214f2ef9-e565-e38d-7dee-8a692acf9fc7}]" 2024-06-01T18:41:40.360Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 Address:dfcaa1d7-5cf1-6fa9-5469-3cfff3685023} {Suffrage:Voter ID:server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e Address:381f7ee6-cc38-f347-6e01-4ff5d1b1592e} {Suffrage:Voter ID:server-214f2ef9-e565-e38d-7dee-8a692acf9fc7 Address:214f2ef9-e565-e38d-7dee-8a692acf9fc7}]" 2024-06-01T18:41:40.360Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: entering follower state: follower="Node at dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 [Follower]" leader-address= leader-id= 2024-06-01T18:41:40.360Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 Address:dfcaa1d7-5cf1-6fa9-5469-3cfff3685023} {Suffrage:Voter ID:server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e Address:381f7ee6-cc38-f347-6e01-4ff5d1b1592e} {Suffrage:Voter ID:server-214f2ef9-e565-e38d-7dee-8a692acf9fc7 Address:214f2ef9-e565-e38d-7dee-8a692acf9fc7}]" 2024-06-01T18:41:40.360Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:40.360Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: entering follower state: follower="Node at 381f7ee6-cc38-f347-6e01-4ff5d1b1592e [Follower]" leader-address= leader-id= 2024-06-01T18:41:40.360Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: entering follower state: follower="Node at 214f2ef9-e565-e38d-7dee-8a692acf9fc7 [Follower]" leader-address= leader-id= 2024-06-01T18:41:40.932Z [WARN] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:40.933Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: entering candidate state: node="Node at 214f2ef9-e565-e38d-7dee-8a692acf9fc7 [Candidate]" term=2 2024-06-01T18:41:40.933Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: election won: term=2 tally=2 2024-06-01T18:41:40.933Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: entering leader state: leader="Node at 214f2ef9-e565-e38d-7dee-8a692acf9fc7 [Leader]" 2024-06-01T18:41:40.933Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: added peer, starting replication: peer=server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 2024-06-01T18:41:40.933Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: added peer, starting replication: peer=server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e 2024-06-01T18:41:40.933Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: pipelining replication: peer="{Voter server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e 381f7ee6-cc38-f347-6e01-4ff5d1b1592e}" 2024-06-01T18:41:40.933Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: pipelining replication: peer="{Voter server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 dfcaa1d7-5cf1-6fa9-5469-3cfff3685023}" 2024-06-01T18:41:41.938Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:40.933251401 +0000 UTC m=+16.431440854 (1 nodes), 572.431296ms from start of poll, 572.713542ms from cluster start. Timeout at 2024-06-01 18:41:41.938331673 +0000 UTC m=+17.436521125, 1.005080271s after stability 2024-06-01T18:41:41.938Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: starting snapshot up to: index=12 2024-06-01T18:41:41.938Z [INFO] creating new snapshot: path=/tmp/raft64146911/snapshots/2-12-1717267301938.tmp 2024-06-01T18:41:41.939Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: snapshot complete up to: index=12 2024-06-01T18:41:41.940Z [INFO] creating new snapshot: path=/tmp/raft64146911/snapshots/2-23-1717267301940.tmp 2024-06-01T18:41:41.940Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: copied to local snapshot: bytes=71 2024-06-01T18:41:41.940Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: snapshot restore progress: id=2-23-1717267301940 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:41.940Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: removing all old logs from log store 2024-06-01T18:41:41.940Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: compacting logs: from=1 to=22 2024-06-01T18:41:41.940Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: restored user snapshot: index=23 2024-06-01T18:41:41.940Z [ERROR] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: failed to get log: index=22 error="log not found" 2024-06-01T18:41:41.940Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: aborting pipeline replication: peer="{Voter server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e 381f7ee6-cc38-f347-6e01-4ff5d1b1592e}" 2024-06-01T18:41:41.940Z [ERROR] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: failed to get log: index=22 error="log not found" 2024-06-01T18:41:41.940Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: aborting pipeline replication: peer="{Voter server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 dfcaa1d7-5cf1-6fa9-5469-3cfff3685023}" 2024-06-01T18:41:41.946Z [ERROR] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: failed to get log: index=22 error="log not found" 2024-06-01T18:41:41.946Z [ERROR] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: failed to get log: index=22 error="log not found" 2024-06-01T18:41:41.946Z [INFO] creating new snapshot: path=/tmp/raft1632335260/snapshots/2-23-1717267301946.tmp 2024-06-01T18:41:41.946Z [INFO] creating new snapshot: path=/tmp/raft647795991/snapshots/2-23-1717267301946.tmp 2024-06-01T18:41:41.947Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:41.947Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:41.948Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: copied to local snapshot: bytes=71 2024-06-01T18:41:41.948Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: copied to local snapshot: bytes=71 2024-06-01T18:41:41.948Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: snapshot restore progress: id=2-23-1717267301946 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:41.948Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: removing all old logs from log store 2024-06-01T18:41:41.948Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: compacting logs: from=1 to=22 2024-06-01T18:41:41.948Z [INFO] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: Installed remote snapshot 2024-06-01T18:41:41.948Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: snapshot restore progress: id=2-23-1717267301946 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:41.948Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: removing all old logs from log store 2024-06-01T18:41:41.948Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: compacting logs: from=1 to=22 2024-06-01T18:41:41.948Z [INFO] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: Installed remote snapshot 2024-06-01T18:41:41.948Z [WARN] server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023: skipping application of old log: index=22 2024-06-01T18:41:41.948Z [WARN] server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e: skipping application of old log: index=23 2024-06-01T18:41:41.948Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: pipelining replication: peer="{Voter server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e 381f7ee6-cc38-f347-6e01-4ff5d1b1592e}" 2024-06-01T18:41:41.948Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: pipelining replication: peer="{Voter server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 dfcaa1d7-5cf1-6fa9-5469-3cfff3685023}" 2024-06-01T18:41:41.948Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:42.954Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:41.949006334 +0000 UTC m=+17.447195786 (2 nodes), 1.012µs from start of poll, 1.588468474s from cluster start. Timeout at 2024-06-01 18:41:42.954073932 +0000 UTC m=+18.452263384, 1.005067598s after stability 2024-06-01T18:41:42.966Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: aborting pipeline replication: peer="{Voter server-381f7ee6-cc38-f347-6e01-4ff5d1b1592e 381f7ee6-cc38-f347-6e01-4ff5d1b1592e}" 2024-06-01T18:41:42.966Z [INFO] server-214f2ef9-e565-e38d-7dee-8a692acf9fc7: aborting pipeline replication: peer="{Voter server-dfcaa1d7-5cf1-6fa9-5469-3cfff3685023 dfcaa1d7-5cf1-6fa9-5469-3cfff3685023}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2024-06-01T18:41:42.968Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13b31071-7349-d5ab-66e2-2065dd8d5281 Address:13b31071-7349-d5ab-66e2-2065dd8d5281} {Suffrage:Voter ID:server-f424fc69-e2e6-10b8-6bbe-48a024b43089 Address:f424fc69-e2e6-10b8-6bbe-48a024b43089} {Suffrage:Voter ID:server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f Address:f79a0f5a-f156-b806-53c3-6c0a69fc1f5f}]" 2024-06-01T18:41:42.968Z [INFO] server-f424fc69-e2e6-10b8-6bbe-48a024b43089: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13b31071-7349-d5ab-66e2-2065dd8d5281 Address:13b31071-7349-d5ab-66e2-2065dd8d5281} {Suffrage:Voter ID:server-f424fc69-e2e6-10b8-6bbe-48a024b43089 Address:f424fc69-e2e6-10b8-6bbe-48a024b43089} {Suffrage:Voter ID:server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f Address:f79a0f5a-f156-b806-53c3-6c0a69fc1f5f}]" 2024-06-01T18:41:42.968Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: entering follower state: follower="Node at 13b31071-7349-d5ab-66e2-2065dd8d5281 [Follower]" leader-address= leader-id= 2024-06-01T18:41:42.968Z [INFO] server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13b31071-7349-d5ab-66e2-2065dd8d5281 Address:13b31071-7349-d5ab-66e2-2065dd8d5281} {Suffrage:Voter ID:server-f424fc69-e2e6-10b8-6bbe-48a024b43089 Address:f424fc69-e2e6-10b8-6bbe-48a024b43089} {Suffrage:Voter ID:server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f Address:f79a0f5a-f156-b806-53c3-6c0a69fc1f5f}]" 2024-06-01T18:41:42.968Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:42.968Z [INFO] server-f424fc69-e2e6-10b8-6bbe-48a024b43089: entering follower state: follower="Node at f424fc69-e2e6-10b8-6bbe-48a024b43089 [Follower]" leader-address= leader-id= 2024-06-01T18:41:42.968Z [INFO] server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f: entering follower state: follower="Node at f79a0f5a-f156-b806-53c3-6c0a69fc1f5f [Follower]" leader-address= leader-id= 2024-06-01T18:41:43.767Z [WARN] server-13b31071-7349-d5ab-66e2-2065dd8d5281: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:43.767Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: entering candidate state: node="Node at 13b31071-7349-d5ab-66e2-2065dd8d5281 [Candidate]" term=2 2024-06-01T18:41:43.767Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: election won: term=2 tally=2 2024-06-01T18:41:43.767Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: entering leader state: leader="Node at 13b31071-7349-d5ab-66e2-2065dd8d5281 [Leader]" 2024-06-01T18:41:43.767Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: added peer, starting replication: peer=server-f424fc69-e2e6-10b8-6bbe-48a024b43089 2024-06-01T18:41:43.767Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: added peer, starting replication: peer=server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f 2024-06-01T18:41:43.767Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: pipelining replication: peer="{Voter server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f f79a0f5a-f156-b806-53c3-6c0a69fc1f5f}" 2024-06-01T18:41:43.768Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: pipelining replication: peer="{Voter server-f424fc69-e2e6-10b8-6bbe-48a024b43089 f424fc69-e2e6-10b8-6bbe-48a024b43089}" 2024-06-01T18:41:44.773Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:43.767997307 +0000 UTC m=+19.266186759 (1 nodes), 799.50288ms from start of poll, 799.820522ms from cluster start. Timeout at 2024-06-01 18:41:44.773895129 +0000 UTC m=+20.272084581, 1.005897822s after stability 2024-06-01T18:41:44.774Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: starting snapshot up to: index=12 2024-06-01T18:41:44.774Z [INFO] creating new snapshot: path=/tmp/raft1167718743/snapshots/2-12-1717267304774.tmp 2024-06-01T18:41:44.775Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: snapshot complete up to: index=12 2024-06-01T18:41:44.776Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d Address:867f295e-9d49-e6bd-e3ce-e0df010e4e8d} {Suffrage:Voter ID:server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 Address:d3901c8e-ff2b-bd23-daee-524b0cc7cdd1} {Suffrage:Voter ID:server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf Address:31ffa1d5-7fc6-4224-b5b3-81a6403373cf}]" 2024-06-01T18:41:44.777Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: entering follower state: follower="Node at 867f295e-9d49-e6bd-e3ce-e0df010e4e8d [Follower]" leader-address= leader-id= 2024-06-01T18:41:44.777Z [INFO] server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d Address:867f295e-9d49-e6bd-e3ce-e0df010e4e8d} {Suffrage:Voter ID:server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 Address:d3901c8e-ff2b-bd23-daee-524b0cc7cdd1} {Suffrage:Voter ID:server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf Address:31ffa1d5-7fc6-4224-b5b3-81a6403373cf}]" 2024-06-01T18:41:44.777Z [INFO] server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d Address:867f295e-9d49-e6bd-e3ce-e0df010e4e8d} {Suffrage:Voter ID:server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 Address:d3901c8e-ff2b-bd23-daee-524b0cc7cdd1} {Suffrage:Voter ID:server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf Address:31ffa1d5-7fc6-4224-b5b3-81a6403373cf}]" 2024-06-01T18:41:44.777Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:44.777Z [INFO] server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf: entering follower state: follower="Node at 31ffa1d5-7fc6-4224-b5b3-81a6403373cf [Follower]" leader-address= leader-id= 2024-06-01T18:41:44.777Z [INFO] server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1: entering follower state: follower="Node at d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 [Follower]" leader-address= leader-id= 2024-06-01T18:41:45.286Z [WARN] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:45.286Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: entering candidate state: node="Node at 867f295e-9d49-e6bd-e3ce-e0df010e4e8d [Candidate]" term=2 2024-06-01T18:41:45.286Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: election won: term=2 tally=2 2024-06-01T18:41:45.286Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: entering leader state: leader="Node at 867f295e-9d49-e6bd-e3ce-e0df010e4e8d [Leader]" 2024-06-01T18:41:45.286Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: added peer, starting replication: peer=server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 2024-06-01T18:41:45.286Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: added peer, starting replication: peer=server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf 2024-06-01T18:41:45.286Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: pipelining replication: peer="{Voter server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf 31ffa1d5-7fc6-4224-b5b3-81a6403373cf}" 2024-06-01T18:41:45.286Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: pipelining replication: peer="{Voter server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 d3901c8e-ff2b-bd23-daee-524b0cc7cdd1}" 2024-06-01T18:41:46.292Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:45.286664471 +0000 UTC m=+20.784853924 (1 nodes), 509.158211ms from start of poll, 509.901294ms from cluster start. Timeout at 2024-06-01 18:41:46.292381067 +0000 UTC m=+21.790570519, 1.005716595s after stability 2024-06-01T18:41:46.292Z [INFO] creating new snapshot: path=/tmp/raft557610843/snapshots/2-13-1717267306292.tmp 2024-06-01T18:41:46.293Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: copied to local snapshot: bytes=71 2024-06-01T18:41:46.293Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: snapshot restore progress: id=2-13-1717267306292 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:46.293Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: restored user snapshot: index=13 2024-06-01T18:41:46.293Z [ERROR] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: failed to get log: index=3 error="log not found" 2024-06-01T18:41:46.293Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: aborting pipeline replication: peer="{Voter server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 d3901c8e-ff2b-bd23-daee-524b0cc7cdd1}" 2024-06-01T18:41:46.293Z [ERROR] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: failed to get log: index=3 error="log not found" 2024-06-01T18:41:46.293Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: aborting pipeline replication: peer="{Voter server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf 31ffa1d5-7fc6-4224-b5b3-81a6403373cf}" 2024-06-01T18:41:46.298Z [ERROR] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: failed to get log: index=3 error="log not found" 2024-06-01T18:41:46.299Z [INFO] creating new snapshot: path=/tmp/raft2357284849/snapshots/2-13-1717267306299.tmp 2024-06-01T18:41:46.299Z [INFO] server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:46.299Z [INFO] server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf: copied to local snapshot: bytes=71 2024-06-01T18:41:46.300Z [INFO] server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf: snapshot restore progress: id=2-13-1717267306299 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:46.300Z [INFO] server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf: Installed remote snapshot 2024-06-01T18:41:46.300Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: pipelining replication: peer="{Voter server-31ffa1d5-7fc6-4224-b5b3-81a6403373cf 31ffa1d5-7fc6-4224-b5b3-81a6403373cf}" 2024-06-01T18:41:46.300Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:46.303Z [ERROR] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: failed to get log: index=3 error="log not found" 2024-06-01T18:41:46.303Z [INFO] creating new snapshot: path=/tmp/raft118339198/snapshots/2-13-1717267306303.tmp 2024-06-01T18:41:46.303Z [INFO] server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:46.303Z [INFO] server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1: copied to local snapshot: bytes=71 2024-06-01T18:41:46.304Z [INFO] server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1: snapshot restore progress: id=2-13-1717267306303 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:46.304Z [INFO] server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1: Installed remote snapshot 2024-06-01T18:41:46.304Z [INFO] server-867f295e-9d49-e6bd-e3ce-e0df010e4e8d: pipelining replication: peer="{Voter server-d3901c8e-ff2b-bd23-daee-524b0cc7cdd1 d3901c8e-ff2b-bd23-daee-524b0cc7cdd1}" 2024-06-01T18:41:47.305Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:46.300180727 +0000 UTC m=+21.798370179 (2 nodes), 762ns from start of poll, 1.523417549s from cluster start. Timeout at 2024-06-01 18:41:47.305414632 +0000 UTC m=+22.803604074, 1.005233895s after stability 2024-06-01T18:41:47.317Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: aborting pipeline replication: peer="{Voter server-f79a0f5a-f156-b806-53c3-6c0a69fc1f5f f79a0f5a-f156-b806-53c3-6c0a69fc1f5f}" 2024-06-01T18:41:47.317Z [INFO] server-13b31071-7349-d5ab-66e2-2065dd8d5281: aborting pipeline replication: peer="{Voter server-f424fc69-e2e6-10b8-6bbe-48a024b43089 f424fc69-e2e6-10b8-6bbe-48a024b43089}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2024-06-01T18:41:47.319Z [INFO] server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2 Address:abdea5ba-42af-cddb-bf5b-03aeb103c8a2} {Suffrage:Voter ID:server-cd569554-2414-cc6b-3d10-557a91614626 Address:cd569554-2414-cc6b-3d10-557a91614626} {Suffrage:Voter ID:server-d6f22738-e7ba-a75d-db97-5b3d692f6810 Address:d6f22738-e7ba-a75d-db97-5b3d692f6810}]" 2024-06-01T18:41:47.319Z [INFO] server-cd569554-2414-cc6b-3d10-557a91614626: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2 Address:abdea5ba-42af-cddb-bf5b-03aeb103c8a2} {Suffrage:Voter ID:server-cd569554-2414-cc6b-3d10-557a91614626 Address:cd569554-2414-cc6b-3d10-557a91614626} {Suffrage:Voter ID:server-d6f22738-e7ba-a75d-db97-5b3d692f6810 Address:d6f22738-e7ba-a75d-db97-5b3d692f6810}]" 2024-06-01T18:41:47.319Z [INFO] server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2: entering follower state: follower="Node at abdea5ba-42af-cddb-bf5b-03aeb103c8a2 [Follower]" leader-address= leader-id= 2024-06-01T18:41:47.319Z [INFO] server-cd569554-2414-cc6b-3d10-557a91614626: entering follower state: follower="Node at cd569554-2414-cc6b-3d10-557a91614626 [Follower]" leader-address= leader-id= 2024-06-01T18:41:47.319Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2 Address:abdea5ba-42af-cddb-bf5b-03aeb103c8a2} {Suffrage:Voter ID:server-cd569554-2414-cc6b-3d10-557a91614626 Address:cd569554-2414-cc6b-3d10-557a91614626} {Suffrage:Voter ID:server-d6f22738-e7ba-a75d-db97-5b3d692f6810 Address:d6f22738-e7ba-a75d-db97-5b3d692f6810}]" 2024-06-01T18:41:47.319Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:47.319Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: entering follower state: follower="Node at d6f22738-e7ba-a75d-db97-5b3d692f6810 [Follower]" leader-address= leader-id= 2024-06-01T18:41:47.862Z [WARN] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:47.862Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: entering candidate state: node="Node at d6f22738-e7ba-a75d-db97-5b3d692f6810 [Candidate]" term=2 2024-06-01T18:41:47.862Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: election won: term=2 tally=2 2024-06-01T18:41:47.862Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: entering leader state: leader="Node at d6f22738-e7ba-a75d-db97-5b3d692f6810 [Leader]" 2024-06-01T18:41:47.862Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: added peer, starting replication: peer=server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2 2024-06-01T18:41:47.862Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: added peer, starting replication: peer=server-cd569554-2414-cc6b-3d10-557a91614626 2024-06-01T18:41:47.862Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: pipelining replication: peer="{Voter server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2 abdea5ba-42af-cddb-bf5b-03aeb103c8a2}" 2024-06-01T18:41:47.862Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: pipelining replication: peer="{Voter server-cd569554-2414-cc6b-3d10-557a91614626 cd569554-2414-cc6b-3d10-557a91614626}" 2024-06-01T18:41:48.867Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:47.86263919 +0000 UTC m=+23.360828642 (1 nodes), 543.025235ms from start of poll, 543.31828ms from cluster start. Timeout at 2024-06-01 18:41:48.867659529 +0000 UTC m=+24.365848981, 1.005020339s after stability 2024-06-01T18:41:48.867Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: starting snapshot up to: index=12 2024-06-01T18:41:48.867Z [INFO] creating new snapshot: path=/tmp/raft3998152480/snapshots/2-12-1717267308867.tmp 2024-06-01T18:41:48.868Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: snapshot complete up to: index=12 2024-06-01T18:41:48.869Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c873484-fa6b-bbe7-c420-772d906a0820 Address:9c873484-fa6b-bbe7-c420-772d906a0820} {Suffrage:Voter ID:server-e81c06f2-30cc-1311-4c30-244868d09477 Address:e81c06f2-30cc-1311-4c30-244868d09477} {Suffrage:Voter ID:server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9 Address:552dff33-995c-2e1a-ffbc-a1ab9948dcc9}]" 2024-06-01T18:41:48.869Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c873484-fa6b-bbe7-c420-772d906a0820 Address:9c873484-fa6b-bbe7-c420-772d906a0820} {Suffrage:Voter ID:server-e81c06f2-30cc-1311-4c30-244868d09477 Address:e81c06f2-30cc-1311-4c30-244868d09477} {Suffrage:Voter ID:server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9 Address:552dff33-995c-2e1a-ffbc-a1ab9948dcc9}]" 2024-06-01T18:41:48.869Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c873484-fa6b-bbe7-c420-772d906a0820 Address:9c873484-fa6b-bbe7-c420-772d906a0820} {Suffrage:Voter ID:server-e81c06f2-30cc-1311-4c30-244868d09477 Address:e81c06f2-30cc-1311-4c30-244868d09477} {Suffrage:Voter ID:server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9 Address:552dff33-995c-2e1a-ffbc-a1ab9948dcc9}]" 2024-06-01T18:41:48.869Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: entering follower state: follower="Node at e81c06f2-30cc-1311-4c30-244868d09477 [Follower]" leader-address= leader-id= 2024-06-01T18:41:48.869Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:48.869Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: entering follower state: follower="Node at 552dff33-995c-2e1a-ffbc-a1ab9948dcc9 [Follower]" leader-address= leader-id= 2024-06-01T18:41:48.869Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: entering follower state: follower="Node at 9c873484-fa6b-bbe7-c420-772d906a0820 [Follower]" leader-address= leader-id= 2024-06-01T18:41:49.521Z [WARN] server-9c873484-fa6b-bbe7-c420-772d906a0820: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:49.521Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: entering candidate state: node="Node at 9c873484-fa6b-bbe7-c420-772d906a0820 [Candidate]" term=2 2024-06-01T18:41:49.521Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: election won: term=2 tally=2 2024-06-01T18:41:49.521Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: entering leader state: leader="Node at 9c873484-fa6b-bbe7-c420-772d906a0820 [Leader]" 2024-06-01T18:41:49.521Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: added peer, starting replication: peer=server-e81c06f2-30cc-1311-4c30-244868d09477 2024-06-01T18:41:49.521Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: added peer, starting replication: peer=server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9 2024-06-01T18:41:49.521Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: pipelining replication: peer="{Voter server-e81c06f2-30cc-1311-4c30-244868d09477 e81c06f2-30cc-1311-4c30-244868d09477}" 2024-06-01T18:41:49.521Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: pipelining replication: peer="{Voter server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9 552dff33-995c-2e1a-ffbc-a1ab9948dcc9}" 2024-06-01T18:41:50.527Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:49.521872324 +0000 UTC m=+25.020061776 (1 nodes), 652.218602ms from start of poll, 652.447258ms from cluster start. Timeout at 2024-06-01 18:41:50.527928872 +0000 UTC m=+26.026118324, 1.006056548s after stability 2024-06-01T18:41:50.528Z [INFO] creating new snapshot: path=/tmp/raft2969411156/snapshots/2-13-1717267310528.tmp 2024-06-01T18:41:50.528Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: copied to local snapshot: bytes=71 2024-06-01T18:41:50.528Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: snapshot restore progress: id=2-13-1717267310528 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:50.529Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: removing all old logs from log store 2024-06-01T18:41:50.529Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: compacting logs: from=1 to=2 2024-06-01T18:41:50.529Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: restored user snapshot: index=13 2024-06-01T18:41:50.529Z [ERROR] server-9c873484-fa6b-bbe7-c420-772d906a0820: failed to get log: index=2 error="log not found" 2024-06-01T18:41:50.529Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: aborting pipeline replication: peer="{Voter server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9 552dff33-995c-2e1a-ffbc-a1ab9948dcc9}" 2024-06-01T18:41:50.529Z [ERROR] server-9c873484-fa6b-bbe7-c420-772d906a0820: failed to get log: index=2 error="log not found" 2024-06-01T18:41:50.529Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: aborting pipeline replication: peer="{Voter server-e81c06f2-30cc-1311-4c30-244868d09477 e81c06f2-30cc-1311-4c30-244868d09477}" 2024-06-01T18:41:50.536Z [ERROR] server-9c873484-fa6b-bbe7-c420-772d906a0820: failed to get log: index=2 error="log not found" 2024-06-01T18:41:50.536Z [INFO] creating new snapshot: path=/tmp/raft1112857096/snapshots/2-13-1717267310536.tmp 2024-06-01T18:41:50.536Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:50.536Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: copied to local snapshot: bytes=71 2024-06-01T18:41:50.536Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: snapshot restore progress: id=2-13-1717267310536 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:50.536Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: removing all old logs from log store 2024-06-01T18:41:50.536Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: compacting logs: from=1 to=2 2024-06-01T18:41:50.536Z [INFO] server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9: Installed remote snapshot 2024-06-01T18:41:50.536Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: pipelining replication: peer="{Voter server-552dff33-995c-2e1a-ffbc-a1ab9948dcc9 552dff33-995c-2e1a-ffbc-a1ab9948dcc9}" 2024-06-01T18:41:50.536Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:50.539Z [ERROR] server-9c873484-fa6b-bbe7-c420-772d906a0820: failed to get log: index=2 error="log not found" 2024-06-01T18:41:50.539Z [INFO] creating new snapshot: path=/tmp/raft635447893/snapshots/2-13-1717267310539.tmp 2024-06-01T18:41:50.539Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:50.539Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: copied to local snapshot: bytes=71 2024-06-01T18:41:50.540Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: snapshot restore progress: id=2-13-1717267310539 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:50.540Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: removing all old logs from log store 2024-06-01T18:41:50.540Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: compacting logs: from=1 to=2 2024-06-01T18:41:50.540Z [INFO] server-e81c06f2-30cc-1311-4c30-244868d09477: Installed remote snapshot 2024-06-01T18:41:50.540Z [INFO] server-9c873484-fa6b-bbe7-c420-772d906a0820: pipelining replication: peer="{Voter server-e81c06f2-30cc-1311-4c30-244868d09477 e81c06f2-30cc-1311-4c30-244868d09477}" 2024-06-01T18:41:51.542Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:50.536817408 +0000 UTC m=+26.035006860 (2 nodes), 541ns from start of poll, 1.667392342s from cluster start. Timeout at 2024-06-01 18:41:51.542112558 +0000 UTC m=+27.040302010, 1.00529515s after stability 2024-06-01T18:41:51.556Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: aborting pipeline replication: peer="{Voter server-cd569554-2414-cc6b-3d10-557a91614626 cd569554-2414-cc6b-3d10-557a91614626}" 2024-06-01T18:41:51.556Z [INFO] server-d6f22738-e7ba-a75d-db97-5b3d692f6810: aborting pipeline replication: peer="{Voter server-abdea5ba-42af-cddb-bf5b-03aeb103c8a2 abdea5ba-42af-cddb-bf5b-03aeb103c8a2}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2024-06-01T18:41:51.557Z [INFO] server-e532edfa-17d7-2bc8-6536-e1edcd4de373: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e532edfa-17d7-2bc8-6536-e1edcd4de373 Address:e532edfa-17d7-2bc8-6536-e1edcd4de373} {Suffrage:Voter ID:server-6b44d8dd-5c7e-f50c-4db0-87a515239925 Address:6b44d8dd-5c7e-f50c-4db0-87a515239925} {Suffrage:Voter ID:server-492ec46f-98b9-f47b-1a06-420eb8714029 Address:492ec46f-98b9-f47b-1a06-420eb8714029}]" 2024-06-01T18:41:51.557Z [INFO] server-6b44d8dd-5c7e-f50c-4db0-87a515239925: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e532edfa-17d7-2bc8-6536-e1edcd4de373 Address:e532edfa-17d7-2bc8-6536-e1edcd4de373} {Suffrage:Voter ID:server-6b44d8dd-5c7e-f50c-4db0-87a515239925 Address:6b44d8dd-5c7e-f50c-4db0-87a515239925} {Suffrage:Voter ID:server-492ec46f-98b9-f47b-1a06-420eb8714029 Address:492ec46f-98b9-f47b-1a06-420eb8714029}]" 2024-06-01T18:41:51.558Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e532edfa-17d7-2bc8-6536-e1edcd4de373 Address:e532edfa-17d7-2bc8-6536-e1edcd4de373} {Suffrage:Voter ID:server-6b44d8dd-5c7e-f50c-4db0-87a515239925 Address:6b44d8dd-5c7e-f50c-4db0-87a515239925} {Suffrage:Voter ID:server-492ec46f-98b9-f47b-1a06-420eb8714029 Address:492ec46f-98b9-f47b-1a06-420eb8714029}]" 2024-06-01T18:41:51.558Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:51.558Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: entering follower state: follower="Node at 492ec46f-98b9-f47b-1a06-420eb8714029 [Follower]" leader-address= leader-id= 2024-06-01T18:41:51.558Z [INFO] server-6b44d8dd-5c7e-f50c-4db0-87a515239925: entering follower state: follower="Node at 6b44d8dd-5c7e-f50c-4db0-87a515239925 [Follower]" leader-address= leader-id= 2024-06-01T18:41:51.557Z [INFO] server-e532edfa-17d7-2bc8-6536-e1edcd4de373: entering follower state: follower="Node at e532edfa-17d7-2bc8-6536-e1edcd4de373 [Follower]" leader-address= leader-id= 2024-06-01T18:41:52.371Z [WARN] server-492ec46f-98b9-f47b-1a06-420eb8714029: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:52.371Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: entering candidate state: node="Node at 492ec46f-98b9-f47b-1a06-420eb8714029 [Candidate]" term=2 2024-06-01T18:41:52.371Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: election won: term=2 tally=2 2024-06-01T18:41:52.371Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: entering leader state: leader="Node at 492ec46f-98b9-f47b-1a06-420eb8714029 [Leader]" 2024-06-01T18:41:52.371Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: added peer, starting replication: peer=server-e532edfa-17d7-2bc8-6536-e1edcd4de373 2024-06-01T18:41:52.371Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: added peer, starting replication: peer=server-6b44d8dd-5c7e-f50c-4db0-87a515239925 2024-06-01T18:41:52.371Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: pipelining replication: peer="{Voter server-6b44d8dd-5c7e-f50c-4db0-87a515239925 6b44d8dd-5c7e-f50c-4db0-87a515239925}" 2024-06-01T18:41:52.371Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: pipelining replication: peer="{Voter server-e532edfa-17d7-2bc8-6536-e1edcd4de373 e532edfa-17d7-2bc8-6536-e1edcd4de373}" 2024-06-01T18:41:53.377Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:52.371908107 +0000 UTC m=+27.870097559 (1 nodes), 813.885242ms from start of poll, 814.157769ms from cluster start. Timeout at 2024-06-01 18:41:53.377300498 +0000 UTC m=+28.875489940, 1.005392381s after stability 2024-06-01T18:41:53.377Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: starting snapshot up to: index=12 2024-06-01T18:41:53.377Z [INFO] creating new snapshot: path=/tmp/raft485737575/snapshots/2-12-1717267313377.tmp 2024-06-01T18:41:53.378Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: snapshot complete up to: index=12 2024-06-01T18:41:53.378Z [INFO] creating new snapshot: path=/tmp/raft485737575/snapshots/2-23-1717267313378.tmp 2024-06-01T18:41:53.379Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: copied to local snapshot: bytes=71 2024-06-01T18:41:53.379Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: snapshot restore progress: id=2-23-1717267313378 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:53.379Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: restored user snapshot: index=23 2024-06-01T18:41:53.379Z [ERROR] server-492ec46f-98b9-f47b-1a06-420eb8714029: failed to get log: index=23 error="log not found" 2024-06-01T18:41:53.379Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: aborting pipeline replication: peer="{Voter server-6b44d8dd-5c7e-f50c-4db0-87a515239925 6b44d8dd-5c7e-f50c-4db0-87a515239925}" 2024-06-01T18:41:53.379Z [ERROR] server-492ec46f-98b9-f47b-1a06-420eb8714029: failed to get log: index=23 error="log not found" 2024-06-01T18:41:53.379Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: aborting pipeline replication: peer="{Voter server-e532edfa-17d7-2bc8-6536-e1edcd4de373 e532edfa-17d7-2bc8-6536-e1edcd4de373}" 2024-06-01T18:41:53.385Z [ERROR] server-492ec46f-98b9-f47b-1a06-420eb8714029: failed to get log: index=23 error="log not found" 2024-06-01T18:41:53.385Z [INFO] creating new snapshot: path=/tmp/raft1786843095/snapshots/2-23-1717267313385.tmp 2024-06-01T18:41:53.386Z [INFO] server-6b44d8dd-5c7e-f50c-4db0-87a515239925: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:53.386Z [INFO] server-6b44d8dd-5c7e-f50c-4db0-87a515239925: copied to local snapshot: bytes=71 2024-06-01T18:41:53.386Z [INFO] server-6b44d8dd-5c7e-f50c-4db0-87a515239925: snapshot restore progress: id=2-23-1717267313385 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:53.386Z [INFO] server-6b44d8dd-5c7e-f50c-4db0-87a515239925: Installed remote snapshot 2024-06-01T18:41:53.386Z [WARN] server-6b44d8dd-5c7e-f50c-4db0-87a515239925: skipping application of old log: index=22 2024-06-01T18:41:53.386Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: pipelining replication: peer="{Voter server-6b44d8dd-5c7e-f50c-4db0-87a515239925 6b44d8dd-5c7e-f50c-4db0-87a515239925}" 2024-06-01T18:41:53.386Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:53.387Z [ERROR] server-492ec46f-98b9-f47b-1a06-420eb8714029: failed to get log: index=23 error="log not found" 2024-06-01T18:41:53.388Z [INFO] creating new snapshot: path=/tmp/raft2155076552/snapshots/2-23-1717267313388.tmp 2024-06-01T18:41:53.388Z [INFO] server-e532edfa-17d7-2bc8-6536-e1edcd4de373: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:53.388Z [INFO] server-e532edfa-17d7-2bc8-6536-e1edcd4de373: copied to local snapshot: bytes=71 2024-06-01T18:41:53.388Z [INFO] server-e532edfa-17d7-2bc8-6536-e1edcd4de373: snapshot restore progress: id=2-23-1717267313388 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:53.388Z [INFO] server-e532edfa-17d7-2bc8-6536-e1edcd4de373: Installed remote snapshot 2024-06-01T18:41:53.388Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: pipelining replication: peer="{Voter server-e532edfa-17d7-2bc8-6536-e1edcd4de373 e532edfa-17d7-2bc8-6536-e1edcd4de373}" 2024-06-01T18:41:54.392Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:53.386760688 +0000 UTC m=+28.884950140 (2 nodes), 1.032µs from start of poll, 1.82901035s from cluster start. Timeout at 2024-06-01 18:41:54.391979525 +0000 UTC m=+29.890168977, 1.005218837s after stability 2024-06-01T18:41:54.403Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: aborting pipeline replication: peer="{Voter server-6b44d8dd-5c7e-f50c-4db0-87a515239925 6b44d8dd-5c7e-f50c-4db0-87a515239925}" 2024-06-01T18:41:54.403Z [INFO] server-492ec46f-98b9-f47b-1a06-420eb8714029: aborting pipeline replication: peer="{Voter server-e532edfa-17d7-2bc8-6536-e1edcd4de373 e532edfa-17d7-2bc8-6536-e1edcd4de373}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2024-06-01T18:41:54.405Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a71a3d90-19ce-09b4-0402-4e1f76420dd1 Address:a71a3d90-19ce-09b4-0402-4e1f76420dd1} {Suffrage:Voter ID:server-9057343f-07c1-7ce5-3f70-d75868f0b17f Address:9057343f-07c1-7ce5-3f70-d75868f0b17f} {Suffrage:Voter ID:server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf Address:7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf}]" 2024-06-01T18:41:54.405Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a71a3d90-19ce-09b4-0402-4e1f76420dd1 Address:a71a3d90-19ce-09b4-0402-4e1f76420dd1} {Suffrage:Voter ID:server-9057343f-07c1-7ce5-3f70-d75868f0b17f Address:9057343f-07c1-7ce5-3f70-d75868f0b17f} {Suffrage:Voter ID:server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf Address:7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf}]" 2024-06-01T18:41:54.405Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a71a3d90-19ce-09b4-0402-4e1f76420dd1 Address:a71a3d90-19ce-09b4-0402-4e1f76420dd1} {Suffrage:Voter ID:server-9057343f-07c1-7ce5-3f70-d75868f0b17f Address:9057343f-07c1-7ce5-3f70-d75868f0b17f} {Suffrage:Voter ID:server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf Address:7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf}]" 2024-06-01T18:41:54.405Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:54.405Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: entering follower state: follower="Node at a71a3d90-19ce-09b4-0402-4e1f76420dd1 [Follower]" leader-address= leader-id= 2024-06-01T18:41:54.405Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: entering follower state: follower="Node at 9057343f-07c1-7ce5-3f70-d75868f0b17f [Follower]" leader-address= leader-id= 2024-06-01T18:41:54.405Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: entering follower state: follower="Node at 7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf [Follower]" leader-address= leader-id= 2024-06-01T18:41:54.919Z [WARN] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:54.919Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: entering candidate state: node="Node at a71a3d90-19ce-09b4-0402-4e1f76420dd1 [Candidate]" term=2 2024-06-01T18:41:54.919Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: election won: term=2 tally=2 2024-06-01T18:41:54.919Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: entering leader state: leader="Node at a71a3d90-19ce-09b4-0402-4e1f76420dd1 [Leader]" 2024-06-01T18:41:54.919Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: added peer, starting replication: peer=server-9057343f-07c1-7ce5-3f70-d75868f0b17f 2024-06-01T18:41:54.919Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: added peer, starting replication: peer=server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf 2024-06-01T18:41:54.919Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: pipelining replication: peer="{Voter server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf 7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf}" 2024-06-01T18:41:54.919Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: pipelining replication: peer="{Voter server-9057343f-07c1-7ce5-3f70-d75868f0b17f 9057343f-07c1-7ce5-3f70-d75868f0b17f}" 2024-06-01T18:41:55.925Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:54.919286881 +0000 UTC m=+30.417476333 (1 nodes), 513.701899ms from start of poll, 514.517558ms from cluster start. Timeout at 2024-06-01 18:41:55.925289235 +0000 UTC m=+31.423478687, 1.006002354s after stability 2024-06-01T18:41:55.925Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: starting snapshot up to: index=12 2024-06-01T18:41:55.925Z [INFO] creating new snapshot: path=/tmp/raft3882261131/snapshots/2-12-1717267315925.tmp 2024-06-01T18:41:55.926Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: snapshot complete up to: index=12 2024-06-01T18:41:55.928Z [INFO] creating new snapshot: path=/tmp/raft3882261131/snapshots/2-23-1717267315928.tmp 2024-06-01T18:41:55.929Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: copied to local snapshot: bytes=71 2024-06-01T18:41:55.929Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: snapshot restore progress: id=2-23-1717267315928 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:55.929Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: removing all old logs from log store 2024-06-01T18:41:55.929Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: compacting logs: from=1 to=22 2024-06-01T18:41:55.929Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: restored user snapshot: index=23 2024-06-01T18:41:55.929Z [ERROR] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: failed to get log: index=22 error="log not found" 2024-06-01T18:41:55.929Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: aborting pipeline replication: peer="{Voter server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf 7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf}" 2024-06-01T18:41:55.929Z [ERROR] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: failed to get log: index=22 error="log not found" 2024-06-01T18:41:55.929Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: aborting pipeline replication: peer="{Voter server-9057343f-07c1-7ce5-3f70-d75868f0b17f 9057343f-07c1-7ce5-3f70-d75868f0b17f}" 2024-06-01T18:41:55.938Z [ERROR] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: failed to get log: index=22 error="log not found" 2024-06-01T18:41:55.939Z [INFO] creating new snapshot: path=/tmp/raft3826058343/snapshots/2-23-1717267315939.tmp 2024-06-01T18:41:55.939Z [ERROR] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: failed to get log: index=22 error="log not found" 2024-06-01T18:41:55.939Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:55.939Z [INFO] creating new snapshot: path=/tmp/raft411443581/snapshots/2-23-1717267315939.tmp 2024-06-01T18:41:55.939Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: copied to local snapshot: bytes=71 2024-06-01T18:41:55.939Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: snapshot restore progress: id=2-23-1717267315939 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:55.939Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:55.939Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: removing all old logs from log store 2024-06-01T18:41:55.939Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: compacting logs: from=1 to=22 2024-06-01T18:41:55.939Z [INFO] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: Installed remote snapshot 2024-06-01T18:41:55.940Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: copied to local snapshot: bytes=71 2024-06-01T18:41:55.940Z [WARN] server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf: skipping application of old log: index=22 2024-06-01T18:41:55.940Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: pipelining replication: peer="{Voter server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf 7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf}" 2024-06-01T18:41:55.940Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: snapshot restore progress: id=2-23-1717267315939 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:41:55.940Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: removing all old logs from log store 2024-06-01T18:41:55.940Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: compacting logs: from=1 to=22 2024-06-01T18:41:55.940Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:41:55.940Z [INFO] server-9057343f-07c1-7ce5-3f70-d75868f0b17f: Installed remote snapshot 2024-06-01T18:41:55.940Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: pipelining replication: peer="{Voter server-9057343f-07c1-7ce5-3f70-d75868f0b17f 9057343f-07c1-7ce5-3f70-d75868f0b17f}" 2024-06-01T18:41:56.945Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:41:55.940211143 +0000 UTC m=+31.438400835 (2 nodes), 2.384µs from start of poll, 1.53544206s from cluster start. Timeout at 2024-06-01 18:41:56.945848409 +0000 UTC m=+32.444037861, 1.005637026s after stability 2024-06-01T18:41:56.957Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: aborting pipeline replication: peer="{Voter server-7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf 7dbe94b4-75f8-96b4-c2ae-e29b000d6ebf}" 2024-06-01T18:41:56.957Z [INFO] server-a71a3d90-19ce-09b4-0402-4e1f76420dd1: aborting pipeline replication: peer="{Voter server-9057343f-07c1-7ce5-3f70-d75868f0b17f 9057343f-07c1-7ce5-3f70-d75868f0b17f}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2024-06-01T18:41:56.959Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eabf6021-0282-0e81-cbdd-9483787095a6 Address:eabf6021-0282-0e81-cbdd-9483787095a6} {Suffrage:Voter ID:server-bbb0446c-acaf-baef-bc25-97f22a9db4fe Address:bbb0446c-acaf-baef-bc25-97f22a9db4fe} {Suffrage:Voter ID:server-5eade8a3-437b-e464-ce06-48a9b3f4ba69 Address:5eade8a3-437b-e464-ce06-48a9b3f4ba69}]" 2024-06-01T18:41:56.959Z [INFO] server-bbb0446c-acaf-baef-bc25-97f22a9db4fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eabf6021-0282-0e81-cbdd-9483787095a6 Address:eabf6021-0282-0e81-cbdd-9483787095a6} {Suffrage:Voter ID:server-bbb0446c-acaf-baef-bc25-97f22a9db4fe Address:bbb0446c-acaf-baef-bc25-97f22a9db4fe} {Suffrage:Voter ID:server-5eade8a3-437b-e464-ce06-48a9b3f4ba69 Address:5eade8a3-437b-e464-ce06-48a9b3f4ba69}]" 2024-06-01T18:41:56.959Z [INFO] server-5eade8a3-437b-e464-ce06-48a9b3f4ba69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eabf6021-0282-0e81-cbdd-9483787095a6 Address:eabf6021-0282-0e81-cbdd-9483787095a6} {Suffrage:Voter ID:server-bbb0446c-acaf-baef-bc25-97f22a9db4fe Address:bbb0446c-acaf-baef-bc25-97f22a9db4fe} {Suffrage:Voter ID:server-5eade8a3-437b-e464-ce06-48a9b3f4ba69 Address:5eade8a3-437b-e464-ce06-48a9b3f4ba69}]" 2024-06-01T18:41:56.959Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:56.959Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: entering follower state: follower="Node at eabf6021-0282-0e81-cbdd-9483787095a6 [Follower]" leader-address= leader-id= 2024-06-01T18:41:56.959Z [INFO] server-bbb0446c-acaf-baef-bc25-97f22a9db4fe: entering follower state: follower="Node at bbb0446c-acaf-baef-bc25-97f22a9db4fe [Follower]" leader-address= leader-id= 2024-06-01T18:41:56.959Z [INFO] server-5eade8a3-437b-e464-ce06-48a9b3f4ba69: entering follower state: follower="Node at 5eade8a3-437b-e464-ce06-48a9b3f4ba69 [Follower]" leader-address= leader-id= 2024-06-01T18:41:57.481Z [WARN] server-eabf6021-0282-0e81-cbdd-9483787095a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:57.481Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: entering candidate state: node="Node at eabf6021-0282-0e81-cbdd-9483787095a6 [Candidate]" term=2 2024-06-01T18:41:57.481Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: election won: term=2 tally=2 2024-06-01T18:41:57.481Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: entering leader state: leader="Node at eabf6021-0282-0e81-cbdd-9483787095a6 [Leader]" 2024-06-01T18:41:57.481Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: added peer, starting replication: peer=server-bbb0446c-acaf-baef-bc25-97f22a9db4fe 2024-06-01T18:41:57.481Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: added peer, starting replication: peer=server-5eade8a3-437b-e464-ce06-48a9b3f4ba69 2024-06-01T18:41:57.481Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: pipelining replication: peer="{Voter server-bbb0446c-acaf-baef-bc25-97f22a9db4fe bbb0446c-acaf-baef-bc25-97f22a9db4fe}" 2024-06-01T18:41:57.481Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: pipelining replication: peer="{Voter server-5eade8a3-437b-e464-ce06-48a9b3f4ba69 5eade8a3-437b-e464-ce06-48a9b3f4ba69}" 2024-06-01T18:41:58.487Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:57.481322035 +0000 UTC m=+32.979511487 (1 nodes), 521.881464ms from start of poll, 522.130718ms from cluster start. Timeout at 2024-06-01 18:41:58.487081668 +0000 UTC m=+33.985271120, 1.005759633s after stability 2024-06-01T18:41:58.487Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: starting snapshot up to: index=12 2024-06-01T18:41:58.487Z [INFO] creating new snapshot: path=/tmp/raft959785263/snapshots/2-12-1717267318487.tmp 2024-06-01T18:41:58.488Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: snapshot complete up to: index=12 2024-06-01T18:41:58.489Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfe26619-2904-b154-deec-84399fce7d26 Address:cfe26619-2904-b154-deec-84399fce7d26} {Suffrage:Voter ID:server-457c7996-7417-81fd-4bcf-09766b6aeea4 Address:457c7996-7417-81fd-4bcf-09766b6aeea4} {Suffrage:Voter ID:server-892c8a9d-0ecb-59d5-4704-27c9c0009edb Address:892c8a9d-0ecb-59d5-4704-27c9c0009edb}]" 2024-06-01T18:41:58.489Z [INFO] server-457c7996-7417-81fd-4bcf-09766b6aeea4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfe26619-2904-b154-deec-84399fce7d26 Address:cfe26619-2904-b154-deec-84399fce7d26} {Suffrage:Voter ID:server-457c7996-7417-81fd-4bcf-09766b6aeea4 Address:457c7996-7417-81fd-4bcf-09766b6aeea4} {Suffrage:Voter ID:server-892c8a9d-0ecb-59d5-4704-27c9c0009edb Address:892c8a9d-0ecb-59d5-4704-27c9c0009edb}]" 2024-06-01T18:41:58.489Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: entering follower state: follower="Node at cfe26619-2904-b154-deec-84399fce7d26 [Follower]" leader-address= leader-id= 2024-06-01T18:41:58.489Z [INFO] server-457c7996-7417-81fd-4bcf-09766b6aeea4: entering follower state: follower="Node at 457c7996-7417-81fd-4bcf-09766b6aeea4 [Follower]" leader-address= leader-id= 2024-06-01T18:41:58.489Z [INFO] server-892c8a9d-0ecb-59d5-4704-27c9c0009edb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfe26619-2904-b154-deec-84399fce7d26 Address:cfe26619-2904-b154-deec-84399fce7d26} {Suffrage:Voter ID:server-457c7996-7417-81fd-4bcf-09766b6aeea4 Address:457c7996-7417-81fd-4bcf-09766b6aeea4} {Suffrage:Voter ID:server-892c8a9d-0ecb-59d5-4704-27c9c0009edb Address:892c8a9d-0ecb-59d5-4704-27c9c0009edb}]" 2024-06-01T18:41:58.489Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:41:58.489Z [INFO] server-892c8a9d-0ecb-59d5-4704-27c9c0009edb: entering follower state: follower="Node at 892c8a9d-0ecb-59d5-4704-27c9c0009edb [Follower]" leader-address= leader-id= 2024-06-01T18:41:59.018Z [WARN] server-cfe26619-2904-b154-deec-84399fce7d26: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:41:59.018Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: entering candidate state: node="Node at cfe26619-2904-b154-deec-84399fce7d26 [Candidate]" term=2 2024-06-01T18:41:59.018Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: election won: term=2 tally=2 2024-06-01T18:41:59.018Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: entering leader state: leader="Node at cfe26619-2904-b154-deec-84399fce7d26 [Leader]" 2024-06-01T18:41:59.018Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: added peer, starting replication: peer=server-457c7996-7417-81fd-4bcf-09766b6aeea4 2024-06-01T18:41:59.018Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: added peer, starting replication: peer=server-892c8a9d-0ecb-59d5-4704-27c9c0009edb 2024-06-01T18:41:59.018Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: pipelining replication: peer="{Voter server-892c8a9d-0ecb-59d5-4704-27c9c0009edb 892c8a9d-0ecb-59d5-4704-27c9c0009edb}" 2024-06-01T18:41:59.018Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: pipelining replication: peer="{Voter server-457c7996-7417-81fd-4bcf-09766b6aeea4 457c7996-7417-81fd-4bcf-09766b6aeea4}" 2024-06-01T18:42:00.023Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:41:59.018669079 +0000 UTC m=+34.516858531 (1 nodes), 529.259207ms from start of poll, 529.569585ms from cluster start. Timeout at 2024-06-01 18:42:00.023756431 +0000 UTC m=+35.521945883, 1.005087352s after stability 2024-06-01T18:42:00.024Z [INFO] creating new snapshot: path=/tmp/raft4143033016/snapshots/2-14-1717267320024.tmp 2024-06-01T18:42:00.024Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: copied to local snapshot: bytes=71 2024-06-01T18:42:00.024Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: snapshot restore progress: id=2-14-1717267320024 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:00.024Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: restored user snapshot: index=14 2024-06-01T18:42:00.024Z [ERROR] server-cfe26619-2904-b154-deec-84399fce7d26: failed to get log: index=3 error="log not found" 2024-06-01T18:42:00.024Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: aborting pipeline replication: peer="{Voter server-892c8a9d-0ecb-59d5-4704-27c9c0009edb 892c8a9d-0ecb-59d5-4704-27c9c0009edb}" 2024-06-01T18:42:00.024Z [ERROR] server-cfe26619-2904-b154-deec-84399fce7d26: failed to get log: index=3 error="log not found" 2024-06-01T18:42:00.024Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: aborting pipeline replication: peer="{Voter server-457c7996-7417-81fd-4bcf-09766b6aeea4 457c7996-7417-81fd-4bcf-09766b6aeea4}" 2024-06-01T18:42:00.031Z [ERROR] server-cfe26619-2904-b154-deec-84399fce7d26: failed to get log: index=3 error="log not found" 2024-06-01T18:42:00.032Z [INFO] creating new snapshot: path=/tmp/raft1843616864/snapshots/2-14-1717267320032.tmp 2024-06-01T18:42:00.032Z [INFO] server-892c8a9d-0ecb-59d5-4704-27c9c0009edb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:00.032Z [INFO] server-892c8a9d-0ecb-59d5-4704-27c9c0009edb: copied to local snapshot: bytes=71 2024-06-01T18:42:00.032Z [INFO] server-892c8a9d-0ecb-59d5-4704-27c9c0009edb: snapshot restore progress: id=2-14-1717267320032 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:00.032Z [INFO] server-892c8a9d-0ecb-59d5-4704-27c9c0009edb: Installed remote snapshot 2024-06-01T18:42:00.032Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: pipelining replication: peer="{Voter server-892c8a9d-0ecb-59d5-4704-27c9c0009edb 892c8a9d-0ecb-59d5-4704-27c9c0009edb}" 2024-06-01T18:42:00.033Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:00.035Z [ERROR] server-cfe26619-2904-b154-deec-84399fce7d26: failed to get log: index=3 error="log not found" 2024-06-01T18:42:00.035Z [INFO] creating new snapshot: path=/tmp/raft907871902/snapshots/2-14-1717267320035.tmp 2024-06-01T18:42:00.035Z [INFO] server-457c7996-7417-81fd-4bcf-09766b6aeea4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:00.036Z [INFO] server-457c7996-7417-81fd-4bcf-09766b6aeea4: copied to local snapshot: bytes=71 2024-06-01T18:42:00.036Z [INFO] server-457c7996-7417-81fd-4bcf-09766b6aeea4: snapshot restore progress: id=2-14-1717267320035 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:00.036Z [INFO] server-457c7996-7417-81fd-4bcf-09766b6aeea4: Installed remote snapshot 2024-06-01T18:42:00.036Z [INFO] server-cfe26619-2904-b154-deec-84399fce7d26: pipelining replication: peer="{Voter server-457c7996-7417-81fd-4bcf-09766b6aeea4 457c7996-7417-81fd-4bcf-09766b6aeea4}" 2024-06-01T18:42:01.038Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:00.033101207 +0000 UTC m=+35.531290669 (2 nodes), 1.172µs from start of poll, 1.544001723s from cluster start. Timeout at 2024-06-01 18:42:01.038145617 +0000 UTC m=+36.536335069, 1.0050444s after stability 2024-06-01T18:42:01.051Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: aborting pipeline replication: peer="{Voter server-5eade8a3-437b-e464-ce06-48a9b3f4ba69 5eade8a3-437b-e464-ce06-48a9b3f4ba69}" 2024-06-01T18:42:01.051Z [INFO] server-eabf6021-0282-0e81-cbdd-9483787095a6: aborting pipeline replication: peer="{Voter server-bbb0446c-acaf-baef-bc25-97f22a9db4fe bbb0446c-acaf-baef-bc25-97f22a9db4fe}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2024-06-01T18:42:01.052Z [INFO] server-6e12b668-506f-0177-3b49-1258029d9f76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e12b668-506f-0177-3b49-1258029d9f76 Address:6e12b668-506f-0177-3b49-1258029d9f76} {Suffrage:Voter ID:server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd Address:e5670b5d-9e63-def7-2938-e7aa69bfc9dd} {Suffrage:Voter ID:server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4 Address:c4fd4cbe-6cf8-494b-e8f2-211adfd02be4}]" 2024-06-01T18:42:01.052Z [INFO] server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e12b668-506f-0177-3b49-1258029d9f76 Address:6e12b668-506f-0177-3b49-1258029d9f76} {Suffrage:Voter ID:server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd Address:e5670b5d-9e63-def7-2938-e7aa69bfc9dd} {Suffrage:Voter ID:server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4 Address:c4fd4cbe-6cf8-494b-e8f2-211adfd02be4}]" 2024-06-01T18:42:01.052Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e12b668-506f-0177-3b49-1258029d9f76 Address:6e12b668-506f-0177-3b49-1258029d9f76} {Suffrage:Voter ID:server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd Address:e5670b5d-9e63-def7-2938-e7aa69bfc9dd} {Suffrage:Voter ID:server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4 Address:c4fd4cbe-6cf8-494b-e8f2-211adfd02be4}]" 2024-06-01T18:42:01.052Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:01.052Z [INFO] server-6e12b668-506f-0177-3b49-1258029d9f76: entering follower state: follower="Node at 6e12b668-506f-0177-3b49-1258029d9f76 [Follower]" leader-address= leader-id= 2024-06-01T18:42:01.052Z [INFO] server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd: entering follower state: follower="Node at e5670b5d-9e63-def7-2938-e7aa69bfc9dd [Follower]" leader-address= leader-id= 2024-06-01T18:42:01.052Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: entering follower state: follower="Node at c4fd4cbe-6cf8-494b-e8f2-211adfd02be4 [Follower]" leader-address= leader-id= 2024-06-01T18:42:01.651Z [WARN] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:01.651Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: entering candidate state: node="Node at c4fd4cbe-6cf8-494b-e8f2-211adfd02be4 [Candidate]" term=2 2024-06-01T18:42:01.651Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: election won: term=2 tally=2 2024-06-01T18:42:01.651Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: entering leader state: leader="Node at c4fd4cbe-6cf8-494b-e8f2-211adfd02be4 [Leader]" 2024-06-01T18:42:01.651Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: added peer, starting replication: peer=server-6e12b668-506f-0177-3b49-1258029d9f76 2024-06-01T18:42:01.651Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: added peer, starting replication: peer=server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd 2024-06-01T18:42:01.651Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: pipelining replication: peer="{Voter server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd e5670b5d-9e63-def7-2938-e7aa69bfc9dd}" 2024-06-01T18:42:01.651Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: pipelining replication: peer="{Voter server-6e12b668-506f-0177-3b49-1258029d9f76 6e12b668-506f-0177-3b49-1258029d9f76}" 2024-06-01T18:42:02.657Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:01.65128934 +0000 UTC m=+37.149478792 (1 nodes), 598.659339ms from start of poll, 598.887404ms from cluster start. Timeout at 2024-06-01 18:42:02.65705355 +0000 UTC m=+38.155243012, 1.00576422s after stability 2024-06-01T18:42:02.657Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: starting snapshot up to: index=12 2024-06-01T18:42:02.657Z [INFO] creating new snapshot: path=/tmp/raft4023250532/snapshots/2-12-1717267322657.tmp 2024-06-01T18:42:02.658Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: snapshot complete up to: index=12 2024-06-01T18:42:02.659Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7 Address:b52a7261-1d36-bcf2-f859-d0d00a1f23f7} {Suffrage:Voter ID:server-b9612829-b75c-7103-cf31-c06b1a9ce3c7 Address:b9612829-b75c-7103-cf31-c06b1a9ce3c7} {Suffrage:Voter ID:server-05b67e87-4724-53da-a6c1-cbfd26ab88f2 Address:05b67e87-4724-53da-a6c1-cbfd26ab88f2}]" 2024-06-01T18:42:02.659Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: entering follower state: follower="Node at b52a7261-1d36-bcf2-f859-d0d00a1f23f7 [Follower]" leader-address= leader-id= 2024-06-01T18:42:02.659Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7 Address:b52a7261-1d36-bcf2-f859-d0d00a1f23f7} {Suffrage:Voter ID:server-b9612829-b75c-7103-cf31-c06b1a9ce3c7 Address:b9612829-b75c-7103-cf31-c06b1a9ce3c7} {Suffrage:Voter ID:server-05b67e87-4724-53da-a6c1-cbfd26ab88f2 Address:05b67e87-4724-53da-a6c1-cbfd26ab88f2}]" 2024-06-01T18:42:02.659Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7 Address:b52a7261-1d36-bcf2-f859-d0d00a1f23f7} {Suffrage:Voter ID:server-b9612829-b75c-7103-cf31-c06b1a9ce3c7 Address:b9612829-b75c-7103-cf31-c06b1a9ce3c7} {Suffrage:Voter ID:server-05b67e87-4724-53da-a6c1-cbfd26ab88f2 Address:05b67e87-4724-53da-a6c1-cbfd26ab88f2}]" 2024-06-01T18:42:02.659Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: entering follower state: follower="Node at b9612829-b75c-7103-cf31-c06b1a9ce3c7 [Follower]" leader-address= leader-id= 2024-06-01T18:42:02.659Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:02.659Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: entering follower state: follower="Node at 05b67e87-4724-53da-a6c1-cbfd26ab88f2 [Follower]" leader-address= leader-id= 2024-06-01T18:42:03.259Z [WARN] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:03.259Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: entering candidate state: node="Node at b9612829-b75c-7103-cf31-c06b1a9ce3c7 [Candidate]" term=2 2024-06-01T18:42:03.259Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: election won: term=2 tally=2 2024-06-01T18:42:03.259Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: entering leader state: leader="Node at b9612829-b75c-7103-cf31-c06b1a9ce3c7 [Leader]" 2024-06-01T18:42:03.259Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: added peer, starting replication: peer=server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7 2024-06-01T18:42:03.259Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: added peer, starting replication: peer=server-05b67e87-4724-53da-a6c1-cbfd26ab88f2 2024-06-01T18:42:03.259Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: pipelining replication: peer="{Voter server-05b67e87-4724-53da-a6c1-cbfd26ab88f2 05b67e87-4724-53da-a6c1-cbfd26ab88f2}" 2024-06-01T18:42:03.259Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: pipelining replication: peer="{Voter server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7 b52a7261-1d36-bcf2-f859-d0d00a1f23f7}" 2024-06-01T18:42:03.259Z [WARN] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: rejecting vote request since our last term is greater: candidate=b9612829-b75c-7103-cf31-c06b1a9ce3c7 last-term=2 last-candidate-term=1 2024-06-01T18:42:04.265Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:03.259739183 +0000 UTC m=+38.757928626 (1 nodes), 600.455012ms from start of poll, 600.77699ms from cluster start. Timeout at 2024-06-01 18:42:04.26500676 +0000 UTC m=+39.763196212, 1.005267586s after stability 2024-06-01T18:42:04.265Z [INFO] creating new snapshot: path=/tmp/raft1355917671/snapshots/2-14-1717267324265.tmp 2024-06-01T18:42:04.265Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: copied to local snapshot: bytes=71 2024-06-01T18:42:04.266Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: snapshot restore progress: id=2-14-1717267324265 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:04.266Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: removing all old logs from log store 2024-06-01T18:42:04.266Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: compacting logs: from=1 to=2 2024-06-01T18:42:04.266Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: restored user snapshot: index=14 2024-06-01T18:42:04.266Z [ERROR] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: failed to get log: index=2 error="log not found" 2024-06-01T18:42:04.266Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: aborting pipeline replication: peer="{Voter server-05b67e87-4724-53da-a6c1-cbfd26ab88f2 05b67e87-4724-53da-a6c1-cbfd26ab88f2}" 2024-06-01T18:42:04.266Z [ERROR] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: failed to get log: index=2 error="log not found" 2024-06-01T18:42:04.266Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: aborting pipeline replication: peer="{Voter server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7 b52a7261-1d36-bcf2-f859-d0d00a1f23f7}" 2024-06-01T18:42:04.275Z [ERROR] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: failed to get log: index=2 error="log not found" 2024-06-01T18:42:04.275Z [INFO] creating new snapshot: path=/tmp/raft2389541161/snapshots/2-14-1717267324275.tmp 2024-06-01T18:42:04.275Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:04.276Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: copied to local snapshot: bytes=71 2024-06-01T18:42:04.276Z [ERROR] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: failed to get log: index=2 error="log not found" 2024-06-01T18:42:04.276Z [INFO] creating new snapshot: path=/tmp/raft4039182000/snapshots/2-14-1717267324276.tmp 2024-06-01T18:42:04.276Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:04.276Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: snapshot restore progress: id=2-14-1717267324275 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:04.276Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: removing all old logs from log store 2024-06-01T18:42:04.276Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: compacting logs: from=1 to=2 2024-06-01T18:42:04.276Z [INFO] server-05b67e87-4724-53da-a6c1-cbfd26ab88f2: Installed remote snapshot 2024-06-01T18:42:04.276Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: copied to local snapshot: bytes=71 2024-06-01T18:42:04.276Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: pipelining replication: peer="{Voter server-05b67e87-4724-53da-a6c1-cbfd26ab88f2 05b67e87-4724-53da-a6c1-cbfd26ab88f2}" 2024-06-01T18:42:04.276Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:04.276Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: snapshot restore progress: id=2-14-1717267324276 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:04.276Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: removing all old logs from log store 2024-06-01T18:42:04.276Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: compacting logs: from=1 to=2 2024-06-01T18:42:04.276Z [INFO] server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7: Installed remote snapshot 2024-06-01T18:42:04.277Z [INFO] server-b9612829-b75c-7103-cf31-c06b1a9ce3c7: pipelining replication: peer="{Voter server-b52a7261-1d36-bcf2-f859-d0d00a1f23f7 b52a7261-1d36-bcf2-f859-d0d00a1f23f7}" 2024-06-01T18:42:05.281Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:04.276809411 +0000 UTC m=+39.774998863 (2 nodes), 501ns from start of poll, 1.617847227s from cluster start. Timeout at 2024-06-01 18:42:05.281908723 +0000 UTC m=+40.780098185, 1.005099322s after stability 2024-06-01T18:42:05.293Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: aborting pipeline replication: peer="{Voter server-6e12b668-506f-0177-3b49-1258029d9f76 6e12b668-506f-0177-3b49-1258029d9f76}" 2024-06-01T18:42:05.293Z [INFO] server-c4fd4cbe-6cf8-494b-e8f2-211adfd02be4: aborting pipeline replication: peer="{Voter server-e5670b5d-9e63-def7-2938-e7aa69bfc9dd e5670b5d-9e63-def7-2938-e7aa69bfc9dd}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2024-06-01T18:42:05.294Z [INFO] server-99e7d35d-a809-103d-5e98-b031e3e68f57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e7d35d-a809-103d-5e98-b031e3e68f57 Address:99e7d35d-a809-103d-5e98-b031e3e68f57} {Suffrage:Voter ID:server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 Address:6868a1e4-6079-a5e5-1840-c88a9d8c24d8} {Suffrage:Voter ID:server-dc40182a-26b3-f575-bcb9-c3f6654fd63d Address:dc40182a-26b3-f575-bcb9-c3f6654fd63d}]" 2024-06-01T18:42:05.294Z [INFO] server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e7d35d-a809-103d-5e98-b031e3e68f57 Address:99e7d35d-a809-103d-5e98-b031e3e68f57} {Suffrage:Voter ID:server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 Address:6868a1e4-6079-a5e5-1840-c88a9d8c24d8} {Suffrage:Voter ID:server-dc40182a-26b3-f575-bcb9-c3f6654fd63d Address:dc40182a-26b3-f575-bcb9-c3f6654fd63d}]" 2024-06-01T18:42:05.294Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99e7d35d-a809-103d-5e98-b031e3e68f57 Address:99e7d35d-a809-103d-5e98-b031e3e68f57} {Suffrage:Voter ID:server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 Address:6868a1e4-6079-a5e5-1840-c88a9d8c24d8} {Suffrage:Voter ID:server-dc40182a-26b3-f575-bcb9-c3f6654fd63d Address:dc40182a-26b3-f575-bcb9-c3f6654fd63d}]" 2024-06-01T18:42:05.294Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:05.294Z [INFO] server-99e7d35d-a809-103d-5e98-b031e3e68f57: entering follower state: follower="Node at 99e7d35d-a809-103d-5e98-b031e3e68f57 [Follower]" leader-address= leader-id= 2024-06-01T18:42:05.294Z [INFO] server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8: entering follower state: follower="Node at 6868a1e4-6079-a5e5-1840-c88a9d8c24d8 [Follower]" leader-address= leader-id= 2024-06-01T18:42:05.294Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: entering follower state: follower="Node at dc40182a-26b3-f575-bcb9-c3f6654fd63d [Follower]" leader-address= leader-id= 2024-06-01T18:42:05.872Z [WARN] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:05.872Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: entering candidate state: node="Node at dc40182a-26b3-f575-bcb9-c3f6654fd63d [Candidate]" term=2 2024-06-01T18:42:05.872Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: election won: term=2 tally=2 2024-06-01T18:42:05.872Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: entering leader state: leader="Node at dc40182a-26b3-f575-bcb9-c3f6654fd63d [Leader]" 2024-06-01T18:42:05.872Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: added peer, starting replication: peer=server-99e7d35d-a809-103d-5e98-b031e3e68f57 2024-06-01T18:42:05.872Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: added peer, starting replication: peer=server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 2024-06-01T18:42:05.872Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: pipelining replication: peer="{Voter server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 6868a1e4-6079-a5e5-1840-c88a9d8c24d8}" 2024-06-01T18:42:05.872Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: pipelining replication: peer="{Voter server-99e7d35d-a809-103d-5e98-b031e3e68f57 99e7d35d-a809-103d-5e98-b031e3e68f57}" 2024-06-01T18:42:06.878Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:05.872704193 +0000 UTC m=+41.370893655 (1 nodes), 578.297775ms from start of poll, 578.495182ms from cluster start. Timeout at 2024-06-01 18:42:06.878105176 +0000 UTC m=+42.376294638, 1.005400983s after stability 2024-06-01T18:42:06.878Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: starting snapshot up to: index=12 2024-06-01T18:42:06.878Z [INFO] creating new snapshot: path=/tmp/raft2847040366/snapshots/2-12-1717267326878.tmp 2024-06-01T18:42:06.879Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: snapshot complete up to: index=12 2024-06-01T18:42:06.879Z [INFO] creating new snapshot: path=/tmp/raft2847040366/snapshots/2-23-1717267326879.tmp 2024-06-01T18:42:06.880Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: copied to local snapshot: bytes=71 2024-06-01T18:42:06.880Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: snapshot restore progress: id=2-23-1717267326879 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:06.880Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: restored user snapshot: index=23 2024-06-01T18:42:06.880Z [ERROR] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: failed to get log: index=23 error="log not found" 2024-06-01T18:42:06.880Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: aborting pipeline replication: peer="{Voter server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 6868a1e4-6079-a5e5-1840-c88a9d8c24d8}" 2024-06-01T18:42:06.880Z [ERROR] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: failed to get log: index=23 error="log not found" 2024-06-01T18:42:06.880Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: aborting pipeline replication: peer="{Voter server-99e7d35d-a809-103d-5e98-b031e3e68f57 99e7d35d-a809-103d-5e98-b031e3e68f57}" 2024-06-01T18:42:06.887Z [ERROR] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: failed to get log: index=23 error="log not found" 2024-06-01T18:42:06.887Z [INFO] creating new snapshot: path=/tmp/raft1515147344/snapshots/2-23-1717267326887.tmp 2024-06-01T18:42:06.888Z [INFO] server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:06.888Z [INFO] server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8: copied to local snapshot: bytes=71 2024-06-01T18:42:06.890Z [ERROR] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: failed to get log: index=23 error="log not found" 2024-06-01T18:42:06.890Z [INFO] server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8: snapshot restore progress: id=2-23-1717267326887 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:06.891Z [INFO] server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8: Installed remote snapshot 2024-06-01T18:42:06.891Z [WARN] server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8: skipping application of old log: index=22 2024-06-01T18:42:06.891Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: pipelining replication: peer="{Voter server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 6868a1e4-6079-a5e5-1840-c88a9d8c24d8}" 2024-06-01T18:42:06.891Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:06.892Z [INFO] creating new snapshot: path=/tmp/raft1832315899/snapshots/2-23-1717267326892.tmp 2024-06-01T18:42:06.892Z [INFO] server-99e7d35d-a809-103d-5e98-b031e3e68f57: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:06.892Z [INFO] server-99e7d35d-a809-103d-5e98-b031e3e68f57: copied to local snapshot: bytes=71 2024-06-01T18:42:06.892Z [INFO] server-99e7d35d-a809-103d-5e98-b031e3e68f57: snapshot restore progress: id=2-23-1717267326892 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:06.892Z [INFO] server-99e7d35d-a809-103d-5e98-b031e3e68f57: Installed remote snapshot 2024-06-01T18:42:06.892Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: pipelining replication: peer="{Voter server-99e7d35d-a809-103d-5e98-b031e3e68f57 99e7d35d-a809-103d-5e98-b031e3e68f57}" 2024-06-01T18:42:07.897Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:06.892024137 +0000 UTC m=+42.390213740 (2 nodes), 2.114µs from start of poll, 1.597815267s from cluster start. Timeout at 2024-06-01 18:42:07.897426182 +0000 UTC m=+43.395615634, 1.005401894s after stability 2024-06-01T18:42:07.908Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: aborting pipeline replication: peer="{Voter server-6868a1e4-6079-a5e5-1840-c88a9d8c24d8 6868a1e4-6079-a5e5-1840-c88a9d8c24d8}" 2024-06-01T18:42:07.908Z [INFO] server-dc40182a-26b3-f575-bcb9-c3f6654fd63d: aborting pipeline replication: peer="{Voter server-99e7d35d-a809-103d-5e98-b031e3e68f57 99e7d35d-a809-103d-5e98-b031e3e68f57}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2024-06-01T18:42:07.910Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ece575fc-4fa8-d4de-2021-346690d4ac3c Address:ece575fc-4fa8-d4de-2021-346690d4ac3c} {Suffrage:Voter ID:server-197497f0-c964-f6f0-39d3-643ebadbca1d Address:197497f0-c964-f6f0-39d3-643ebadbca1d} {Suffrage:Voter ID:server-55cba1f8-9425-fbb3-16dd-b2ace10b3736 Address:55cba1f8-9425-fbb3-16dd-b2ace10b3736}]" 2024-06-01T18:42:07.910Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ece575fc-4fa8-d4de-2021-346690d4ac3c Address:ece575fc-4fa8-d4de-2021-346690d4ac3c} {Suffrage:Voter ID:server-197497f0-c964-f6f0-39d3-643ebadbca1d Address:197497f0-c964-f6f0-39d3-643ebadbca1d} {Suffrage:Voter ID:server-55cba1f8-9425-fbb3-16dd-b2ace10b3736 Address:55cba1f8-9425-fbb3-16dd-b2ace10b3736}]" 2024-06-01T18:42:07.910Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ece575fc-4fa8-d4de-2021-346690d4ac3c Address:ece575fc-4fa8-d4de-2021-346690d4ac3c} {Suffrage:Voter ID:server-197497f0-c964-f6f0-39d3-643ebadbca1d Address:197497f0-c964-f6f0-39d3-643ebadbca1d} {Suffrage:Voter ID:server-55cba1f8-9425-fbb3-16dd-b2ace10b3736 Address:55cba1f8-9425-fbb3-16dd-b2ace10b3736}]" 2024-06-01T18:42:07.910Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:07.910Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: entering follower state: follower="Node at ece575fc-4fa8-d4de-2021-346690d4ac3c [Follower]" leader-address= leader-id= 2024-06-01T18:42:07.910Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: entering follower state: follower="Node at 197497f0-c964-f6f0-39d3-643ebadbca1d [Follower]" leader-address= leader-id= 2024-06-01T18:42:07.910Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: entering follower state: follower="Node at 55cba1f8-9425-fbb3-16dd-b2ace10b3736 [Follower]" leader-address= leader-id= 2024-06-01T18:42:08.457Z [WARN] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:08.457Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: entering candidate state: node="Node at 55cba1f8-9425-fbb3-16dd-b2ace10b3736 [Candidate]" term=2 2024-06-01T18:42:08.457Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: election won: term=2 tally=2 2024-06-01T18:42:08.457Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: entering leader state: leader="Node at 55cba1f8-9425-fbb3-16dd-b2ace10b3736 [Leader]" 2024-06-01T18:42:08.457Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: added peer, starting replication: peer=server-ece575fc-4fa8-d4de-2021-346690d4ac3c 2024-06-01T18:42:08.457Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: added peer, starting replication: peer=server-197497f0-c964-f6f0-39d3-643ebadbca1d 2024-06-01T18:42:08.458Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: pipelining replication: peer="{Voter server-197497f0-c964-f6f0-39d3-643ebadbca1d 197497f0-c964-f6f0-39d3-643ebadbca1d}" 2024-06-01T18:42:08.458Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: pipelining replication: peer="{Voter server-ece575fc-4fa8-d4de-2021-346690d4ac3c ece575fc-4fa8-d4de-2021-346690d4ac3c}" 2024-06-01T18:42:09.464Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:08.458142118 +0000 UTC m=+43.956331570 (1 nodes), 547.827464ms from start of poll, 548.057632ms from cluster start. Timeout at 2024-06-01 18:42:09.464109756 +0000 UTC m=+44.962299208, 1.005967638s after stability 2024-06-01T18:42:09.464Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: starting snapshot up to: index=12 2024-06-01T18:42:09.464Z [INFO] creating new snapshot: path=/tmp/raft444303554/snapshots/2-12-1717267329464.tmp 2024-06-01T18:42:09.465Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: snapshot complete up to: index=12 2024-06-01T18:42:09.465Z [INFO] creating new snapshot: path=/tmp/raft444303554/snapshots/2-23-1717267329465.tmp 2024-06-01T18:42:09.465Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: copied to local snapshot: bytes=71 2024-06-01T18:42:09.465Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: snapshot restore progress: id=2-23-1717267329465 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:09.466Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: removing all old logs from log store 2024-06-01T18:42:09.466Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: compacting logs: from=1 to=22 2024-06-01T18:42:09.466Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: restored user snapshot: index=23 2024-06-01T18:42:09.466Z [ERROR] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: failed to get log: index=22 error="log not found" 2024-06-01T18:42:09.466Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: aborting pipeline replication: peer="{Voter server-197497f0-c964-f6f0-39d3-643ebadbca1d 197497f0-c964-f6f0-39d3-643ebadbca1d}" 2024-06-01T18:42:09.466Z [ERROR] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: failed to get log: index=22 error="log not found" 2024-06-01T18:42:09.466Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: aborting pipeline replication: peer="{Voter server-ece575fc-4fa8-d4de-2021-346690d4ac3c ece575fc-4fa8-d4de-2021-346690d4ac3c}" 2024-06-01T18:42:09.475Z [ERROR] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: failed to get log: index=22 error="log not found" 2024-06-01T18:42:09.475Z [ERROR] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: failed to get log: index=22 error="log not found" 2024-06-01T18:42:09.475Z [INFO] creating new snapshot: path=/tmp/raft173571615/snapshots/2-23-1717267329475.tmp 2024-06-01T18:42:09.475Z [INFO] creating new snapshot: path=/tmp/raft3672577322/snapshots/2-23-1717267329475.tmp 2024-06-01T18:42:09.475Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:09.475Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:09.476Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: copied to local snapshot: bytes=71 2024-06-01T18:42:09.476Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: copied to local snapshot: bytes=71 2024-06-01T18:42:09.476Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: snapshot restore progress: id=2-23-1717267329475 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:09.476Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: removing all old logs from log store 2024-06-01T18:42:09.476Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: compacting logs: from=1 to=22 2024-06-01T18:42:09.476Z [INFO] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: Installed remote snapshot 2024-06-01T18:42:09.476Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: snapshot restore progress: id=2-23-1717267329475 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:09.476Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: removing all old logs from log store 2024-06-01T18:42:09.476Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: compacting logs: from=1 to=22 2024-06-01T18:42:09.476Z [INFO] server-197497f0-c964-f6f0-39d3-643ebadbca1d: Installed remote snapshot 2024-06-01T18:42:09.476Z [WARN] server-ece575fc-4fa8-d4de-2021-346690d4ac3c: skipping application of old log: index=22 2024-06-01T18:42:09.476Z [WARN] server-197497f0-c964-f6f0-39d3-643ebadbca1d: skipping application of old log: index=22 2024-06-01T18:42:09.476Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: pipelining replication: peer="{Voter server-ece575fc-4fa8-d4de-2021-346690d4ac3c ece575fc-4fa8-d4de-2021-346690d4ac3c}" 2024-06-01T18:42:09.476Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: pipelining replication: peer="{Voter server-197497f0-c964-f6f0-39d3-643ebadbca1d 197497f0-c964-f6f0-39d3-643ebadbca1d}" 2024-06-01T18:42:09.476Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:10.482Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:09.476270212 +0000 UTC m=+44.974459664 (2 nodes), 551ns from start of poll, 1.566185726s from cluster start. Timeout at 2024-06-01 18:42:10.482326506 +0000 UTC m=+45.980515958, 1.006056294s after stability 2024-06-01T18:42:10.493Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: aborting pipeline replication: peer="{Voter server-197497f0-c964-f6f0-39d3-643ebadbca1d 197497f0-c964-f6f0-39d3-643ebadbca1d}" 2024-06-01T18:42:10.493Z [INFO] server-55cba1f8-9425-fbb3-16dd-b2ace10b3736: aborting pipeline replication: peer="{Voter server-ece575fc-4fa8-d4de-2021-346690d4ac3c ece575fc-4fa8-d4de-2021-346690d4ac3c}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2024-06-01T18:42:10.494Z [INFO] server-5df0d9b1-6047-c0b0-91ce-c08153944f5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5df0d9b1-6047-c0b0-91ce-c08153944f5e Address:5df0d9b1-6047-c0b0-91ce-c08153944f5e} {Suffrage:Voter ID:server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55 Address:cd6fae06-0eb3-ba42-6621-c42c0fa31a55} {Suffrage:Voter ID:server-18c918b3-3457-92f9-8250-5fec2c5ee0a4 Address:18c918b3-3457-92f9-8250-5fec2c5ee0a4}]" 2024-06-01T18:42:10.494Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5df0d9b1-6047-c0b0-91ce-c08153944f5e Address:5df0d9b1-6047-c0b0-91ce-c08153944f5e} {Suffrage:Voter ID:server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55 Address:cd6fae06-0eb3-ba42-6621-c42c0fa31a55} {Suffrage:Voter ID:server-18c918b3-3457-92f9-8250-5fec2c5ee0a4 Address:18c918b3-3457-92f9-8250-5fec2c5ee0a4}]" 2024-06-01T18:42:10.494Z [INFO] server-18c918b3-3457-92f9-8250-5fec2c5ee0a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5df0d9b1-6047-c0b0-91ce-c08153944f5e Address:5df0d9b1-6047-c0b0-91ce-c08153944f5e} {Suffrage:Voter ID:server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55 Address:cd6fae06-0eb3-ba42-6621-c42c0fa31a55} {Suffrage:Voter ID:server-18c918b3-3457-92f9-8250-5fec2c5ee0a4 Address:18c918b3-3457-92f9-8250-5fec2c5ee0a4}]" 2024-06-01T18:42:10.494Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:10.494Z [INFO] server-5df0d9b1-6047-c0b0-91ce-c08153944f5e: entering follower state: follower="Node at 5df0d9b1-6047-c0b0-91ce-c08153944f5e [Follower]" leader-address= leader-id= 2024-06-01T18:42:10.494Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: entering follower state: follower="Node at cd6fae06-0eb3-ba42-6621-c42c0fa31a55 [Follower]" leader-address= leader-id= 2024-06-01T18:42:10.494Z [INFO] server-18c918b3-3457-92f9-8250-5fec2c5ee0a4: entering follower state: follower="Node at 18c918b3-3457-92f9-8250-5fec2c5ee0a4 [Follower]" leader-address= leader-id= 2024-06-01T18:42:11.177Z [WARN] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:11.177Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: entering candidate state: node="Node at cd6fae06-0eb3-ba42-6621-c42c0fa31a55 [Candidate]" term=2 2024-06-01T18:42:11.177Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: election won: term=2 tally=2 2024-06-01T18:42:11.177Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: entering leader state: leader="Node at cd6fae06-0eb3-ba42-6621-c42c0fa31a55 [Leader]" 2024-06-01T18:42:11.177Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: added peer, starting replication: peer=server-5df0d9b1-6047-c0b0-91ce-c08153944f5e 2024-06-01T18:42:11.177Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: added peer, starting replication: peer=server-18c918b3-3457-92f9-8250-5fec2c5ee0a4 2024-06-01T18:42:11.177Z [WARN] server-5df0d9b1-6047-c0b0-91ce-c08153944f5e: rejecting vote request since our last term is greater: candidate=cd6fae06-0eb3-ba42-6621-c42c0fa31a55 last-term=2 last-candidate-term=1 2024-06-01T18:42:11.177Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: pipelining replication: peer="{Voter server-5df0d9b1-6047-c0b0-91ce-c08153944f5e 5df0d9b1-6047-c0b0-91ce-c08153944f5e}" 2024-06-01T18:42:11.177Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: pipelining replication: peer="{Voter server-18c918b3-3457-92f9-8250-5fec2c5ee0a4 18c918b3-3457-92f9-8250-5fec2c5ee0a4}" 2024-06-01T18:42:12.183Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:11.177718796 +0000 UTC m=+46.675908248 (1 nodes), 682.785723ms from start of poll, 682.997376ms from cluster start. Timeout at 2024-06-01 18:42:12.183335239 +0000 UTC m=+47.681524691, 1.005616443s after stability 2024-06-01T18:42:12.183Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: starting snapshot up to: index=12 2024-06-01T18:42:12.183Z [INFO] creating new snapshot: path=/tmp/raft162425198/snapshots/2-12-1717267332183.tmp 2024-06-01T18:42:12.184Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: snapshot complete up to: index=12 2024-06-01T18:42:12.185Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd Address:68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd} {Suffrage:Voter ID:server-7df0e494-8f97-7657-ac06-1095ba3f6195 Address:7df0e494-8f97-7657-ac06-1095ba3f6195} {Suffrage:Voter ID:server-94081c8f-468c-d3b2-bb53-6acb818d7862 Address:94081c8f-468c-d3b2-bb53-6acb818d7862}]" 2024-06-01T18:42:12.185Z [INFO] server-7df0e494-8f97-7657-ac06-1095ba3f6195: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd Address:68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd} {Suffrage:Voter ID:server-7df0e494-8f97-7657-ac06-1095ba3f6195 Address:7df0e494-8f97-7657-ac06-1095ba3f6195} {Suffrage:Voter ID:server-94081c8f-468c-d3b2-bb53-6acb818d7862 Address:94081c8f-468c-d3b2-bb53-6acb818d7862}]" 2024-06-01T18:42:12.185Z [INFO] server-94081c8f-468c-d3b2-bb53-6acb818d7862: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd Address:68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd} {Suffrage:Voter ID:server-7df0e494-8f97-7657-ac06-1095ba3f6195 Address:7df0e494-8f97-7657-ac06-1095ba3f6195} {Suffrage:Voter ID:server-94081c8f-468c-d3b2-bb53-6acb818d7862 Address:94081c8f-468c-d3b2-bb53-6acb818d7862}]" 2024-06-01T18:42:12.185Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:12.185Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: entering follower state: follower="Node at 68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd [Follower]" leader-address= leader-id= 2024-06-01T18:42:12.185Z [INFO] server-7df0e494-8f97-7657-ac06-1095ba3f6195: entering follower state: follower="Node at 7df0e494-8f97-7657-ac06-1095ba3f6195 [Follower]" leader-address= leader-id= 2024-06-01T18:42:12.185Z [INFO] server-94081c8f-468c-d3b2-bb53-6acb818d7862: entering follower state: follower="Node at 94081c8f-468c-d3b2-bb53-6acb818d7862 [Follower]" leader-address= leader-id= 2024-06-01T18:42:12.713Z [WARN] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:12.713Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: entering candidate state: node="Node at 68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd [Candidate]" term=2 2024-06-01T18:42:12.714Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: election won: term=2 tally=2 2024-06-01T18:42:12.714Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: entering leader state: leader="Node at 68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd [Leader]" 2024-06-01T18:42:12.714Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: added peer, starting replication: peer=server-7df0e494-8f97-7657-ac06-1095ba3f6195 2024-06-01T18:42:12.714Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: added peer, starting replication: peer=server-94081c8f-468c-d3b2-bb53-6acb818d7862 2024-06-01T18:42:12.714Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: pipelining replication: peer="{Voter server-7df0e494-8f97-7657-ac06-1095ba3f6195 7df0e494-8f97-7657-ac06-1095ba3f6195}" 2024-06-01T18:42:12.714Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: pipelining replication: peer="{Voter server-94081c8f-468c-d3b2-bb53-6acb818d7862 94081c8f-468c-d3b2-bb53-6acb818d7862}" 2024-06-01T18:42:13.719Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:12.714046949 +0000 UTC m=+48.212236401 (1 nodes), 528.871504ms from start of poll, 529.058022ms from cluster start. Timeout at 2024-06-01 18:42:13.719826667 +0000 UTC m=+49.218016129, 1.005779728s after stability 2024-06-01T18:42:13.720Z [INFO] creating new snapshot: path=/tmp/raft1464010139/snapshots/2-15-1717267333720.tmp 2024-06-01T18:42:13.789Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: copied to local snapshot: bytes=71 2024-06-01T18:42:13.790Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: snapshot restore progress: id=2-15-1717267333720 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:13.790Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: restored user snapshot: index=15 2024-06-01T18:42:13.790Z [ERROR] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: failed to get log: index=3 error="log not found" 2024-06-01T18:42:13.790Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: aborting pipeline replication: peer="{Voter server-94081c8f-468c-d3b2-bb53-6acb818d7862 94081c8f-468c-d3b2-bb53-6acb818d7862}" 2024-06-01T18:42:13.790Z [ERROR] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: failed to get log: index=3 error="log not found" 2024-06-01T18:42:13.790Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: aborting pipeline replication: peer="{Voter server-7df0e494-8f97-7657-ac06-1095ba3f6195 7df0e494-8f97-7657-ac06-1095ba3f6195}" 2024-06-01T18:42:13.797Z [ERROR] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: failed to get log: index=3 error="log not found" 2024-06-01T18:42:13.798Z [ERROR] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: failed to get log: index=3 error="log not found" 2024-06-01T18:42:13.799Z [INFO] creating new snapshot: path=/tmp/raft232064498/snapshots/2-15-1717267333799.tmp 2024-06-01T18:42:13.821Z [INFO] creating new snapshot: path=/tmp/raft2278398863/snapshots/2-15-1717267333821.tmp 2024-06-01T18:42:13.822Z [INFO] server-7df0e494-8f97-7657-ac06-1095ba3f6195: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:13.822Z [INFO] server-94081c8f-468c-d3b2-bb53-6acb818d7862: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:13.832Z [INFO] server-7df0e494-8f97-7657-ac06-1095ba3f6195: copied to local snapshot: bytes=71 2024-06-01T18:42:13.832Z [INFO] server-7df0e494-8f97-7657-ac06-1095ba3f6195: snapshot restore progress: id=2-15-1717267333821 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:13.832Z [INFO] server-7df0e494-8f97-7657-ac06-1095ba3f6195: Installed remote snapshot 2024-06-01T18:42:13.832Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: pipelining replication: peer="{Voter server-7df0e494-8f97-7657-ac06-1095ba3f6195 7df0e494-8f97-7657-ac06-1095ba3f6195}" 2024-06-01T18:42:13.832Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:13.832Z [INFO] server-94081c8f-468c-d3b2-bb53-6acb818d7862: copied to local snapshot: bytes=71 2024-06-01T18:42:13.832Z [INFO] server-94081c8f-468c-d3b2-bb53-6acb818d7862: snapshot restore progress: id=2-15-1717267333799 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:13.832Z [INFO] server-94081c8f-468c-d3b2-bb53-6acb818d7862: Installed remote snapshot 2024-06-01T18:42:13.833Z [INFO] server-68f3b0ce-3ee7-3e90-c093-eb007f0e4bbd: pipelining replication: peer="{Voter server-94081c8f-468c-d3b2-bb53-6acb818d7862 94081c8f-468c-d3b2-bb53-6acb818d7862}" 2024-06-01T18:42:14.837Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:13.832605549 +0000 UTC m=+49.330795002 (2 nodes), 832ns from start of poll, 1.647616623s from cluster start. Timeout at 2024-06-01 18:42:14.83764579 +0000 UTC m=+50.335835242, 1.00504024s after stability 2024-06-01T18:42:14.852Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: aborting pipeline replication: peer="{Voter server-18c918b3-3457-92f9-8250-5fec2c5ee0a4 18c918b3-3457-92f9-8250-5fec2c5ee0a4}" 2024-06-01T18:42:14.852Z [INFO] server-cd6fae06-0eb3-ba42-6621-c42c0fa31a55: aborting pipeline replication: peer="{Voter server-5df0d9b1-6047-c0b0-91ce-c08153944f5e 5df0d9b1-6047-c0b0-91ce-c08153944f5e}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2024-06-01T18:42:14.853Z [INFO] server-18329a70-5194-dbea-530e-518f5161b086: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18329a70-5194-dbea-530e-518f5161b086 Address:18329a70-5194-dbea-530e-518f5161b086} {Suffrage:Voter ID:server-8a1c756a-1b6b-dd71-da30-65b0d083177a Address:8a1c756a-1b6b-dd71-da30-65b0d083177a} {Suffrage:Voter ID:server-c9ba91bf-9993-5055-02c4-9436a888f3ce Address:c9ba91bf-9993-5055-02c4-9436a888f3ce}]" 2024-06-01T18:42:14.853Z [INFO] server-8a1c756a-1b6b-dd71-da30-65b0d083177a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18329a70-5194-dbea-530e-518f5161b086 Address:18329a70-5194-dbea-530e-518f5161b086} {Suffrage:Voter ID:server-8a1c756a-1b6b-dd71-da30-65b0d083177a Address:8a1c756a-1b6b-dd71-da30-65b0d083177a} {Suffrage:Voter ID:server-c9ba91bf-9993-5055-02c4-9436a888f3ce Address:c9ba91bf-9993-5055-02c4-9436a888f3ce}]" 2024-06-01T18:42:14.853Z [INFO] server-18329a70-5194-dbea-530e-518f5161b086: entering follower state: follower="Node at 18329a70-5194-dbea-530e-518f5161b086 [Follower]" leader-address= leader-id= 2024-06-01T18:42:14.853Z [INFO] server-8a1c756a-1b6b-dd71-da30-65b0d083177a: entering follower state: follower="Node at 8a1c756a-1b6b-dd71-da30-65b0d083177a [Follower]" leader-address= leader-id= 2024-06-01T18:42:14.853Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18329a70-5194-dbea-530e-518f5161b086 Address:18329a70-5194-dbea-530e-518f5161b086} {Suffrage:Voter ID:server-8a1c756a-1b6b-dd71-da30-65b0d083177a Address:8a1c756a-1b6b-dd71-da30-65b0d083177a} {Suffrage:Voter ID:server-c9ba91bf-9993-5055-02c4-9436a888f3ce Address:c9ba91bf-9993-5055-02c4-9436a888f3ce}]" 2024-06-01T18:42:14.853Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:14.853Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: entering follower state: follower="Node at c9ba91bf-9993-5055-02c4-9436a888f3ce [Follower]" leader-address= leader-id= 2024-06-01T18:42:15.439Z [WARN] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:15.439Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: entering candidate state: node="Node at c9ba91bf-9993-5055-02c4-9436a888f3ce [Candidate]" term=2 2024-06-01T18:42:15.439Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: election won: term=2 tally=2 2024-06-01T18:42:15.439Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: entering leader state: leader="Node at c9ba91bf-9993-5055-02c4-9436a888f3ce [Leader]" 2024-06-01T18:42:15.439Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: added peer, starting replication: peer=server-18329a70-5194-dbea-530e-518f5161b086 2024-06-01T18:42:15.439Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: added peer, starting replication: peer=server-8a1c756a-1b6b-dd71-da30-65b0d083177a 2024-06-01T18:42:15.439Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: pipelining replication: peer="{Voter server-8a1c756a-1b6b-dd71-da30-65b0d083177a 8a1c756a-1b6b-dd71-da30-65b0d083177a}" 2024-06-01T18:42:15.439Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: pipelining replication: peer="{Voter server-18329a70-5194-dbea-530e-518f5161b086 18329a70-5194-dbea-530e-518f5161b086}" 2024-06-01T18:42:16.445Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:15.439500375 +0000 UTC m=+50.937689827 (1 nodes), 585.946041ms from start of poll, 586.231663ms from cluster start. Timeout at 2024-06-01 18:42:16.444927245 +0000 UTC m=+51.943116697, 1.00542687s after stability 2024-06-01T18:42:16.447Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: starting snapshot up to: index=12 2024-06-01T18:42:16.447Z [INFO] creating new snapshot: path=/tmp/raft1100205022/snapshots/2-12-1717267336447.tmp 2024-06-01T18:42:16.447Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: snapshot complete up to: index=12 2024-06-01T18:42:16.448Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a68b00e-daf1-4ae1-62cb-53bf25804a86 Address:0a68b00e-daf1-4ae1-62cb-53bf25804a86} {Suffrage:Voter ID:server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 Address:cb7d1e9f-170f-e7a4-df53-5ed5e8228b29} {Suffrage:Voter ID:server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200 Address:3fdf0a83-1359-58e6-3ecf-1e72b29b3200}]" 2024-06-01T18:42:16.448Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: entering follower state: follower="Node at 0a68b00e-daf1-4ae1-62cb-53bf25804a86 [Follower]" leader-address= leader-id= 2024-06-01T18:42:16.448Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a68b00e-daf1-4ae1-62cb-53bf25804a86 Address:0a68b00e-daf1-4ae1-62cb-53bf25804a86} {Suffrage:Voter ID:server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 Address:cb7d1e9f-170f-e7a4-df53-5ed5e8228b29} {Suffrage:Voter ID:server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200 Address:3fdf0a83-1359-58e6-3ecf-1e72b29b3200}]" 2024-06-01T18:42:16.448Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: entering follower state: follower="Node at cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 [Follower]" leader-address= leader-id= 2024-06-01T18:42:16.449Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a68b00e-daf1-4ae1-62cb-53bf25804a86 Address:0a68b00e-daf1-4ae1-62cb-53bf25804a86} {Suffrage:Voter ID:server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 Address:cb7d1e9f-170f-e7a4-df53-5ed5e8228b29} {Suffrage:Voter ID:server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200 Address:3fdf0a83-1359-58e6-3ecf-1e72b29b3200}]" 2024-06-01T18:42:16.449Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:16.449Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: entering follower state: follower="Node at 3fdf0a83-1359-58e6-3ecf-1e72b29b3200 [Follower]" leader-address= leader-id= 2024-06-01T18:42:17.037Z [WARN] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:17.037Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: entering candidate state: node="Node at 3fdf0a83-1359-58e6-3ecf-1e72b29b3200 [Candidate]" term=2 2024-06-01T18:42:17.037Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: election won: term=2 tally=2 2024-06-01T18:42:17.037Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: entering leader state: leader="Node at 3fdf0a83-1359-58e6-3ecf-1e72b29b3200 [Leader]" 2024-06-01T18:42:17.037Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: added peer, starting replication: peer=server-0a68b00e-daf1-4ae1-62cb-53bf25804a86 2024-06-01T18:42:17.037Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: added peer, starting replication: peer=server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 2024-06-01T18:42:17.037Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: pipelining replication: peer="{Voter server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 cb7d1e9f-170f-e7a4-df53-5ed5e8228b29}" 2024-06-01T18:42:17.037Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: pipelining replication: peer="{Voter server-0a68b00e-daf1-4ae1-62cb-53bf25804a86 0a68b00e-daf1-4ae1-62cb-53bf25804a86}" 2024-06-01T18:42:18.043Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:17.037497076 +0000 UTC m=+52.535686538 (1 nodes), 588.442778ms from start of poll, 588.838445ms from cluster start. Timeout at 2024-06-01 18:42:18.043228002 +0000 UTC m=+53.541417454, 1.005730916s after stability 2024-06-01T18:42:18.043Z [INFO] creating new snapshot: path=/tmp/raft3913606587/snapshots/2-15-1717267338043.tmp 2024-06-01T18:42:18.043Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: copied to local snapshot: bytes=71 2024-06-01T18:42:18.044Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: snapshot restore progress: id=2-15-1717267338043 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:18.044Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: removing all old logs from log store 2024-06-01T18:42:18.044Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: compacting logs: from=1 to=2 2024-06-01T18:42:18.044Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: restored user snapshot: index=15 2024-06-01T18:42:18.044Z [ERROR] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: failed to get log: index=2 error="log not found" 2024-06-01T18:42:18.044Z [ERROR] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: failed to get log: index=2 error="log not found" 2024-06-01T18:42:18.044Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: aborting pipeline replication: peer="{Voter server-0a68b00e-daf1-4ae1-62cb-53bf25804a86 0a68b00e-daf1-4ae1-62cb-53bf25804a86}" 2024-06-01T18:42:18.044Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: aborting pipeline replication: peer="{Voter server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 cb7d1e9f-170f-e7a4-df53-5ed5e8228b29}" 2024-06-01T18:42:18.051Z [ERROR] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: failed to get log: index=2 error="log not found" 2024-06-01T18:42:18.051Z [INFO] creating new snapshot: path=/tmp/raft2789544503/snapshots/2-15-1717267338051.tmp 2024-06-01T18:42:18.051Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:18.051Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: copied to local snapshot: bytes=71 2024-06-01T18:42:18.052Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: snapshot restore progress: id=2-15-1717267338051 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:18.052Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: removing all old logs from log store 2024-06-01T18:42:18.052Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: compacting logs: from=1 to=2 2024-06-01T18:42:18.052Z [INFO] server-0a68b00e-daf1-4ae1-62cb-53bf25804a86: Installed remote snapshot 2024-06-01T18:42:18.052Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: pipelining replication: peer="{Voter server-0a68b00e-daf1-4ae1-62cb-53bf25804a86 0a68b00e-daf1-4ae1-62cb-53bf25804a86}" 2024-06-01T18:42:18.052Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:18.052Z [ERROR] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: failed to get log: index=2 error="log not found" 2024-06-01T18:42:18.052Z [INFO] creating new snapshot: path=/tmp/raft3416479458/snapshots/2-15-1717267338052.tmp 2024-06-01T18:42:18.052Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:18.053Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: copied to local snapshot: bytes=71 2024-06-01T18:42:18.053Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: snapshot restore progress: id=2-15-1717267338052 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:18.053Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: removing all old logs from log store 2024-06-01T18:42:18.053Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: compacting logs: from=1 to=2 2024-06-01T18:42:18.053Z [INFO] server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29: Installed remote snapshot 2024-06-01T18:42:18.053Z [INFO] server-3fdf0a83-1359-58e6-3ecf-1e72b29b3200: pipelining replication: peer="{Voter server-cb7d1e9f-170f-e7a4-df53-5ed5e8228b29 cb7d1e9f-170f-e7a4-df53-5ed5e8228b29}" 2024-06-01T18:42:19.057Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:18.052223517 +0000 UTC m=+53.550412969 (2 nodes), 841ns from start of poll, 1.603564876s from cluster start. Timeout at 2024-06-01 18:42:19.057410892 +0000 UTC m=+54.555600344, 1.005187375s after stability 2024-06-01T18:42:19.069Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: aborting pipeline replication: peer="{Voter server-18329a70-5194-dbea-530e-518f5161b086 18329a70-5194-dbea-530e-518f5161b086}" 2024-06-01T18:42:19.069Z [INFO] server-c9ba91bf-9993-5055-02c4-9436a888f3ce: aborting pipeline replication: peer="{Voter server-8a1c756a-1b6b-dd71-da30-65b0d083177a 8a1c756a-1b6b-dd71-da30-65b0d083177a}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2024-06-01T18:42:19.070Z [INFO] server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 Address:bd3d965c-5c8f-655a-e8e6-e37a2b687fb7} {Suffrage:Voter ID:server-5046950a-1d36-dbca-1a20-efa629fe0669 Address:5046950a-1d36-dbca-1a20-efa629fe0669} {Suffrage:Voter ID:server-aaa4914b-73e7-d73f-90bf-8655ffadc914 Address:aaa4914b-73e7-d73f-90bf-8655ffadc914}]" 2024-06-01T18:42:19.070Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 Address:bd3d965c-5c8f-655a-e8e6-e37a2b687fb7} {Suffrage:Voter ID:server-5046950a-1d36-dbca-1a20-efa629fe0669 Address:5046950a-1d36-dbca-1a20-efa629fe0669} {Suffrage:Voter ID:server-aaa4914b-73e7-d73f-90bf-8655ffadc914 Address:aaa4914b-73e7-d73f-90bf-8655ffadc914}]" 2024-06-01T18:42:19.070Z [INFO] server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7: entering follower state: follower="Node at bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 [Follower]" leader-address= leader-id= 2024-06-01T18:42:19.070Z [INFO] server-aaa4914b-73e7-d73f-90bf-8655ffadc914: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 Address:bd3d965c-5c8f-655a-e8e6-e37a2b687fb7} {Suffrage:Voter ID:server-5046950a-1d36-dbca-1a20-efa629fe0669 Address:5046950a-1d36-dbca-1a20-efa629fe0669} {Suffrage:Voter ID:server-aaa4914b-73e7-d73f-90bf-8655ffadc914 Address:aaa4914b-73e7-d73f-90bf-8655ffadc914}]" 2024-06-01T18:42:19.070Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: entering follower state: follower="Node at 5046950a-1d36-dbca-1a20-efa629fe0669 [Follower]" leader-address= leader-id= 2024-06-01T18:42:19.070Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:19.070Z [INFO] server-aaa4914b-73e7-d73f-90bf-8655ffadc914: entering follower state: follower="Node at aaa4914b-73e7-d73f-90bf-8655ffadc914 [Follower]" leader-address= leader-id= 2024-06-01T18:42:19.630Z [WARN] server-5046950a-1d36-dbca-1a20-efa629fe0669: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:19.630Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: entering candidate state: node="Node at 5046950a-1d36-dbca-1a20-efa629fe0669 [Candidate]" term=2 2024-06-01T18:42:19.630Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: election won: term=2 tally=2 2024-06-01T18:42:19.630Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: entering leader state: leader="Node at 5046950a-1d36-dbca-1a20-efa629fe0669 [Leader]" 2024-06-01T18:42:19.630Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: added peer, starting replication: peer=server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 2024-06-01T18:42:19.630Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: added peer, starting replication: peer=server-aaa4914b-73e7-d73f-90bf-8655ffadc914 2024-06-01T18:42:19.630Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: pipelining replication: peer="{Voter server-aaa4914b-73e7-d73f-90bf-8655ffadc914 aaa4914b-73e7-d73f-90bf-8655ffadc914}" 2024-06-01T18:42:19.631Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: pipelining replication: peer="{Voter server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 bd3d965c-5c8f-655a-e8e6-e37a2b687fb7}" 2024-06-01T18:42:20.636Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:19.631122396 +0000 UTC m=+55.129311848 (1 nodes), 560.844693ms from start of poll, 561.071274ms from cluster start. Timeout at 2024-06-01 18:42:20.636811084 +0000 UTC m=+56.135000536, 1.005688688s after stability 2024-06-01T18:42:20.637Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: starting snapshot up to: index=12 2024-06-01T18:42:20.637Z [INFO] creating new snapshot: path=/tmp/raft3685633134/snapshots/2-12-1717267340637.tmp 2024-06-01T18:42:20.749Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: snapshot complete up to: index=12 2024-06-01T18:42:20.750Z [INFO] creating new snapshot: path=/tmp/raft3685633134/snapshots/2-113-1717267340750.tmp 2024-06-01T18:42:20.751Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: copied to local snapshot: bytes=71 2024-06-01T18:42:20.751Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: snapshot restore progress: id=2-113-1717267340750 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:20.751Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: restored user snapshot: index=113 2024-06-01T18:42:20.751Z [ERROR] server-5046950a-1d36-dbca-1a20-efa629fe0669: failed to get log: index=23 error="log not found" 2024-06-01T18:42:20.751Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: aborting pipeline replication: peer="{Voter server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 bd3d965c-5c8f-655a-e8e6-e37a2b687fb7}" 2024-06-01T18:42:20.751Z [ERROR] server-5046950a-1d36-dbca-1a20-efa629fe0669: failed to get log: index=23 error="log not found" 2024-06-01T18:42:20.751Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: aborting pipeline replication: peer="{Voter server-aaa4914b-73e7-d73f-90bf-8655ffadc914 aaa4914b-73e7-d73f-90bf-8655ffadc914}" 2024-06-01T18:42:20.759Z [ERROR] server-5046950a-1d36-dbca-1a20-efa629fe0669: failed to get log: index=23 error="log not found" 2024-06-01T18:42:20.760Z [ERROR] server-5046950a-1d36-dbca-1a20-efa629fe0669: failed to get log: index=23 error="log not found" 2024-06-01T18:42:20.809Z [INFO] creating new snapshot: path=/tmp/raft507902564/snapshots/2-113-1717267340809.tmp 2024-06-01T18:42:20.809Z [INFO] creating new snapshot: path=/tmp/raft4192199044/snapshots/2-113-1717267340809.tmp 2024-06-01T18:42:20.810Z [INFO] server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:20.810Z [INFO] server-aaa4914b-73e7-d73f-90bf-8655ffadc914: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:20.810Z [INFO] server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7: copied to local snapshot: bytes=71 2024-06-01T18:42:20.810Z [INFO] server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7: snapshot restore progress: id=2-113-1717267340809 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:20.810Z [INFO] server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7: Installed remote snapshot 2024-06-01T18:42:20.810Z [WARN] server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7: skipping application of old log: index=22 2024-06-01T18:42:20.810Z [INFO] server-aaa4914b-73e7-d73f-90bf-8655ffadc914: copied to local snapshot: bytes=71 2024-06-01T18:42:20.810Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: pipelining replication: peer="{Voter server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 bd3d965c-5c8f-655a-e8e6-e37a2b687fb7}" 2024-06-01T18:42:20.810Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:20.810Z [INFO] server-aaa4914b-73e7-d73f-90bf-8655ffadc914: snapshot restore progress: id=2-113-1717267340809 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:20.810Z [INFO] server-aaa4914b-73e7-d73f-90bf-8655ffadc914: Installed remote snapshot 2024-06-01T18:42:20.810Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: pipelining replication: peer="{Voter server-aaa4914b-73e7-d73f-90bf-8655ffadc914 aaa4914b-73e7-d73f-90bf-8655ffadc914}" 2024-06-01T18:42:21.816Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:20.810623862 +0000 UTC m=+56.308813314 (2 nodes), 721ns from start of poll, 1.74057274s from cluster start. Timeout at 2024-06-01 18:42:21.816192164 +0000 UTC m=+57.314381617, 1.005568303s after stability 2024-06-01T18:42:21.827Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: aborting pipeline replication: peer="{Voter server-aaa4914b-73e7-d73f-90bf-8655ffadc914 aaa4914b-73e7-d73f-90bf-8655ffadc914}" 2024-06-01T18:42:21.827Z [INFO] server-5046950a-1d36-dbca-1a20-efa629fe0669: aborting pipeline replication: peer="{Voter server-bd3d965c-5c8f-655a-e8e6-e37a2b687fb7 bd3d965c-5c8f-655a-e8e6-e37a2b687fb7}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2024-06-01T18:42:21.829Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6de7797a-675c-0728-f41b-9bcb5cedb436 Address:6de7797a-675c-0728-f41b-9bcb5cedb436} {Suffrage:Voter ID:server-53ac17e9-fa49-55b6-232e-edc2f70ac813 Address:53ac17e9-fa49-55b6-232e-edc2f70ac813} {Suffrage:Voter ID:server-367a307e-5219-df94-d1e4-15ab6178c6f9 Address:367a307e-5219-df94-d1e4-15ab6178c6f9}]" 2024-06-01T18:42:21.829Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: entering follower state: follower="Node at 6de7797a-675c-0728-f41b-9bcb5cedb436 [Follower]" leader-address= leader-id= 2024-06-01T18:42:21.829Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6de7797a-675c-0728-f41b-9bcb5cedb436 Address:6de7797a-675c-0728-f41b-9bcb5cedb436} {Suffrage:Voter ID:server-53ac17e9-fa49-55b6-232e-edc2f70ac813 Address:53ac17e9-fa49-55b6-232e-edc2f70ac813} {Suffrage:Voter ID:server-367a307e-5219-df94-d1e4-15ab6178c6f9 Address:367a307e-5219-df94-d1e4-15ab6178c6f9}]" 2024-06-01T18:42:21.829Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: entering follower state: follower="Node at 53ac17e9-fa49-55b6-232e-edc2f70ac813 [Follower]" leader-address= leader-id= 2024-06-01T18:42:21.829Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6de7797a-675c-0728-f41b-9bcb5cedb436 Address:6de7797a-675c-0728-f41b-9bcb5cedb436} {Suffrage:Voter ID:server-53ac17e9-fa49-55b6-232e-edc2f70ac813 Address:53ac17e9-fa49-55b6-232e-edc2f70ac813} {Suffrage:Voter ID:server-367a307e-5219-df94-d1e4-15ab6178c6f9 Address:367a307e-5219-df94-d1e4-15ab6178c6f9}]" 2024-06-01T18:42:21.829Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:21.829Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: entering follower state: follower="Node at 367a307e-5219-df94-d1e4-15ab6178c6f9 [Follower]" leader-address= leader-id= 2024-06-01T18:42:22.405Z [WARN] server-367a307e-5219-df94-d1e4-15ab6178c6f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:22.405Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: entering candidate state: node="Node at 367a307e-5219-df94-d1e4-15ab6178c6f9 [Candidate]" term=2 2024-06-01T18:42:22.405Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: election won: term=2 tally=2 2024-06-01T18:42:22.405Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: entering leader state: leader="Node at 367a307e-5219-df94-d1e4-15ab6178c6f9 [Leader]" 2024-06-01T18:42:22.405Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: added peer, starting replication: peer=server-6de7797a-675c-0728-f41b-9bcb5cedb436 2024-06-01T18:42:22.405Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: added peer, starting replication: peer=server-53ac17e9-fa49-55b6-232e-edc2f70ac813 2024-06-01T18:42:22.405Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: pipelining replication: peer="{Voter server-53ac17e9-fa49-55b6-232e-edc2f70ac813 53ac17e9-fa49-55b6-232e-edc2f70ac813}" 2024-06-01T18:42:22.405Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: pipelining replication: peer="{Voter server-6de7797a-675c-0728-f41b-9bcb5cedb436 6de7797a-675c-0728-f41b-9bcb5cedb436}" 2024-06-01T18:42:23.411Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:22.405736112 +0000 UTC m=+57.903925564 (1 nodes), 576.471432ms from start of poll, 576.699868ms from cluster start. Timeout at 2024-06-01 18:42:23.410996341 +0000 UTC m=+58.909185793, 1.005260229s after stability 2024-06-01T18:42:23.411Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: starting snapshot up to: index=12 2024-06-01T18:42:23.411Z [INFO] creating new snapshot: path=/tmp/raft4135863071/snapshots/2-12-1717267343411.tmp 2024-06-01T18:42:23.411Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: snapshot complete up to: index=12 2024-06-01T18:42:23.412Z [INFO] creating new snapshot: path=/tmp/raft4135863071/snapshots/2-113-1717267343412.tmp 2024-06-01T18:42:23.412Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: copied to local snapshot: bytes=71 2024-06-01T18:42:23.412Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: snapshot restore progress: id=2-113-1717267343412 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:23.412Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: removing all old logs from log store 2024-06-01T18:42:23.412Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: compacting logs: from=1 to=22 2024-06-01T18:42:23.412Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: restored user snapshot: index=113 2024-06-01T18:42:23.412Z [ERROR] server-367a307e-5219-df94-d1e4-15ab6178c6f9: failed to get log: index=22 error="log not found" 2024-06-01T18:42:23.412Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: aborting pipeline replication: peer="{Voter server-6de7797a-675c-0728-f41b-9bcb5cedb436 6de7797a-675c-0728-f41b-9bcb5cedb436}" 2024-06-01T18:42:23.412Z [ERROR] server-367a307e-5219-df94-d1e4-15ab6178c6f9: failed to get log: index=20 error="log not found" 2024-06-01T18:42:23.412Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: aborting pipeline replication: peer="{Voter server-53ac17e9-fa49-55b6-232e-edc2f70ac813 53ac17e9-fa49-55b6-232e-edc2f70ac813}" 2024-06-01T18:42:23.412Z [ERROR] server-367a307e-5219-df94-d1e4-15ab6178c6f9: failed to get log: index=20 error="log not found" 2024-06-01T18:42:23.413Z [INFO] creating new snapshot: path=/tmp/raft1573722809/snapshots/2-113-1717267343413.tmp 2024-06-01T18:42:23.413Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:23.413Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: copied to local snapshot: bytes=71 2024-06-01T18:42:23.413Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: snapshot restore progress: id=2-113-1717267343413 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:23.413Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: removing all old logs from log store 2024-06-01T18:42:23.413Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: compacting logs: from=1 to=20 2024-06-01T18:42:23.413Z [INFO] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: Installed remote snapshot 2024-06-01T18:42:23.413Z [WARN] server-53ac17e9-fa49-55b6-232e-edc2f70ac813: skipping application of old log: index=22 2024-06-01T18:42:23.413Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: pipelining replication: peer="{Voter server-53ac17e9-fa49-55b6-232e-edc2f70ac813 53ac17e9-fa49-55b6-232e-edc2f70ac813}" 2024-06-01T18:42:23.413Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:23.419Z [ERROR] server-367a307e-5219-df94-d1e4-15ab6178c6f9: failed to get log: index=22 error="log not found" 2024-06-01T18:42:23.419Z [INFO] creating new snapshot: path=/tmp/raft3799495796/snapshots/2-113-1717267343419.tmp 2024-06-01T18:42:23.420Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:23.420Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: copied to local snapshot: bytes=71 2024-06-01T18:42:23.420Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: snapshot restore progress: id=2-113-1717267343419 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:23.420Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: removing all old logs from log store 2024-06-01T18:42:23.420Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: compacting logs: from=1 to=22 2024-06-01T18:42:23.420Z [INFO] server-6de7797a-675c-0728-f41b-9bcb5cedb436: Installed remote snapshot 2024-06-01T18:42:23.420Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: pipelining replication: peer="{Voter server-6de7797a-675c-0728-f41b-9bcb5cedb436 6de7797a-675c-0728-f41b-9bcb5cedb436}" 2024-06-01T18:42:24.419Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:23.41364445 +0000 UTC m=+58.911833902 (2 nodes), 400ns from start of poll, 1.584608206s from cluster start. Timeout at 2024-06-01 18:42:24.419052885 +0000 UTC m=+59.917242337, 1.005408435s after stability 2024-06-01T18:42:24.433Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: aborting pipeline replication: peer="{Voter server-53ac17e9-fa49-55b6-232e-edc2f70ac813 53ac17e9-fa49-55b6-232e-edc2f70ac813}" 2024-06-01T18:42:24.433Z [INFO] server-367a307e-5219-df94-d1e4-15ab6178c6f9: aborting pipeline replication: peer="{Voter server-6de7797a-675c-0728-f41b-9bcb5cedb436 6de7797a-675c-0728-f41b-9bcb5cedb436}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2024-06-01T18:42:24.435Z [INFO] server-752258cb-5984-7a72-aeed-f700f63daf0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-752258cb-5984-7a72-aeed-f700f63daf0e Address:752258cb-5984-7a72-aeed-f700f63daf0e} {Suffrage:Voter ID:server-641f12be-4a79-6cba-c449-595cac148b37 Address:641f12be-4a79-6cba-c449-595cac148b37} {Suffrage:Voter ID:server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308 Address:8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308}]" 2024-06-01T18:42:24.435Z [INFO] server-641f12be-4a79-6cba-c449-595cac148b37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-752258cb-5984-7a72-aeed-f700f63daf0e Address:752258cb-5984-7a72-aeed-f700f63daf0e} {Suffrage:Voter ID:server-641f12be-4a79-6cba-c449-595cac148b37 Address:641f12be-4a79-6cba-c449-595cac148b37} {Suffrage:Voter ID:server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308 Address:8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308}]" 2024-06-01T18:42:24.435Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-752258cb-5984-7a72-aeed-f700f63daf0e Address:752258cb-5984-7a72-aeed-f700f63daf0e} {Suffrage:Voter ID:server-641f12be-4a79-6cba-c449-595cac148b37 Address:641f12be-4a79-6cba-c449-595cac148b37} {Suffrage:Voter ID:server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308 Address:8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308}]" 2024-06-01T18:42:24.435Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:24.435Z [INFO] server-752258cb-5984-7a72-aeed-f700f63daf0e: entering follower state: follower="Node at 752258cb-5984-7a72-aeed-f700f63daf0e [Follower]" leader-address= leader-id= 2024-06-01T18:42:24.435Z [INFO] server-641f12be-4a79-6cba-c449-595cac148b37: entering follower state: follower="Node at 641f12be-4a79-6cba-c449-595cac148b37 [Follower]" leader-address= leader-id= 2024-06-01T18:42:24.435Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: entering follower state: follower="Node at 8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308 [Follower]" leader-address= leader-id= 2024-06-01T18:42:24.956Z [WARN] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:24.956Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: entering candidate state: node="Node at 8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308 [Candidate]" term=2 2024-06-01T18:42:24.956Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: election won: term=2 tally=2 2024-06-01T18:42:24.956Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: entering leader state: leader="Node at 8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308 [Leader]" 2024-06-01T18:42:24.956Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: added peer, starting replication: peer=server-752258cb-5984-7a72-aeed-f700f63daf0e 2024-06-01T18:42:24.956Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: added peer, starting replication: peer=server-641f12be-4a79-6cba-c449-595cac148b37 2024-06-01T18:42:24.956Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: pipelining replication: peer="{Voter server-641f12be-4a79-6cba-c449-595cac148b37 641f12be-4a79-6cba-c449-595cac148b37}" 2024-06-01T18:42:24.956Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: pipelining replication: peer="{Voter server-752258cb-5984-7a72-aeed-f700f63daf0e 752258cb-5984-7a72-aeed-f700f63daf0e}" 2024-06-01T18:42:25.962Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:24.956539626 +0000 UTC m=+60.454729078 (1 nodes), 520.932765ms from start of poll, 521.211825ms from cluster start. Timeout at 2024-06-01 18:42:25.962568786 +0000 UTC m=+61.460758248, 1.00602917s after stability 2024-06-01T18:42:25.963Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: starting snapshot up to: index=12 2024-06-01T18:42:25.963Z [INFO] creating new snapshot: path=/tmp/raft2062182277/snapshots/2-12-1717267345963.tmp 2024-06-01T18:42:26.050Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: snapshot complete up to: index=12 2024-06-01T18:42:26.052Z [INFO] server-b54fe764-c20a-f0e3-7024-1481706ad549: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54fe764-c20a-f0e3-7024-1481706ad549 Address:b54fe764-c20a-f0e3-7024-1481706ad549} {Suffrage:Voter ID:server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79 Address:5fcbf571-85b5-874a-bd41-a7eaf0d10f79} {Suffrage:Voter ID:server-f6749622-5a9c-19a2-50b4-485d798eadd6 Address:f6749622-5a9c-19a2-50b4-485d798eadd6}]" 2024-06-01T18:42:26.052Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54fe764-c20a-f0e3-7024-1481706ad549 Address:b54fe764-c20a-f0e3-7024-1481706ad549} {Suffrage:Voter ID:server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79 Address:5fcbf571-85b5-874a-bd41-a7eaf0d10f79} {Suffrage:Voter ID:server-f6749622-5a9c-19a2-50b4-485d798eadd6 Address:f6749622-5a9c-19a2-50b4-485d798eadd6}]" 2024-06-01T18:42:26.052Z [INFO] server-b54fe764-c20a-f0e3-7024-1481706ad549: entering follower state: follower="Node at b54fe764-c20a-f0e3-7024-1481706ad549 [Follower]" leader-address= leader-id= 2024-06-01T18:42:26.052Z [INFO] server-f6749622-5a9c-19a2-50b4-485d798eadd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54fe764-c20a-f0e3-7024-1481706ad549 Address:b54fe764-c20a-f0e3-7024-1481706ad549} {Suffrage:Voter ID:server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79 Address:5fcbf571-85b5-874a-bd41-a7eaf0d10f79} {Suffrage:Voter ID:server-f6749622-5a9c-19a2-50b4-485d798eadd6 Address:f6749622-5a9c-19a2-50b4-485d798eadd6}]" 2024-06-01T18:42:26.052Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: entering follower state: follower="Node at 5fcbf571-85b5-874a-bd41-a7eaf0d10f79 [Follower]" leader-address= leader-id= 2024-06-01T18:42:26.052Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:26.052Z [INFO] server-f6749622-5a9c-19a2-50b4-485d798eadd6: entering follower state: follower="Node at f6749622-5a9c-19a2-50b4-485d798eadd6 [Follower]" leader-address= leader-id= 2024-06-01T18:42:26.685Z [WARN] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:26.685Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: entering candidate state: node="Node at 5fcbf571-85b5-874a-bd41-a7eaf0d10f79 [Candidate]" term=2 2024-06-01T18:42:26.685Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: election won: term=2 tally=2 2024-06-01T18:42:26.685Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: entering leader state: leader="Node at 5fcbf571-85b5-874a-bd41-a7eaf0d10f79 [Leader]" 2024-06-01T18:42:26.685Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: added peer, starting replication: peer=server-b54fe764-c20a-f0e3-7024-1481706ad549 2024-06-01T18:42:26.685Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: added peer, starting replication: peer=server-f6749622-5a9c-19a2-50b4-485d798eadd6 2024-06-01T18:42:26.685Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: pipelining replication: peer="{Voter server-f6749622-5a9c-19a2-50b4-485d798eadd6 f6749622-5a9c-19a2-50b4-485d798eadd6}" 2024-06-01T18:42:26.685Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: pipelining replication: peer="{Voter server-b54fe764-c20a-f0e3-7024-1481706ad549 b54fe764-c20a-f0e3-7024-1481706ad549}" 2024-06-01T18:42:27.691Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:26.685526017 +0000 UTC m=+62.183715469 (1 nodes), 633.275435ms from start of poll, 633.615959ms from cluster start. Timeout at 2024-06-01 18:42:27.69103102 +0000 UTC m=+63.189220472, 1.005505003s after stability 2024-06-01T18:42:27.691Z [INFO] creating new snapshot: path=/tmp/raft2426779030/snapshots/2-113-1717267347691.tmp 2024-06-01T18:42:27.709Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: copied to local snapshot: bytes=71 2024-06-01T18:42:27.709Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: snapshot restore progress: id=2-113-1717267347691 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:27.709Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: restored user snapshot: index=113 2024-06-01T18:42:27.709Z [ERROR] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: failed to get log: index=3 error="log not found" 2024-06-01T18:42:27.709Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: aborting pipeline replication: peer="{Voter server-f6749622-5a9c-19a2-50b4-485d798eadd6 f6749622-5a9c-19a2-50b4-485d798eadd6}" 2024-06-01T18:42:27.709Z [ERROR] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: failed to get log: index=3 error="log not found" 2024-06-01T18:42:27.709Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: aborting pipeline replication: peer="{Voter server-b54fe764-c20a-f0e3-7024-1481706ad549 b54fe764-c20a-f0e3-7024-1481706ad549}" 2024-06-01T18:42:27.715Z [ERROR] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: failed to get log: index=3 error="log not found" 2024-06-01T18:42:27.719Z [ERROR] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: failed to get log: index=3 error="log not found" 2024-06-01T18:42:27.814Z [INFO] creating new snapshot: path=/tmp/raft3852095809/snapshots/2-113-1717267347814.tmp 2024-06-01T18:42:27.814Z [INFO] server-f6749622-5a9c-19a2-50b4-485d798eadd6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:27.814Z [INFO] creating new snapshot: path=/tmp/raft3777507014/snapshots/2-113-1717267347814.tmp 2024-06-01T18:42:27.815Z [INFO] server-f6749622-5a9c-19a2-50b4-485d798eadd6: copied to local snapshot: bytes=71 2024-06-01T18:42:27.815Z [INFO] server-b54fe764-c20a-f0e3-7024-1481706ad549: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:27.815Z [INFO] server-f6749622-5a9c-19a2-50b4-485d798eadd6: snapshot restore progress: id=2-113-1717267347814 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:27.815Z [INFO] server-f6749622-5a9c-19a2-50b4-485d798eadd6: Installed remote snapshot 2024-06-01T18:42:27.815Z [INFO] server-b54fe764-c20a-f0e3-7024-1481706ad549: copied to local snapshot: bytes=71 2024-06-01T18:42:27.815Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: pipelining replication: peer="{Voter server-f6749622-5a9c-19a2-50b4-485d798eadd6 f6749622-5a9c-19a2-50b4-485d798eadd6}" 2024-06-01T18:42:27.815Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:27.815Z [INFO] server-b54fe764-c20a-f0e3-7024-1481706ad549: snapshot restore progress: id=2-113-1717267347814 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:27.815Z [INFO] server-b54fe764-c20a-f0e3-7024-1481706ad549: Installed remote snapshot 2024-06-01T18:42:27.815Z [INFO] server-5fcbf571-85b5-874a-bd41-a7eaf0d10f79: pipelining replication: peer="{Voter server-b54fe764-c20a-f0e3-7024-1481706ad549 b54fe764-c20a-f0e3-7024-1481706ad549}" 2024-06-01T18:42:28.821Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:27.815393856 +0000 UTC m=+63.313583318 (2 nodes), 972ns from start of poll, 1.763483808s from cluster start. Timeout at 2024-06-01 18:42:28.821314312 +0000 UTC m=+64.319503774, 1.005920456s after stability 2024-06-01T18:42:28.833Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: aborting pipeline replication: peer="{Voter server-641f12be-4a79-6cba-c449-595cac148b37 641f12be-4a79-6cba-c449-595cac148b37}" 2024-06-01T18:42:28.833Z [INFO] server-8f5dd474-e0b8-3c1a-d9e4-4dfa620a2308: aborting pipeline replication: peer="{Voter server-752258cb-5984-7a72-aeed-f700f63daf0e 752258cb-5984-7a72-aeed-f700f63daf0e}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2024-06-01T18:42:28.845Z [INFO] server-a55b212c-273b-4ac8-1f11-82a236e9a720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a55b212c-273b-4ac8-1f11-82a236e9a720 Address:a55b212c-273b-4ac8-1f11-82a236e9a720} {Suffrage:Voter ID:server-97e4e2d9-8306-31dc-1b3b-44e243d3f680 Address:97e4e2d9-8306-31dc-1b3b-44e243d3f680} {Suffrage:Voter ID:server-8c2b22b2-8176-95ed-76cf-59e6c0e69455 Address:8c2b22b2-8176-95ed-76cf-59e6c0e69455}]" 2024-06-01T18:42:28.846Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a55b212c-273b-4ac8-1f11-82a236e9a720 Address:a55b212c-273b-4ac8-1f11-82a236e9a720} {Suffrage:Voter ID:server-97e4e2d9-8306-31dc-1b3b-44e243d3f680 Address:97e4e2d9-8306-31dc-1b3b-44e243d3f680} {Suffrage:Voter ID:server-8c2b22b2-8176-95ed-76cf-59e6c0e69455 Address:8c2b22b2-8176-95ed-76cf-59e6c0e69455}]" 2024-06-01T18:42:28.846Z [INFO] server-8c2b22b2-8176-95ed-76cf-59e6c0e69455: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a55b212c-273b-4ac8-1f11-82a236e9a720 Address:a55b212c-273b-4ac8-1f11-82a236e9a720} {Suffrage:Voter ID:server-97e4e2d9-8306-31dc-1b3b-44e243d3f680 Address:97e4e2d9-8306-31dc-1b3b-44e243d3f680} {Suffrage:Voter ID:server-8c2b22b2-8176-95ed-76cf-59e6c0e69455 Address:8c2b22b2-8176-95ed-76cf-59e6c0e69455}]" 2024-06-01T18:42:28.846Z [INFO] server-a55b212c-273b-4ac8-1f11-82a236e9a720: entering follower state: follower="Node at a55b212c-273b-4ac8-1f11-82a236e9a720 [Follower]" leader-address= leader-id= 2024-06-01T18:42:28.846Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:28.846Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: entering follower state: follower="Node at 97e4e2d9-8306-31dc-1b3b-44e243d3f680 [Follower]" leader-address= leader-id= 2024-06-01T18:42:28.846Z [INFO] server-8c2b22b2-8176-95ed-76cf-59e6c0e69455: entering follower state: follower="Node at 8c2b22b2-8176-95ed-76cf-59e6c0e69455 [Follower]" leader-address= leader-id= 2024-06-01T18:42:29.484Z [WARN] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:29.484Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: entering candidate state: node="Node at 97e4e2d9-8306-31dc-1b3b-44e243d3f680 [Candidate]" term=2 2024-06-01T18:42:29.484Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: election won: term=2 tally=2 2024-06-01T18:42:29.484Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: entering leader state: leader="Node at 97e4e2d9-8306-31dc-1b3b-44e243d3f680 [Leader]" 2024-06-01T18:42:29.484Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: added peer, starting replication: peer=server-a55b212c-273b-4ac8-1f11-82a236e9a720 2024-06-01T18:42:29.484Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: added peer, starting replication: peer=server-8c2b22b2-8176-95ed-76cf-59e6c0e69455 2024-06-01T18:42:29.485Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: pipelining replication: peer="{Voter server-8c2b22b2-8176-95ed-76cf-59e6c0e69455 8c2b22b2-8176-95ed-76cf-59e6c0e69455}" 2024-06-01T18:42:29.485Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: pipelining replication: peer="{Voter server-a55b212c-273b-4ac8-1f11-82a236e9a720 a55b212c-273b-4ac8-1f11-82a236e9a720}" 2024-06-01T18:42:30.490Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:29.485204598 +0000 UTC m=+64.983394050 (1 nodes), 639.100088ms from start of poll, 639.459647ms from cluster start. Timeout at 2024-06-01 18:42:30.49036519 +0000 UTC m=+65.988554642, 1.005160592s after stability 2024-06-01T18:42:30.490Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: starting snapshot up to: index=12 2024-06-01T18:42:30.490Z [INFO] creating new snapshot: path=/tmp/raft1592093366/snapshots/2-12-1717267350490.tmp 2024-06-01T18:42:30.521Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: snapshot complete up to: index=12 2024-06-01T18:42:30.525Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f782783f-e298-8919-0ad4-afe3f76f3dbd Address:f782783f-e298-8919-0ad4-afe3f76f3dbd} {Suffrage:Voter ID:server-060a4569-c424-a9f3-1458-dd5c438d0de5 Address:060a4569-c424-a9f3-1458-dd5c438d0de5} {Suffrage:Voter ID:server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba Address:aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba}]" 2024-06-01T18:42:30.525Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f782783f-e298-8919-0ad4-afe3f76f3dbd Address:f782783f-e298-8919-0ad4-afe3f76f3dbd} {Suffrage:Voter ID:server-060a4569-c424-a9f3-1458-dd5c438d0de5 Address:060a4569-c424-a9f3-1458-dd5c438d0de5} {Suffrage:Voter ID:server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba Address:aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba}]" 2024-06-01T18:42:30.525Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: entering follower state: follower="Node at f782783f-e298-8919-0ad4-afe3f76f3dbd [Follower]" leader-address= leader-id= 2024-06-01T18:42:30.525Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: entering follower state: follower="Node at 060a4569-c424-a9f3-1458-dd5c438d0de5 [Follower]" leader-address= leader-id= 2024-06-01T18:42:30.525Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f782783f-e298-8919-0ad4-afe3f76f3dbd Address:f782783f-e298-8919-0ad4-afe3f76f3dbd} {Suffrage:Voter ID:server-060a4569-c424-a9f3-1458-dd5c438d0de5 Address:060a4569-c424-a9f3-1458-dd5c438d0de5} {Suffrage:Voter ID:server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba Address:aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba}]" 2024-06-01T18:42:30.525Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:30.525Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: entering follower state: follower="Node at aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba [Follower]" leader-address= leader-id= 2024-06-01T18:42:31.185Z [WARN] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:31.185Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: entering candidate state: node="Node at f782783f-e298-8919-0ad4-afe3f76f3dbd [Candidate]" term=2 2024-06-01T18:42:31.185Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: election won: term=2 tally=2 2024-06-01T18:42:31.185Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: entering leader state: leader="Node at f782783f-e298-8919-0ad4-afe3f76f3dbd [Leader]" 2024-06-01T18:42:31.185Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: added peer, starting replication: peer=server-060a4569-c424-a9f3-1458-dd5c438d0de5 2024-06-01T18:42:31.185Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: added peer, starting replication: peer=server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba 2024-06-01T18:42:31.185Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: pipelining replication: peer="{Voter server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba}" 2024-06-01T18:42:31.185Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: pipelining replication: peer="{Voter server-060a4569-c424-a9f3-1458-dd5c438d0de5 060a4569-c424-a9f3-1458-dd5c438d0de5}" 2024-06-01T18:42:32.191Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:31.185874354 +0000 UTC m=+66.684063806 (1 nodes), 660.48714ms from start of poll, 660.770166ms from cluster start. Timeout at 2024-06-01 18:42:32.191345945 +0000 UTC m=+67.689535398, 1.005471592s after stability 2024-06-01T18:42:32.191Z [INFO] creating new snapshot: path=/tmp/raft3707711568/snapshots/2-113-1717267352191.tmp 2024-06-01T18:42:32.192Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: copied to local snapshot: bytes=71 2024-06-01T18:42:32.192Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: snapshot restore progress: id=2-113-1717267352191 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:32.192Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: removing all old logs from log store 2024-06-01T18:42:32.192Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: compacting logs: from=1 to=2 2024-06-01T18:42:32.192Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: restored user snapshot: index=113 2024-06-01T18:42:32.192Z [ERROR] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: failed to get log: index=2 error="log not found" 2024-06-01T18:42:32.192Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: aborting pipeline replication: peer="{Voter server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba}" 2024-06-01T18:42:32.192Z [ERROR] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: failed to get log: index=2 error="log not found" 2024-06-01T18:42:32.192Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: aborting pipeline replication: peer="{Voter server-060a4569-c424-a9f3-1458-dd5c438d0de5 060a4569-c424-a9f3-1458-dd5c438d0de5}" 2024-06-01T18:42:32.199Z [ERROR] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: failed to get log: index=2 error="log not found" 2024-06-01T18:42:32.199Z [INFO] creating new snapshot: path=/tmp/raft1197932109/snapshots/2-113-1717267352199.tmp 2024-06-01T18:42:32.199Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:32.200Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: copied to local snapshot: bytes=71 2024-06-01T18:42:32.200Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: snapshot restore progress: id=2-113-1717267352199 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:32.200Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: removing all old logs from log store 2024-06-01T18:42:32.200Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: compacting logs: from=1 to=2 2024-06-01T18:42:32.200Z [INFO] server-060a4569-c424-a9f3-1458-dd5c438d0de5: Installed remote snapshot 2024-06-01T18:42:32.200Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: pipelining replication: peer="{Voter server-060a4569-c424-a9f3-1458-dd5c438d0de5 060a4569-c424-a9f3-1458-dd5c438d0de5}" 2024-06-01T18:42:32.200Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:32.200Z [ERROR] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: failed to get log: index=2 error="log not found" 2024-06-01T18:42:32.200Z [INFO] creating new snapshot: path=/tmp/raft2809785357/snapshots/2-113-1717267352200.tmp 2024-06-01T18:42:32.205Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:32.205Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: copied to local snapshot: bytes=71 2024-06-01T18:42:32.205Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: snapshot restore progress: id=2-113-1717267352200 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:32.205Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: removing all old logs from log store 2024-06-01T18:42:32.205Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: compacting logs: from=1 to=2 2024-06-01T18:42:32.205Z [INFO] server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba: Installed remote snapshot 2024-06-01T18:42:32.205Z [INFO] server-f782783f-e298-8919-0ad4-afe3f76f3dbd: pipelining replication: peer="{Voter server-aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba aebd2c90-cf44-4f39-2fcc-9ef99dbde2ba}" 2024-06-01T18:42:33.205Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:32.200353423 +0000 UTC m=+67.698542885 (2 nodes), 561ns from start of poll, 1.675249245s from cluster start. Timeout at 2024-06-01 18:42:33.205567264 +0000 UTC m=+68.703756716, 1.005213831s after stability 2024-06-01T18:42:33.217Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: aborting pipeline replication: peer="{Voter server-a55b212c-273b-4ac8-1f11-82a236e9a720 a55b212c-273b-4ac8-1f11-82a236e9a720}" 2024-06-01T18:42:33.217Z [INFO] server-97e4e2d9-8306-31dc-1b3b-44e243d3f680: aborting pipeline replication: peer="{Voter server-8c2b22b2-8176-95ed-76cf-59e6c0e69455 8c2b22b2-8176-95ed-76cf-59e6c0e69455}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2024-06-01T18:42:33.261Z [INFO] server-a40b2760-a8d9-beee-1cc7-e09a27b51edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a40b2760-a8d9-beee-1cc7-e09a27b51edc Address:a40b2760-a8d9-beee-1cc7-e09a27b51edc} {Suffrage:Voter ID:server-ab0b1285-e5ed-cbd6-5071-c829c6011da7 Address:ab0b1285-e5ed-cbd6-5071-c829c6011da7} {Suffrage:Voter ID:server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 Address:018016e6-1595-3a90-cfb5-6fb2621cf9f9}]" 2024-06-01T18:42:33.261Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a40b2760-a8d9-beee-1cc7-e09a27b51edc Address:a40b2760-a8d9-beee-1cc7-e09a27b51edc} {Suffrage:Voter ID:server-ab0b1285-e5ed-cbd6-5071-c829c6011da7 Address:ab0b1285-e5ed-cbd6-5071-c829c6011da7} {Suffrage:Voter ID:server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 Address:018016e6-1595-3a90-cfb5-6fb2621cf9f9}]" 2024-06-01T18:42:33.261Z [INFO] server-a40b2760-a8d9-beee-1cc7-e09a27b51edc: entering follower state: follower="Node at a40b2760-a8d9-beee-1cc7-e09a27b51edc [Follower]" leader-address= leader-id= 2024-06-01T18:42:33.261Z [INFO] server-018016e6-1595-3a90-cfb5-6fb2621cf9f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a40b2760-a8d9-beee-1cc7-e09a27b51edc Address:a40b2760-a8d9-beee-1cc7-e09a27b51edc} {Suffrage:Voter ID:server-ab0b1285-e5ed-cbd6-5071-c829c6011da7 Address:ab0b1285-e5ed-cbd6-5071-c829c6011da7} {Suffrage:Voter ID:server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 Address:018016e6-1595-3a90-cfb5-6fb2621cf9f9}]" 2024-06-01T18:42:33.261Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: entering follower state: follower="Node at ab0b1285-e5ed-cbd6-5071-c829c6011da7 [Follower]" leader-address= leader-id= 2024-06-01T18:42:33.261Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:33.261Z [INFO] server-018016e6-1595-3a90-cfb5-6fb2621cf9f9: entering follower state: follower="Node at 018016e6-1595-3a90-cfb5-6fb2621cf9f9 [Follower]" leader-address= leader-id= 2024-06-01T18:42:33.836Z [WARN] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:33.836Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: entering candidate state: node="Node at ab0b1285-e5ed-cbd6-5071-c829c6011da7 [Candidate]" term=2 2024-06-01T18:42:33.836Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: election won: term=2 tally=2 2024-06-01T18:42:33.836Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: entering leader state: leader="Node at ab0b1285-e5ed-cbd6-5071-c829c6011da7 [Leader]" 2024-06-01T18:42:33.836Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: added peer, starting replication: peer=server-a40b2760-a8d9-beee-1cc7-e09a27b51edc 2024-06-01T18:42:33.836Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: added peer, starting replication: peer=server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 2024-06-01T18:42:33.836Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: pipelining replication: peer="{Voter server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 018016e6-1595-3a90-cfb5-6fb2621cf9f9}" 2024-06-01T18:42:33.836Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: pipelining replication: peer="{Voter server-a40b2760-a8d9-beee-1cc7-e09a27b51edc a40b2760-a8d9-beee-1cc7-e09a27b51edc}" 2024-06-01T18:42:34.842Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:33.836851261 +0000 UTC m=+69.335040713 (1 nodes), 575.515793ms from start of poll, 575.82536ms from cluster start. Timeout at 2024-06-01 18:42:34.842662041 +0000 UTC m=+70.340851493, 1.00581078s after stability 2024-06-01T18:42:34.843Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: starting snapshot up to: index=12 2024-06-01T18:42:34.843Z [INFO] creating new snapshot: path=/tmp/raft1168204609/snapshots/2-12-1717267354843.tmp 2024-06-01T18:42:34.890Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: snapshot complete up to: index=12 2024-06-01T18:42:34.891Z [INFO] creating new snapshot: path=/tmp/raft1168204609/snapshots/2-1013-1717267354891.tmp 2024-06-01T18:42:34.891Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: copied to local snapshot: bytes=71 2024-06-01T18:42:34.892Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: snapshot restore progress: id=2-1013-1717267354891 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:34.892Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: restored user snapshot: index=1013 2024-06-01T18:42:34.892Z [ERROR] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: failed to get log: index=23 error="log not found" 2024-06-01T18:42:34.892Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: aborting pipeline replication: peer="{Voter server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 018016e6-1595-3a90-cfb5-6fb2621cf9f9}" 2024-06-01T18:42:34.892Z [ERROR] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: failed to get log: index=23 error="log not found" 2024-06-01T18:42:34.892Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: aborting pipeline replication: peer="{Voter server-a40b2760-a8d9-beee-1cc7-e09a27b51edc a40b2760-a8d9-beee-1cc7-e09a27b51edc}" 2024-06-01T18:42:34.901Z [ERROR] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: failed to get log: index=23 error="log not found" 2024-06-01T18:42:34.901Z [ERROR] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: failed to get log: index=23 error="log not found" 2024-06-01T18:42:34.993Z [INFO] creating new snapshot: path=/tmp/raft25328662/snapshots/2-1013-1717267354993.tmp 2024-06-01T18:42:34.993Z [INFO] server-a40b2760-a8d9-beee-1cc7-e09a27b51edc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:34.993Z [INFO] creating new snapshot: path=/tmp/raft441731218/snapshots/2-1013-1717267354993.tmp 2024-06-01T18:42:34.993Z [INFO] server-a40b2760-a8d9-beee-1cc7-e09a27b51edc: copied to local snapshot: bytes=71 2024-06-01T18:42:34.993Z [INFO] server-018016e6-1595-3a90-cfb5-6fb2621cf9f9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:34.994Z [INFO] server-a40b2760-a8d9-beee-1cc7-e09a27b51edc: snapshot restore progress: id=2-1013-1717267354993 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:34.995Z [INFO] server-018016e6-1595-3a90-cfb5-6fb2621cf9f9: copied to local snapshot: bytes=71 2024-06-01T18:42:34.995Z [INFO] server-a40b2760-a8d9-beee-1cc7-e09a27b51edc: Installed remote snapshot 2024-06-01T18:42:34.995Z [INFO] server-018016e6-1595-3a90-cfb5-6fb2621cf9f9: snapshot restore progress: id=2-1013-1717267354993 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:34.995Z [INFO] server-018016e6-1595-3a90-cfb5-6fb2621cf9f9: Installed remote snapshot 2024-06-01T18:42:34.995Z [WARN] server-a40b2760-a8d9-beee-1cc7-e09a27b51edc: skipping application of old log: index=1013 2024-06-01T18:42:34.995Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: pipelining replication: peer="{Voter server-a40b2760-a8d9-beee-1cc7-e09a27b51edc a40b2760-a8d9-beee-1cc7-e09a27b51edc}" 2024-06-01T18:42:34.995Z [WARN] server-018016e6-1595-3a90-cfb5-6fb2621cf9f9: skipping application of old log: index=22 2024-06-01T18:42:34.995Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: pipelining replication: peer="{Voter server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 018016e6-1595-3a90-cfb5-6fb2621cf9f9}" 2024-06-01T18:42:34.995Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:36.001Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:34.995922782 +0000 UTC m=+70.494112234 (2 nodes), 761ns from start of poll, 1.734896881s from cluster start. Timeout at 2024-06-01 18:42:36.001178159 +0000 UTC m=+71.499367621, 1.005255387s after stability 2024-06-01T18:42:36.013Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: aborting pipeline replication: peer="{Voter server-a40b2760-a8d9-beee-1cc7-e09a27b51edc a40b2760-a8d9-beee-1cc7-e09a27b51edc}" 2024-06-01T18:42:36.013Z [INFO] server-ab0b1285-e5ed-cbd6-5071-c829c6011da7: aborting pipeline replication: peer="{Voter server-018016e6-1595-3a90-cfb5-6fb2621cf9f9 018016e6-1595-3a90-cfb5-6fb2621cf9f9}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2024-06-01T18:42:36.025Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 Address:8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2} {Suffrage:Voter ID:server-4955c996-4fe2-d0e4-9567-4e476e2c45d4 Address:4955c996-4fe2-d0e4-9567-4e476e2c45d4} {Suffrage:Voter ID:server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 Address:92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02}]" 2024-06-01T18:42:36.025Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: entering follower state: follower="Node at 8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 [Follower]" leader-address= leader-id= 2024-06-01T18:42:36.025Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 Address:8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2} {Suffrage:Voter ID:server-4955c996-4fe2-d0e4-9567-4e476e2c45d4 Address:4955c996-4fe2-d0e4-9567-4e476e2c45d4} {Suffrage:Voter ID:server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 Address:92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02}]" 2024-06-01T18:42:36.025Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: entering follower state: follower="Node at 4955c996-4fe2-d0e4-9567-4e476e2c45d4 [Follower]" leader-address= leader-id= 2024-06-01T18:42:36.025Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 Address:8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2} {Suffrage:Voter ID:server-4955c996-4fe2-d0e4-9567-4e476e2c45d4 Address:4955c996-4fe2-d0e4-9567-4e476e2c45d4} {Suffrage:Voter ID:server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 Address:92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02}]" 2024-06-01T18:42:36.026Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:36.026Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: entering follower state: follower="Node at 92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 [Follower]" leader-address= leader-id= 2024-06-01T18:42:36.696Z [WARN] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:36.696Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: entering candidate state: node="Node at 4955c996-4fe2-d0e4-9567-4e476e2c45d4 [Candidate]" term=2 2024-06-01T18:42:36.696Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: election won: term=2 tally=2 2024-06-01T18:42:36.696Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: entering leader state: leader="Node at 4955c996-4fe2-d0e4-9567-4e476e2c45d4 [Leader]" 2024-06-01T18:42:36.696Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: added peer, starting replication: peer=server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 2024-06-01T18:42:36.696Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: added peer, starting replication: peer=server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 2024-06-01T18:42:36.696Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: pipelining replication: peer="{Voter server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02}" 2024-06-01T18:42:36.696Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: pipelining replication: peer="{Voter server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2}" 2024-06-01T18:42:37.701Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:36.696623905 +0000 UTC m=+72.194813357 (1 nodes), 670.587391ms from start of poll, 670.900955ms from cluster start. Timeout at 2024-06-01 18:42:37.701796247 +0000 UTC m=+73.199985699, 1.005172342s after stability 2024-06-01T18:42:37.702Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: starting snapshot up to: index=12 2024-06-01T18:42:37.702Z [INFO] creating new snapshot: path=/tmp/raft3493714714/snapshots/2-12-1717267357702.tmp 2024-06-01T18:42:37.741Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: snapshot complete up to: index=12 2024-06-01T18:42:37.741Z [INFO] creating new snapshot: path=/tmp/raft3493714714/snapshots/2-1013-1717267357741.tmp 2024-06-01T18:42:37.801Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: copied to local snapshot: bytes=71 2024-06-01T18:42:37.802Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: snapshot restore progress: id=2-1013-1717267357741 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:37.802Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: removing all old logs from log store 2024-06-01T18:42:37.802Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: compacting logs: from=1 to=22 2024-06-01T18:42:37.802Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: restored user snapshot: index=1013 2024-06-01T18:42:37.802Z [ERROR] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: failed to get log: index=22 error="log not found" 2024-06-01T18:42:37.802Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: aborting pipeline replication: peer="{Voter server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02}" 2024-06-01T18:42:37.802Z [ERROR] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: failed to get log: index=22 error="log not found" 2024-06-01T18:42:37.802Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: aborting pipeline replication: peer="{Voter server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2}" 2024-06-01T18:42:37.809Z [ERROR] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: failed to get log: index=22 error="log not found" 2024-06-01T18:42:37.809Z [INFO] creating new snapshot: path=/tmp/raft2702174496/snapshots/2-1013-1717267357809.tmp 2024-06-01T18:42:37.810Z [ERROR] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: failed to get log: index=22 error="log not found" 2024-06-01T18:42:37.810Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:37.810Z [INFO] creating new snapshot: path=/tmp/raft1322445537/snapshots/2-1013-1717267357810.tmp 2024-06-01T18:42:37.810Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:37.896Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: copied to local snapshot: bytes=71 2024-06-01T18:42:37.896Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: copied to local snapshot: bytes=71 2024-06-01T18:42:37.897Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: snapshot restore progress: id=2-1013-1717267357809 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:37.897Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: removing all old logs from log store 2024-06-01T18:42:37.897Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: compacting logs: from=1 to=22 2024-06-01T18:42:37.897Z [INFO] server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2: Installed remote snapshot 2024-06-01T18:42:37.897Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: snapshot restore progress: id=2-1013-1717267357810 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:37.897Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: removing all old logs from log store 2024-06-01T18:42:37.897Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: compacting logs: from=1 to=22 2024-06-01T18:42:37.897Z [INFO] server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02: Installed remote snapshot 2024-06-01T18:42:37.897Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: pipelining replication: peer="{Voter server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02}" 2024-06-01T18:42:37.897Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: pipelining replication: peer="{Voter server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2}" 2024-06-01T18:42:37.897Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:38.902Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:37.897418134 +0000 UTC m=+73.395607586 (2 nodes), 1.122µs from start of poll, 1.871695184s from cluster start. Timeout at 2024-06-01 18:42:38.902837116 +0000 UTC m=+74.401026568, 1.005418982s after stability 2024-06-01T18:42:38.915Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: aborting pipeline replication: peer="{Voter server-92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02 92bc2ce0-f5eb-56bd-c2dd-0830bb2d1b02}" 2024-06-01T18:42:38.915Z [INFO] server-4955c996-4fe2-d0e4-9567-4e476e2c45d4: aborting pipeline replication: peer="{Voter server-8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2 8a4eee0e-990e-de0b-bf98-9fcc4a5e52f2}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2024-06-01T18:42:38.916Z [INFO] server-21730e7d-45e0-e503-4cc0-02745e6a33c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21730e7d-45e0-e503-4cc0-02745e6a33c2 Address:21730e7d-45e0-e503-4cc0-02745e6a33c2} {Suffrage:Voter ID:server-ba199d4d-0c62-9cc1-31d0-20b680005bf5 Address:ba199d4d-0c62-9cc1-31d0-20b680005bf5} {Suffrage:Voter ID:server-744dfcf8-960b-59da-7fcd-f80bd7d2583e Address:744dfcf8-960b-59da-7fcd-f80bd7d2583e}]" 2024-06-01T18:42:38.916Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21730e7d-45e0-e503-4cc0-02745e6a33c2 Address:21730e7d-45e0-e503-4cc0-02745e6a33c2} {Suffrage:Voter ID:server-ba199d4d-0c62-9cc1-31d0-20b680005bf5 Address:ba199d4d-0c62-9cc1-31d0-20b680005bf5} {Suffrage:Voter ID:server-744dfcf8-960b-59da-7fcd-f80bd7d2583e Address:744dfcf8-960b-59da-7fcd-f80bd7d2583e}]" 2024-06-01T18:42:38.916Z [INFO] server-744dfcf8-960b-59da-7fcd-f80bd7d2583e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21730e7d-45e0-e503-4cc0-02745e6a33c2 Address:21730e7d-45e0-e503-4cc0-02745e6a33c2} {Suffrage:Voter ID:server-ba199d4d-0c62-9cc1-31d0-20b680005bf5 Address:ba199d4d-0c62-9cc1-31d0-20b680005bf5} {Suffrage:Voter ID:server-744dfcf8-960b-59da-7fcd-f80bd7d2583e Address:744dfcf8-960b-59da-7fcd-f80bd7d2583e}]" 2024-06-01T18:42:38.916Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:38.916Z [INFO] server-21730e7d-45e0-e503-4cc0-02745e6a33c2: entering follower state: follower="Node at 21730e7d-45e0-e503-4cc0-02745e6a33c2 [Follower]" leader-address= leader-id= 2024-06-01T18:42:38.916Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: entering follower state: follower="Node at ba199d4d-0c62-9cc1-31d0-20b680005bf5 [Follower]" leader-address= leader-id= 2024-06-01T18:42:38.916Z [INFO] server-744dfcf8-960b-59da-7fcd-f80bd7d2583e: entering follower state: follower="Node at 744dfcf8-960b-59da-7fcd-f80bd7d2583e [Follower]" leader-address= leader-id= 2024-06-01T18:42:39.469Z [WARN] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:39.469Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: entering candidate state: node="Node at ba199d4d-0c62-9cc1-31d0-20b680005bf5 [Candidate]" term=2 2024-06-01T18:42:39.469Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: election won: term=2 tally=2 2024-06-01T18:42:39.469Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: entering leader state: leader="Node at ba199d4d-0c62-9cc1-31d0-20b680005bf5 [Leader]" 2024-06-01T18:42:39.469Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: added peer, starting replication: peer=server-21730e7d-45e0-e503-4cc0-02745e6a33c2 2024-06-01T18:42:39.469Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: added peer, starting replication: peer=server-744dfcf8-960b-59da-7fcd-f80bd7d2583e 2024-06-01T18:42:39.469Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: pipelining replication: peer="{Voter server-744dfcf8-960b-59da-7fcd-f80bd7d2583e 744dfcf8-960b-59da-7fcd-f80bd7d2583e}" 2024-06-01T18:42:39.469Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: pipelining replication: peer="{Voter server-21730e7d-45e0-e503-4cc0-02745e6a33c2 21730e7d-45e0-e503-4cc0-02745e6a33c2}" 2024-06-01T18:42:40.475Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:39.469999477 +0000 UTC m=+74.968188929 (1 nodes), 553.388731ms from start of poll, 553.693479ms from cluster start. Timeout at 2024-06-01 18:42:40.475502706 +0000 UTC m=+75.973692158, 1.005503229s after stability 2024-06-01T18:42:40.475Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: starting snapshot up to: index=12 2024-06-01T18:42:40.476Z [INFO] creating new snapshot: path=/tmp/raft818206918/snapshots/2-12-1717267360476.tmp 2024-06-01T18:42:40.493Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: snapshot complete up to: index=12 2024-06-01T18:42:40.494Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aad36277-0bf0-6fed-9cd7-5fbcb7940834 Address:aad36277-0bf0-6fed-9cd7-5fbcb7940834} {Suffrage:Voter ID:server-c701170f-bee3-f0d4-39c9-b35b9230287a Address:c701170f-bee3-f0d4-39c9-b35b9230287a} {Suffrage:Voter ID:server-40ec0dcf-b298-a32a-b338-ac4a5181753b Address:40ec0dcf-b298-a32a-b338-ac4a5181753b}]" 2024-06-01T18:42:40.494Z [INFO] server-c701170f-bee3-f0d4-39c9-b35b9230287a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aad36277-0bf0-6fed-9cd7-5fbcb7940834 Address:aad36277-0bf0-6fed-9cd7-5fbcb7940834} {Suffrage:Voter ID:server-c701170f-bee3-f0d4-39c9-b35b9230287a Address:c701170f-bee3-f0d4-39c9-b35b9230287a} {Suffrage:Voter ID:server-40ec0dcf-b298-a32a-b338-ac4a5181753b Address:40ec0dcf-b298-a32a-b338-ac4a5181753b}]" 2024-06-01T18:42:40.494Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: entering follower state: follower="Node at aad36277-0bf0-6fed-9cd7-5fbcb7940834 [Follower]" leader-address= leader-id= 2024-06-01T18:42:40.494Z [INFO] server-40ec0dcf-b298-a32a-b338-ac4a5181753b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aad36277-0bf0-6fed-9cd7-5fbcb7940834 Address:aad36277-0bf0-6fed-9cd7-5fbcb7940834} {Suffrage:Voter ID:server-c701170f-bee3-f0d4-39c9-b35b9230287a Address:c701170f-bee3-f0d4-39c9-b35b9230287a} {Suffrage:Voter ID:server-40ec0dcf-b298-a32a-b338-ac4a5181753b Address:40ec0dcf-b298-a32a-b338-ac4a5181753b}]" 2024-06-01T18:42:40.494Z [INFO] server-c701170f-bee3-f0d4-39c9-b35b9230287a: entering follower state: follower="Node at c701170f-bee3-f0d4-39c9-b35b9230287a [Follower]" leader-address= leader-id= 2024-06-01T18:42:40.494Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:40.494Z [INFO] server-40ec0dcf-b298-a32a-b338-ac4a5181753b: entering follower state: follower="Node at 40ec0dcf-b298-a32a-b338-ac4a5181753b [Follower]" leader-address= leader-id= 2024-06-01T18:42:41.050Z [WARN] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:41.050Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: entering candidate state: node="Node at aad36277-0bf0-6fed-9cd7-5fbcb7940834 [Candidate]" term=2 2024-06-01T18:42:41.050Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: election won: term=2 tally=2 2024-06-01T18:42:41.050Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: entering leader state: leader="Node at aad36277-0bf0-6fed-9cd7-5fbcb7940834 [Leader]" 2024-06-01T18:42:41.050Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: added peer, starting replication: peer=server-c701170f-bee3-f0d4-39c9-b35b9230287a 2024-06-01T18:42:41.050Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: added peer, starting replication: peer=server-40ec0dcf-b298-a32a-b338-ac4a5181753b 2024-06-01T18:42:41.050Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: pipelining replication: peer="{Voter server-c701170f-bee3-f0d4-39c9-b35b9230287a c701170f-bee3-f0d4-39c9-b35b9230287a}" 2024-06-01T18:42:41.050Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: pipelining replication: peer="{Voter server-40ec0dcf-b298-a32a-b338-ac4a5181753b 40ec0dcf-b298-a32a-b338-ac4a5181753b}" 2024-06-01T18:42:42.056Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:41.050914924 +0000 UTC m=+76.549104376 (1 nodes), 556.203611ms from start of poll, 556.526862ms from cluster start. Timeout at 2024-06-01 18:42:42.056807266 +0000 UTC m=+77.554996719, 1.005892343s after stability 2024-06-01T18:42:42.057Z [INFO] creating new snapshot: path=/tmp/raft4138469489/snapshots/2-1013-1717267362057.tmp 2024-06-01T18:42:42.057Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: copied to local snapshot: bytes=71 2024-06-01T18:42:42.058Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: snapshot restore progress: id=2-1013-1717267362057 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:42.058Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: restored user snapshot: index=1013 2024-06-01T18:42:42.058Z [ERROR] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: failed to get log: index=3 error="log not found" 2024-06-01T18:42:42.058Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: aborting pipeline replication: peer="{Voter server-c701170f-bee3-f0d4-39c9-b35b9230287a c701170f-bee3-f0d4-39c9-b35b9230287a}" 2024-06-01T18:42:42.058Z [ERROR] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: failed to get log: index=3 error="log not found" 2024-06-01T18:42:42.058Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: aborting pipeline replication: peer="{Voter server-40ec0dcf-b298-a32a-b338-ac4a5181753b 40ec0dcf-b298-a32a-b338-ac4a5181753b}" 2024-06-01T18:42:42.067Z [ERROR] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: failed to get log: index=3 error="log not found" 2024-06-01T18:42:42.067Z [INFO] creating new snapshot: path=/tmp/raft3401592885/snapshots/2-1013-1717267362067.tmp 2024-06-01T18:42:42.068Z [ERROR] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: failed to get log: index=3 error="log not found" 2024-06-01T18:42:42.068Z [INFO] server-40ec0dcf-b298-a32a-b338-ac4a5181753b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:42.068Z [INFO] creating new snapshot: path=/tmp/raft3526169973/snapshots/2-1013-1717267362068.tmp 2024-06-01T18:42:42.068Z [INFO] server-c701170f-bee3-f0d4-39c9-b35b9230287a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:42.068Z [INFO] server-40ec0dcf-b298-a32a-b338-ac4a5181753b: copied to local snapshot: bytes=71 2024-06-01T18:42:42.069Z [INFO] server-40ec0dcf-b298-a32a-b338-ac4a5181753b: snapshot restore progress: id=2-1013-1717267362067 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:42.069Z [INFO] server-40ec0dcf-b298-a32a-b338-ac4a5181753b: Installed remote snapshot 2024-06-01T18:42:42.069Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: pipelining replication: peer="{Voter server-40ec0dcf-b298-a32a-b338-ac4a5181753b 40ec0dcf-b298-a32a-b338-ac4a5181753b}" 2024-06-01T18:42:42.069Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:42.074Z [INFO] server-c701170f-bee3-f0d4-39c9-b35b9230287a: copied to local snapshot: bytes=71 2024-06-01T18:42:42.074Z [INFO] server-c701170f-bee3-f0d4-39c9-b35b9230287a: snapshot restore progress: id=2-1013-1717267362068 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:42.074Z [INFO] server-c701170f-bee3-f0d4-39c9-b35b9230287a: Installed remote snapshot 2024-06-01T18:42:42.075Z [INFO] server-aad36277-0bf0-6fed-9cd7-5fbcb7940834: pipelining replication: peer="{Voter server-c701170f-bee3-f0d4-39c9-b35b9230287a c701170f-bee3-f0d4-39c9-b35b9230287a}" 2024-06-01T18:42:43.074Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:42.069362478 +0000 UTC m=+77.567551930 (2 nodes), 1.122µs from start of poll, 1.574974416s from cluster start. Timeout at 2024-06-01 18:42:43.074579413 +0000 UTC m=+78.572768865, 1.005216935s after stability 2024-06-01T18:42:43.086Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: aborting pipeline replication: peer="{Voter server-744dfcf8-960b-59da-7fcd-f80bd7d2583e 744dfcf8-960b-59da-7fcd-f80bd7d2583e}" 2024-06-01T18:42:43.086Z [INFO] server-ba199d4d-0c62-9cc1-31d0-20b680005bf5: aborting pipeline replication: peer="{Voter server-21730e7d-45e0-e503-4cc0-02745e6a33c2 21730e7d-45e0-e503-4cc0-02745e6a33c2}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2024-06-01T18:42:43.098Z [INFO] server-573ec00c-452c-aea2-bdf4-a669df306329: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573ec00c-452c-aea2-bdf4-a669df306329 Address:573ec00c-452c-aea2-bdf4-a669df306329} {Suffrage:Voter ID:server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a Address:b6cb225b-25c7-a9cc-78e9-18d87e4bc06a} {Suffrage:Voter ID:server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326 Address:aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326}]" 2024-06-01T18:42:43.098Z [INFO] server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573ec00c-452c-aea2-bdf4-a669df306329 Address:573ec00c-452c-aea2-bdf4-a669df306329} {Suffrage:Voter ID:server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a Address:b6cb225b-25c7-a9cc-78e9-18d87e4bc06a} {Suffrage:Voter ID:server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326 Address:aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326}]" 2024-06-01T18:42:43.098Z [INFO] server-573ec00c-452c-aea2-bdf4-a669df306329: entering follower state: follower="Node at 573ec00c-452c-aea2-bdf4-a669df306329 [Follower]" leader-address= leader-id= 2024-06-01T18:42:43.098Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573ec00c-452c-aea2-bdf4-a669df306329 Address:573ec00c-452c-aea2-bdf4-a669df306329} {Suffrage:Voter ID:server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a Address:b6cb225b-25c7-a9cc-78e9-18d87e4bc06a} {Suffrage:Voter ID:server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326 Address:aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326}]" 2024-06-01T18:42:43.098Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:43.098Z [INFO] server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a: entering follower state: follower="Node at b6cb225b-25c7-a9cc-78e9-18d87e4bc06a [Follower]" leader-address= leader-id= 2024-06-01T18:42:43.098Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: entering follower state: follower="Node at aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326 [Follower]" leader-address= leader-id= 2024-06-01T18:42:43.604Z [WARN] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:43.604Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: entering candidate state: node="Node at aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326 [Candidate]" term=2 2024-06-01T18:42:43.604Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: election won: term=2 tally=2 2024-06-01T18:42:43.604Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: entering leader state: leader="Node at aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326 [Leader]" 2024-06-01T18:42:43.604Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: added peer, starting replication: peer=server-573ec00c-452c-aea2-bdf4-a669df306329 2024-06-01T18:42:43.604Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: added peer, starting replication: peer=server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a 2024-06-01T18:42:43.604Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: pipelining replication: peer="{Voter server-573ec00c-452c-aea2-bdf4-a669df306329 573ec00c-452c-aea2-bdf4-a669df306329}" 2024-06-01T18:42:43.604Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: pipelining replication: peer="{Voter server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a b6cb225b-25c7-a9cc-78e9-18d87e4bc06a}" 2024-06-01T18:42:44.610Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:43.604513803 +0000 UTC m=+79.102703255 (1 nodes), 506.237026ms from start of poll, 506.563845ms from cluster start. Timeout at 2024-06-01 18:42:44.610176436 +0000 UTC m=+80.108365888, 1.005662633s after stability 2024-06-01T18:42:44.610Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: starting snapshot up to: index=12 2024-06-01T18:42:44.610Z [INFO] creating new snapshot: path=/tmp/raft3841900552/snapshots/2-12-1717267364610.tmp 2024-06-01T18:42:44.623Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: snapshot complete up to: index=12 2024-06-01T18:42:44.624Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589 Address:aa1f3998-4c4c-d1d2-7acb-c9c67830c589} {Suffrage:Voter ID:server-6f91d20d-13e6-c092-6615-7b3994deed7c Address:6f91d20d-13e6-c092-6615-7b3994deed7c} {Suffrage:Voter ID:server-76b32d50-29cf-9396-f2b9-6866fbdbe262 Address:76b32d50-29cf-9396-f2b9-6866fbdbe262}]" 2024-06-01T18:42:44.624Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589 Address:aa1f3998-4c4c-d1d2-7acb-c9c67830c589} {Suffrage:Voter ID:server-6f91d20d-13e6-c092-6615-7b3994deed7c Address:6f91d20d-13e6-c092-6615-7b3994deed7c} {Suffrage:Voter ID:server-76b32d50-29cf-9396-f2b9-6866fbdbe262 Address:76b32d50-29cf-9396-f2b9-6866fbdbe262}]" 2024-06-01T18:42:44.624Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: entering follower state: follower="Node at aa1f3998-4c4c-d1d2-7acb-c9c67830c589 [Follower]" leader-address= leader-id= 2024-06-01T18:42:44.624Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589 Address:aa1f3998-4c4c-d1d2-7acb-c9c67830c589} {Suffrage:Voter ID:server-6f91d20d-13e6-c092-6615-7b3994deed7c Address:6f91d20d-13e6-c092-6615-7b3994deed7c} {Suffrage:Voter ID:server-76b32d50-29cf-9396-f2b9-6866fbdbe262 Address:76b32d50-29cf-9396-f2b9-6866fbdbe262}]" 2024-06-01T18:42:44.624Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:44.624Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: entering follower state: follower="Node at 6f91d20d-13e6-c092-6615-7b3994deed7c [Follower]" leader-address= leader-id= 2024-06-01T18:42:44.624Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: entering follower state: follower="Node at 76b32d50-29cf-9396-f2b9-6866fbdbe262 [Follower]" leader-address= leader-id= 2024-06-01T18:42:45.354Z [WARN] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:45.354Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: entering candidate state: node="Node at 76b32d50-29cf-9396-f2b9-6866fbdbe262 [Candidate]" term=2 2024-06-01T18:42:45.354Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: election won: term=2 tally=2 2024-06-01T18:42:45.354Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: entering leader state: leader="Node at 76b32d50-29cf-9396-f2b9-6866fbdbe262 [Leader]" 2024-06-01T18:42:45.354Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: added peer, starting replication: peer=server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589 2024-06-01T18:42:45.354Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: added peer, starting replication: peer=server-6f91d20d-13e6-c092-6615-7b3994deed7c 2024-06-01T18:42:45.354Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: pipelining replication: peer="{Voter server-6f91d20d-13e6-c092-6615-7b3994deed7c 6f91d20d-13e6-c092-6615-7b3994deed7c}" 2024-06-01T18:42:45.354Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: pipelining replication: peer="{Voter server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589 aa1f3998-4c4c-d1d2-7acb-c9c67830c589}" 2024-06-01T18:42:46.360Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:45.354705021 +0000 UTC m=+80.852894473 (1 nodes), 730.111186ms from start of poll, 730.380157ms from cluster start. Timeout at 2024-06-01 18:42:46.360270063 +0000 UTC m=+81.858459515, 1.005565042s after stability 2024-06-01T18:42:46.360Z [INFO] creating new snapshot: path=/tmp/raft369117062/snapshots/2-1013-1717267366360.tmp 2024-06-01T18:42:46.377Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: copied to local snapshot: bytes=71 2024-06-01T18:42:46.377Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: snapshot restore progress: id=2-1013-1717267366360 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:46.377Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: removing all old logs from log store 2024-06-01T18:42:46.377Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: compacting logs: from=1 to=2 2024-06-01T18:42:46.377Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: restored user snapshot: index=1013 2024-06-01T18:42:46.377Z [ERROR] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: failed to get log: index=2 error="log not found" 2024-06-01T18:42:46.377Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: aborting pipeline replication: peer="{Voter server-6f91d20d-13e6-c092-6615-7b3994deed7c 6f91d20d-13e6-c092-6615-7b3994deed7c}" 2024-06-01T18:42:46.377Z [ERROR] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: failed to get log: index=2 error="log not found" 2024-06-01T18:42:46.377Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: aborting pipeline replication: peer="{Voter server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589 aa1f3998-4c4c-d1d2-7acb-c9c67830c589}" 2024-06-01T18:42:46.377Z [ERROR] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: failed to get log: index=2 error="log not found" 2024-06-01T18:42:46.378Z [INFO] creating new snapshot: path=/tmp/raft3825935025/snapshots/2-1013-1717267366378.tmp 2024-06-01T18:42:46.379Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:46.379Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: copied to local snapshot: bytes=71 2024-06-01T18:42:46.379Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: snapshot restore progress: id=2-1013-1717267366378 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:46.379Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: removing all old logs from log store 2024-06-01T18:42:46.379Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: compacting logs: from=1 to=2 2024-06-01T18:42:46.379Z [INFO] server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589: Installed remote snapshot 2024-06-01T18:42:46.379Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: pipelining replication: peer="{Voter server-aa1f3998-4c4c-d1d2-7acb-c9c67830c589 aa1f3998-4c4c-d1d2-7acb-c9c67830c589}" 2024-06-01T18:42:46.379Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:46.384Z [ERROR] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: failed to get log: index=2 error="log not found" 2024-06-01T18:42:46.385Z [INFO] creating new snapshot: path=/tmp/raft2158040657/snapshots/2-1013-1717267366385.tmp 2024-06-01T18:42:46.385Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:46.385Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: copied to local snapshot: bytes=71 2024-06-01T18:42:46.385Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: snapshot restore progress: id=2-1013-1717267366385 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:46.385Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: removing all old logs from log store 2024-06-01T18:42:46.385Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: compacting logs: from=1 to=2 2024-06-01T18:42:46.385Z [INFO] server-6f91d20d-13e6-c092-6615-7b3994deed7c: Installed remote snapshot 2024-06-01T18:42:46.385Z [INFO] server-76b32d50-29cf-9396-f2b9-6866fbdbe262: pipelining replication: peer="{Voter server-6f91d20d-13e6-c092-6615-7b3994deed7c 6f91d20d-13e6-c092-6615-7b3994deed7c}" 2024-06-01T18:42:47.385Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:46.379870058 +0000 UTC m=+81.878059510 (2 nodes), 932ns from start of poll, 1.755545194s from cluster start. Timeout at 2024-06-01 18:42:47.385410164 +0000 UTC m=+82.883599616, 1.005540106s after stability 2024-06-01T18:42:47.397Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: aborting pipeline replication: peer="{Voter server-b6cb225b-25c7-a9cc-78e9-18d87e4bc06a b6cb225b-25c7-a9cc-78e9-18d87e4bc06a}" 2024-06-01T18:42:47.397Z [INFO] server-aa11a9cf-3fd1-e704-84b0-3f5c3f8a0326: aborting pipeline replication: peer="{Voter server-573ec00c-452c-aea2-bdf4-a669df306329 573ec00c-452c-aea2-bdf4-a669df306329}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2024-06-01T18:42:47.398Z [INFO] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32926d28-57a8-25bc-31f1-dd18cc2a2534 Address:32926d28-57a8-25bc-31f1-dd18cc2a2534} {Suffrage:Voter ID:server-51d862b1-93e5-0682-c587-60406dcc495a Address:51d862b1-93e5-0682-c587-60406dcc495a} {Suffrage:Voter ID:server-a47715ad-741f-eca4-a158-5ef2083b4dc0 Address:a47715ad-741f-eca4-a158-5ef2083b4dc0}]" 2024-06-01T18:42:47.398Z [INFO] server-51d862b1-93e5-0682-c587-60406dcc495a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32926d28-57a8-25bc-31f1-dd18cc2a2534 Address:32926d28-57a8-25bc-31f1-dd18cc2a2534} {Suffrage:Voter ID:server-51d862b1-93e5-0682-c587-60406dcc495a Address:51d862b1-93e5-0682-c587-60406dcc495a} {Suffrage:Voter ID:server-a47715ad-741f-eca4-a158-5ef2083b4dc0 Address:a47715ad-741f-eca4-a158-5ef2083b4dc0}]" 2024-06-01T18:42:47.398Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32926d28-57a8-25bc-31f1-dd18cc2a2534 Address:32926d28-57a8-25bc-31f1-dd18cc2a2534} {Suffrage:Voter ID:server-51d862b1-93e5-0682-c587-60406dcc495a Address:51d862b1-93e5-0682-c587-60406dcc495a} {Suffrage:Voter ID:server-a47715ad-741f-eca4-a158-5ef2083b4dc0 Address:a47715ad-741f-eca4-a158-5ef2083b4dc0}]" 2024-06-01T18:42:47.398Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:47.398Z [INFO] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: entering follower state: follower="Node at 32926d28-57a8-25bc-31f1-dd18cc2a2534 [Follower]" leader-address= leader-id= 2024-06-01T18:42:47.398Z [INFO] server-51d862b1-93e5-0682-c587-60406dcc495a: entering follower state: follower="Node at 51d862b1-93e5-0682-c587-60406dcc495a [Follower]" leader-address= leader-id= 2024-06-01T18:42:47.398Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: entering follower state: follower="Node at a47715ad-741f-eca4-a158-5ef2083b4dc0 [Follower]" leader-address= leader-id= 2024-06-01T18:42:47.958Z [WARN] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:47.958Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: entering candidate state: node="Node at a47715ad-741f-eca4-a158-5ef2083b4dc0 [Candidate]" term=2 2024-06-01T18:42:47.958Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: election won: term=2 tally=2 2024-06-01T18:42:47.958Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: entering leader state: leader="Node at a47715ad-741f-eca4-a158-5ef2083b4dc0 [Leader]" 2024-06-01T18:42:47.958Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: added peer, starting replication: peer=server-32926d28-57a8-25bc-31f1-dd18cc2a2534 2024-06-01T18:42:47.958Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: added peer, starting replication: peer=server-51d862b1-93e5-0682-c587-60406dcc495a 2024-06-01T18:42:47.958Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: pipelining replication: peer="{Voter server-51d862b1-93e5-0682-c587-60406dcc495a 51d862b1-93e5-0682-c587-60406dcc495a}" 2024-06-01T18:42:47.958Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: pipelining replication: peer="{Voter server-32926d28-57a8-25bc-31f1-dd18cc2a2534 32926d28-57a8-25bc-31f1-dd18cc2a2534}" 2024-06-01T18:42:47.958Z [WARN] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: rejecting vote request since our last term is greater: candidate=a47715ad-741f-eca4-a158-5ef2083b4dc0 last-term=2 last-candidate-term=1 2024-06-01T18:42:48.964Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:47.95839286 +0000 UTC m=+83.456582312 (1 nodes), 559.868083ms from start of poll, 560.139087ms from cluster start. Timeout at 2024-06-01 18:42:48.964121525 +0000 UTC m=+84.462310977, 1.005728665s after stability 2024-06-01T18:42:48.964Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: starting snapshot up to: index=12 2024-06-01T18:42:48.964Z [INFO] creating new snapshot: path=/tmp/raft397414440/snapshots/2-12-1717267368964.tmp 2024-06-01T18:42:48.984Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: snapshot complete up to: index=12 2024-06-01T18:42:48.986Z [INFO] creating new snapshot: path=/tmp/raft397414440/snapshots/2-10013-1717267368986.tmp 2024-06-01T18:42:48.987Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: copied to local snapshot: bytes=71 2024-06-01T18:42:48.987Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: snapshot restore progress: id=2-10013-1717267368986 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:48.987Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: restored user snapshot: index=10013 2024-06-01T18:42:48.987Z [ERROR] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: failed to get log: index=23 error="log not found" 2024-06-01T18:42:48.987Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: aborting pipeline replication: peer="{Voter server-51d862b1-93e5-0682-c587-60406dcc495a 51d862b1-93e5-0682-c587-60406dcc495a}" 2024-06-01T18:42:48.987Z [ERROR] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: failed to get log: index=23 error="log not found" 2024-06-01T18:42:48.987Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: aborting pipeline replication: peer="{Voter server-32926d28-57a8-25bc-31f1-dd18cc2a2534 32926d28-57a8-25bc-31f1-dd18cc2a2534}" 2024-06-01T18:42:48.994Z [ERROR] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: failed to get log: index=23 error="log not found" 2024-06-01T18:42:48.995Z [INFO] creating new snapshot: path=/tmp/raft4023688845/snapshots/2-10013-1717267368995.tmp 2024-06-01T18:42:48.995Z [INFO] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:48.995Z [ERROR] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: failed to get log: index=23 error="log not found" 2024-06-01T18:42:48.996Z [INFO] creating new snapshot: path=/tmp/raft2607946245/snapshots/2-10013-1717267368996.tmp 2024-06-01T18:42:48.996Z [INFO] server-51d862b1-93e5-0682-c587-60406dcc495a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:49.085Z [INFO] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: copied to local snapshot: bytes=71 2024-06-01T18:42:49.085Z [INFO] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: snapshot restore progress: id=2-10013-1717267368995 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:49.085Z [INFO] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: Installed remote snapshot 2024-06-01T18:42:49.085Z [INFO] server-51d862b1-93e5-0682-c587-60406dcc495a: copied to local snapshot: bytes=71 2024-06-01T18:42:49.085Z [WARN] server-32926d28-57a8-25bc-31f1-dd18cc2a2534: skipping application of old log: index=22 2024-06-01T18:42:49.085Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: pipelining replication: peer="{Voter server-32926d28-57a8-25bc-31f1-dd18cc2a2534 32926d28-57a8-25bc-31f1-dd18cc2a2534}" 2024-06-01T18:42:49.085Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:49.086Z [INFO] server-51d862b1-93e5-0682-c587-60406dcc495a: snapshot restore progress: id=2-10013-1717267368996 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:49.086Z [INFO] server-51d862b1-93e5-0682-c587-60406dcc495a: Installed remote snapshot 2024-06-01T18:42:49.086Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: pipelining replication: peer="{Voter server-51d862b1-93e5-0682-c587-60406dcc495a 51d862b1-93e5-0682-c587-60406dcc495a}" 2024-06-01T18:42:50.091Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:49.085888548 +0000 UTC m=+84.584078000 (2 nodes), 681ns from start of poll, 1.687634775s from cluster start. Timeout at 2024-06-01 18:42:50.09144819 +0000 UTC m=+85.589637642, 1.005559642s after stability 2024-06-01T18:42:50.102Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: aborting pipeline replication: peer="{Voter server-51d862b1-93e5-0682-c587-60406dcc495a 51d862b1-93e5-0682-c587-60406dcc495a}" 2024-06-01T18:42:50.102Z [INFO] server-a47715ad-741f-eca4-a158-5ef2083b4dc0: aborting pipeline replication: peer="{Voter server-32926d28-57a8-25bc-31f1-dd18cc2a2534 32926d28-57a8-25bc-31f1-dd18cc2a2534}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2024-06-01T18:42:50.113Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-590be5ff-a9ba-21c8-4a12-86764d10fad0 Address:590be5ff-a9ba-21c8-4a12-86764d10fad0} {Suffrage:Voter ID:server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b Address:b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b} {Suffrage:Voter ID:server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 Address:cc5bfc02-5829-c26c-284b-92b3ac2dece9}]" 2024-06-01T18:42:50.113Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-590be5ff-a9ba-21c8-4a12-86764d10fad0 Address:590be5ff-a9ba-21c8-4a12-86764d10fad0} {Suffrage:Voter ID:server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b Address:b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b} {Suffrage:Voter ID:server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 Address:cc5bfc02-5829-c26c-284b-92b3ac2dece9}]" 2024-06-01T18:42:50.113Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: entering follower state: follower="Node at 590be5ff-a9ba-21c8-4a12-86764d10fad0 [Follower]" leader-address= leader-id= 2024-06-01T18:42:50.113Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-590be5ff-a9ba-21c8-4a12-86764d10fad0 Address:590be5ff-a9ba-21c8-4a12-86764d10fad0} {Suffrage:Voter ID:server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b Address:b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b} {Suffrage:Voter ID:server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 Address:cc5bfc02-5829-c26c-284b-92b3ac2dece9}]" 2024-06-01T18:42:50.113Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:50.113Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: entering follower state: follower="Node at b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b [Follower]" leader-address= leader-id= 2024-06-01T18:42:50.113Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: entering follower state: follower="Node at cc5bfc02-5829-c26c-284b-92b3ac2dece9 [Follower]" leader-address= leader-id= 2024-06-01T18:42:50.701Z [WARN] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:50.701Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: entering candidate state: node="Node at b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b [Candidate]" term=2 2024-06-01T18:42:50.702Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: election won: term=2 tally=2 2024-06-01T18:42:50.702Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: entering leader state: leader="Node at b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b [Leader]" 2024-06-01T18:42:50.702Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: added peer, starting replication: peer=server-590be5ff-a9ba-21c8-4a12-86764d10fad0 2024-06-01T18:42:50.702Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: added peer, starting replication: peer=server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 2024-06-01T18:42:50.702Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: pipelining replication: peer="{Voter server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 cc5bfc02-5829-c26c-284b-92b3ac2dece9}" 2024-06-01T18:42:50.702Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: pipelining replication: peer="{Voter server-590be5ff-a9ba-21c8-4a12-86764d10fad0 590be5ff-a9ba-21c8-4a12-86764d10fad0}" 2024-06-01T18:42:51.707Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:50.702162132 +0000 UTC m=+86.200351585 (1 nodes), 588.739117ms from start of poll, 588.99335ms from cluster start. Timeout at 2024-06-01 18:42:51.707555052 +0000 UTC m=+87.205744514, 1.005392929s after stability 2024-06-01T18:42:51.708Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: starting snapshot up to: index=12 2024-06-01T18:42:51.708Z [INFO] creating new snapshot: path=/tmp/raft1795246479/snapshots/2-12-1717267371708.tmp 2024-06-01T18:42:51.708Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: snapshot complete up to: index=12 2024-06-01T18:42:51.709Z [INFO] creating new snapshot: path=/tmp/raft1795246479/snapshots/2-10013-1717267371709.tmp 2024-06-01T18:42:51.809Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: copied to local snapshot: bytes=71 2024-06-01T18:42:51.809Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: snapshot restore progress: id=2-10013-1717267371709 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:51.809Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: removing all old logs from log store 2024-06-01T18:42:51.809Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: compacting logs: from=1 to=22 2024-06-01T18:42:51.809Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: restored user snapshot: index=10013 2024-06-01T18:42:51.810Z [ERROR] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: failed to get log: index=22 error="log not found" 2024-06-01T18:42:51.810Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: aborting pipeline replication: peer="{Voter server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 cc5bfc02-5829-c26c-284b-92b3ac2dece9}" 2024-06-01T18:42:51.810Z [ERROR] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: failed to get log: index=22 error="log not found" 2024-06-01T18:42:51.810Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: aborting pipeline replication: peer="{Voter server-590be5ff-a9ba-21c8-4a12-86764d10fad0 590be5ff-a9ba-21c8-4a12-86764d10fad0}" 2024-06-01T18:42:51.816Z [ERROR] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: failed to get log: index=22 error="log not found" 2024-06-01T18:42:51.817Z [INFO] creating new snapshot: path=/tmp/raft3624884128/snapshots/2-10013-1717267371817.tmp 2024-06-01T18:42:51.817Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:51.820Z [ERROR] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: failed to get log: index=22 error="log not found" 2024-06-01T18:42:51.820Z [INFO] creating new snapshot: path=/tmp/raft2360373645/snapshots/2-10013-1717267371820.tmp 2024-06-01T18:42:51.821Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:51.912Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: copied to local snapshot: bytes=71 2024-06-01T18:42:51.912Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: copied to local snapshot: bytes=71 2024-06-01T18:42:51.912Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: snapshot restore progress: id=2-10013-1717267371820 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:51.912Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: removing all old logs from log store 2024-06-01T18:42:51.912Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: compacting logs: from=1 to=22 2024-06-01T18:42:51.912Z [INFO] server-cc5bfc02-5829-c26c-284b-92b3ac2dece9: Installed remote snapshot 2024-06-01T18:42:51.912Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: snapshot restore progress: id=2-10013-1717267371817 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:51.912Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: removing all old logs from log store 2024-06-01T18:42:51.912Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: compacting logs: from=1 to=22 2024-06-01T18:42:51.912Z [INFO] server-590be5ff-a9ba-21c8-4a12-86764d10fad0: Installed remote snapshot 2024-06-01T18:42:51.912Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: pipelining replication: peer="{Voter server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 cc5bfc02-5829-c26c-284b-92b3ac2dece9}" 2024-06-01T18:42:51.912Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:51.912Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: pipelining replication: peer="{Voter server-590be5ff-a9ba-21c8-4a12-86764d10fad0 590be5ff-a9ba-21c8-4a12-86764d10fad0}" 2024-06-01T18:42:52.918Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:51.912763134 +0000 UTC m=+87.410952586 (2 nodes), 762ns from start of poll, 1.799594351s from cluster start. Timeout at 2024-06-01 18:42:52.91819684 +0000 UTC m=+88.416386292, 1.005433706s after stability 2024-06-01T18:42:52.930Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: aborting pipeline replication: peer="{Voter server-cc5bfc02-5829-c26c-284b-92b3ac2dece9 cc5bfc02-5829-c26c-284b-92b3ac2dece9}" 2024-06-01T18:42:52.930Z [INFO] server-b06ffcfe-62b9-2e8c-fbd6-73fceebdd68b: aborting pipeline replication: peer="{Voter server-590be5ff-a9ba-21c8-4a12-86764d10fad0 590be5ff-a9ba-21c8-4a12-86764d10fad0}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2024-06-01T18:42:52.931Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c3e1fa0-e22a-de86-9009-191c6714d85c Address:7c3e1fa0-e22a-de86-9009-191c6714d85c} {Suffrage:Voter ID:server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4 Address:f5d8990f-12ab-56cf-ab5c-d768c510b0a4} {Suffrage:Voter ID:server-ca748944-8544-916c-8f59-da2df0a8f05a Address:ca748944-8544-916c-8f59-da2df0a8f05a}]" 2024-06-01T18:42:52.931Z [INFO] server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c3e1fa0-e22a-de86-9009-191c6714d85c Address:7c3e1fa0-e22a-de86-9009-191c6714d85c} {Suffrage:Voter ID:server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4 Address:f5d8990f-12ab-56cf-ab5c-d768c510b0a4} {Suffrage:Voter ID:server-ca748944-8544-916c-8f59-da2df0a8f05a Address:ca748944-8544-916c-8f59-da2df0a8f05a}]" 2024-06-01T18:42:52.931Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: entering follower state: follower="Node at 7c3e1fa0-e22a-de86-9009-191c6714d85c [Follower]" leader-address= leader-id= 2024-06-01T18:42:52.932Z [INFO] server-ca748944-8544-916c-8f59-da2df0a8f05a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c3e1fa0-e22a-de86-9009-191c6714d85c Address:7c3e1fa0-e22a-de86-9009-191c6714d85c} {Suffrage:Voter ID:server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4 Address:f5d8990f-12ab-56cf-ab5c-d768c510b0a4} {Suffrage:Voter ID:server-ca748944-8544-916c-8f59-da2df0a8f05a Address:ca748944-8544-916c-8f59-da2df0a8f05a}]" 2024-06-01T18:42:52.932Z [INFO] server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4: entering follower state: follower="Node at f5d8990f-12ab-56cf-ab5c-d768c510b0a4 [Follower]" leader-address= leader-id= 2024-06-01T18:42:52.932Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:52.932Z [INFO] server-ca748944-8544-916c-8f59-da2df0a8f05a: entering follower state: follower="Node at ca748944-8544-916c-8f59-da2df0a8f05a [Follower]" leader-address= leader-id= 2024-06-01T18:42:53.566Z [WARN] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:53.566Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: entering candidate state: node="Node at 7c3e1fa0-e22a-de86-9009-191c6714d85c [Candidate]" term=2 2024-06-01T18:42:53.566Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: election won: term=2 tally=2 2024-06-01T18:42:53.566Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: entering leader state: leader="Node at 7c3e1fa0-e22a-de86-9009-191c6714d85c [Leader]" 2024-06-01T18:42:53.566Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: added peer, starting replication: peer=server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4 2024-06-01T18:42:53.566Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: added peer, starting replication: peer=server-ca748944-8544-916c-8f59-da2df0a8f05a 2024-06-01T18:42:53.566Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: pipelining replication: peer="{Voter server-ca748944-8544-916c-8f59-da2df0a8f05a ca748944-8544-916c-8f59-da2df0a8f05a}" 2024-06-01T18:42:53.566Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: pipelining replication: peer="{Voter server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4 f5d8990f-12ab-56cf-ab5c-d768c510b0a4}" 2024-06-01T18:42:54.572Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:53.566950856 +0000 UTC m=+89.065140308 (1 nodes), 634.824446ms from start of poll, 635.134953ms from cluster start. Timeout at 2024-06-01 18:42:54.572214186 +0000 UTC m=+90.070403638, 1.00526333s after stability 2024-06-01T18:42:54.572Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: starting snapshot up to: index=12 2024-06-01T18:42:54.572Z [INFO] creating new snapshot: path=/tmp/raft894130115/snapshots/2-12-1717267374572.tmp 2024-06-01T18:42:54.588Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: snapshot complete up to: index=12 2024-06-01T18:42:54.589Z [INFO] server-1fbe6127-9344-06c7-8766-48eccc6cd203: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1fbe6127-9344-06c7-8766-48eccc6cd203 Address:1fbe6127-9344-06c7-8766-48eccc6cd203} {Suffrage:Voter ID:server-701e37f3-7160-49d8-0218-812ec547992c Address:701e37f3-7160-49d8-0218-812ec547992c} {Suffrage:Voter ID:server-6310b326-2abd-dff3-5be7-575cb060e63a Address:6310b326-2abd-dff3-5be7-575cb060e63a}]" 2024-06-01T18:42:54.589Z [INFO] server-701e37f3-7160-49d8-0218-812ec547992c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1fbe6127-9344-06c7-8766-48eccc6cd203 Address:1fbe6127-9344-06c7-8766-48eccc6cd203} {Suffrage:Voter ID:server-701e37f3-7160-49d8-0218-812ec547992c Address:701e37f3-7160-49d8-0218-812ec547992c} {Suffrage:Voter ID:server-6310b326-2abd-dff3-5be7-575cb060e63a Address:6310b326-2abd-dff3-5be7-575cb060e63a}]" 2024-06-01T18:42:54.590Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1fbe6127-9344-06c7-8766-48eccc6cd203 Address:1fbe6127-9344-06c7-8766-48eccc6cd203} {Suffrage:Voter ID:server-701e37f3-7160-49d8-0218-812ec547992c Address:701e37f3-7160-49d8-0218-812ec547992c} {Suffrage:Voter ID:server-6310b326-2abd-dff3-5be7-575cb060e63a Address:6310b326-2abd-dff3-5be7-575cb060e63a}]" 2024-06-01T18:42:54.590Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:54.590Z [INFO] server-1fbe6127-9344-06c7-8766-48eccc6cd203: entering follower state: follower="Node at 1fbe6127-9344-06c7-8766-48eccc6cd203 [Follower]" leader-address= leader-id= 2024-06-01T18:42:54.590Z [INFO] server-701e37f3-7160-49d8-0218-812ec547992c: entering follower state: follower="Node at 701e37f3-7160-49d8-0218-812ec547992c [Follower]" leader-address= leader-id= 2024-06-01T18:42:54.590Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: entering follower state: follower="Node at 6310b326-2abd-dff3-5be7-575cb060e63a [Follower]" leader-address= leader-id= 2024-06-01T18:42:55.140Z [WARN] server-6310b326-2abd-dff3-5be7-575cb060e63a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:55.140Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: entering candidate state: node="Node at 6310b326-2abd-dff3-5be7-575cb060e63a [Candidate]" term=2 2024-06-01T18:42:55.140Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: election won: term=2 tally=2 2024-06-01T18:42:55.140Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: entering leader state: leader="Node at 6310b326-2abd-dff3-5be7-575cb060e63a [Leader]" 2024-06-01T18:42:55.140Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: added peer, starting replication: peer=server-1fbe6127-9344-06c7-8766-48eccc6cd203 2024-06-01T18:42:55.140Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: added peer, starting replication: peer=server-701e37f3-7160-49d8-0218-812ec547992c 2024-06-01T18:42:55.140Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: pipelining replication: peer="{Voter server-701e37f3-7160-49d8-0218-812ec547992c 701e37f3-7160-49d8-0218-812ec547992c}" 2024-06-01T18:42:55.140Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: pipelining replication: peer="{Voter server-1fbe6127-9344-06c7-8766-48eccc6cd203 1fbe6127-9344-06c7-8766-48eccc6cd203}" 2024-06-01T18:42:56.146Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:55.140326383 +0000 UTC m=+90.638515835 (1 nodes), 550.279221ms from start of poll, 550.526852ms from cluster start. Timeout at 2024-06-01 18:42:56.146147972 +0000 UTC m=+91.644337424, 1.005821589s after stability 2024-06-01T18:42:56.146Z [INFO] creating new snapshot: path=/tmp/raft1634810935/snapshots/2-10013-1717267376146.tmp 2024-06-01T18:42:56.194Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: copied to local snapshot: bytes=71 2024-06-01T18:42:56.194Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: snapshot restore progress: id=2-10013-1717267376146 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:56.194Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: restored user snapshot: index=10013 2024-06-01T18:42:56.194Z [ERROR] server-6310b326-2abd-dff3-5be7-575cb060e63a: failed to get log: index=3 error="log not found" 2024-06-01T18:42:56.194Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: aborting pipeline replication: peer="{Voter server-701e37f3-7160-49d8-0218-812ec547992c 701e37f3-7160-49d8-0218-812ec547992c}" 2024-06-01T18:42:56.194Z [ERROR] server-6310b326-2abd-dff3-5be7-575cb060e63a: failed to get log: index=3 error="log not found" 2024-06-01T18:42:56.194Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: aborting pipeline replication: peer="{Voter server-1fbe6127-9344-06c7-8766-48eccc6cd203 1fbe6127-9344-06c7-8766-48eccc6cd203}" 2024-06-01T18:42:56.201Z [ERROR] server-6310b326-2abd-dff3-5be7-575cb060e63a: failed to get log: index=3 error="log not found" 2024-06-01T18:42:56.201Z [ERROR] server-6310b326-2abd-dff3-5be7-575cb060e63a: failed to get log: index=3 error="log not found" 2024-06-01T18:42:56.201Z [INFO] creating new snapshot: path=/tmp/raft1021207119/snapshots/2-10013-1717267376201.tmp 2024-06-01T18:42:56.201Z [INFO] creating new snapshot: path=/tmp/raft3451950340/snapshots/2-10013-1717267376201.tmp 2024-06-01T18:42:56.201Z [INFO] server-701e37f3-7160-49d8-0218-812ec547992c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:56.201Z [INFO] server-1fbe6127-9344-06c7-8766-48eccc6cd203: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:56.294Z [INFO] server-1fbe6127-9344-06c7-8766-48eccc6cd203: copied to local snapshot: bytes=71 2024-06-01T18:42:56.294Z [INFO] server-1fbe6127-9344-06c7-8766-48eccc6cd203: snapshot restore progress: id=2-10013-1717267376201 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:56.294Z [INFO] server-701e37f3-7160-49d8-0218-812ec547992c: copied to local snapshot: bytes=71 2024-06-01T18:42:56.294Z [INFO] server-1fbe6127-9344-06c7-8766-48eccc6cd203: Installed remote snapshot 2024-06-01T18:42:56.294Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: pipelining replication: peer="{Voter server-1fbe6127-9344-06c7-8766-48eccc6cd203 1fbe6127-9344-06c7-8766-48eccc6cd203}" 2024-06-01T18:42:56.294Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:42:56.294Z [INFO] server-701e37f3-7160-49d8-0218-812ec547992c: snapshot restore progress: id=2-10013-1717267376201 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:42:56.294Z [INFO] server-701e37f3-7160-49d8-0218-812ec547992c: Installed remote snapshot 2024-06-01T18:42:56.295Z [INFO] server-6310b326-2abd-dff3-5be7-575cb060e63a: pipelining replication: peer="{Voter server-701e37f3-7160-49d8-0218-812ec547992c 701e37f3-7160-49d8-0218-812ec547992c}" 2024-06-01T18:42:57.301Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:42:56.294934152 +0000 UTC m=+91.793123604 (2 nodes), 1.884µs from start of poll, 1.705134621s from cluster start. Timeout at 2024-06-01 18:42:57.301063773 +0000 UTC m=+92.799253225, 1.006129621s after stability 2024-06-01T18:42:57.313Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: aborting pipeline replication: peer="{Voter server-ca748944-8544-916c-8f59-da2df0a8f05a ca748944-8544-916c-8f59-da2df0a8f05a}" 2024-06-01T18:42:57.313Z [INFO] server-7c3e1fa0-e22a-de86-9009-191c6714d85c: aborting pipeline replication: peer="{Voter server-f5d8990f-12ab-56cf-ab5c-d768c510b0a4 f5d8990f-12ab-56cf-ab5c-d768c510b0a4}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2024-06-01T18:42:57.403Z [INFO] server-a1d82919-2465-611e-ba90-75ca448c6611: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1d82919-2465-611e-ba90-75ca448c6611 Address:a1d82919-2465-611e-ba90-75ca448c6611} {Suffrage:Voter ID:server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5 Address:5a63c006-2b3d-54a1-6ab1-a108a27a1ec5} {Suffrage:Voter ID:server-81ae3033-d08b-8e47-70bb-1a087a25b53b Address:81ae3033-d08b-8e47-70bb-1a087a25b53b}]" 2024-06-01T18:42:57.403Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1d82919-2465-611e-ba90-75ca448c6611 Address:a1d82919-2465-611e-ba90-75ca448c6611} {Suffrage:Voter ID:server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5 Address:5a63c006-2b3d-54a1-6ab1-a108a27a1ec5} {Suffrage:Voter ID:server-81ae3033-d08b-8e47-70bb-1a087a25b53b Address:81ae3033-d08b-8e47-70bb-1a087a25b53b}]" 2024-06-01T18:42:57.403Z [INFO] server-a1d82919-2465-611e-ba90-75ca448c6611: entering follower state: follower="Node at a1d82919-2465-611e-ba90-75ca448c6611 [Follower]" leader-address= leader-id= 2024-06-01T18:42:57.403Z [INFO] server-81ae3033-d08b-8e47-70bb-1a087a25b53b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1d82919-2465-611e-ba90-75ca448c6611 Address:a1d82919-2465-611e-ba90-75ca448c6611} {Suffrage:Voter ID:server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5 Address:5a63c006-2b3d-54a1-6ab1-a108a27a1ec5} {Suffrage:Voter ID:server-81ae3033-d08b-8e47-70bb-1a087a25b53b Address:81ae3033-d08b-8e47-70bb-1a087a25b53b}]" 2024-06-01T18:42:57.403Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:57.403Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: entering follower state: follower="Node at 5a63c006-2b3d-54a1-6ab1-a108a27a1ec5 [Follower]" leader-address= leader-id= 2024-06-01T18:42:57.403Z [INFO] server-81ae3033-d08b-8e47-70bb-1a087a25b53b: entering follower state: follower="Node at 81ae3033-d08b-8e47-70bb-1a087a25b53b [Follower]" leader-address= leader-id= 2024-06-01T18:42:57.936Z [WARN] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:57.936Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: entering candidate state: node="Node at 5a63c006-2b3d-54a1-6ab1-a108a27a1ec5 [Candidate]" term=2 2024-06-01T18:42:57.936Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: election won: term=2 tally=2 2024-06-01T18:42:57.936Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: entering leader state: leader="Node at 5a63c006-2b3d-54a1-6ab1-a108a27a1ec5 [Leader]" 2024-06-01T18:42:57.936Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: added peer, starting replication: peer=server-a1d82919-2465-611e-ba90-75ca448c6611 2024-06-01T18:42:57.936Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: added peer, starting replication: peer=server-81ae3033-d08b-8e47-70bb-1a087a25b53b 2024-06-01T18:42:57.936Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: pipelining replication: peer="{Voter server-81ae3033-d08b-8e47-70bb-1a087a25b53b 81ae3033-d08b-8e47-70bb-1a087a25b53b}" 2024-06-01T18:42:57.936Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: pipelining replication: peer="{Voter server-a1d82919-2465-611e-ba90-75ca448c6611 a1d82919-2465-611e-ba90-75ca448c6611}" 2024-06-01T18:42:58.942Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:57.936796779 +0000 UTC m=+93.434986231 (1 nodes), 532.900641ms from start of poll, 533.24406ms from cluster start. Timeout at 2024-06-01 18:42:58.942256442 +0000 UTC m=+94.440445894, 1.005459663s after stability 2024-06-01T18:42:58.942Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: starting snapshot up to: index=12 2024-06-01T18:42:58.942Z [INFO] creating new snapshot: path=/tmp/raft27895488/snapshots/2-12-1717267378942.tmp 2024-06-01T18:42:58.943Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: snapshot complete up to: index=12 2024-06-01T18:42:58.944Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58abee37-2d89-5b0b-6f07-e4bbf1247b37 Address:58abee37-2d89-5b0b-6f07-e4bbf1247b37} {Suffrage:Voter ID:server-76baec6a-31ac-fc47-0355-a08cf5345795 Address:76baec6a-31ac-fc47-0355-a08cf5345795} {Suffrage:Voter ID:server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 Address:a0c4b961-bf6c-7e0f-7bde-3bb1113b9709}]" 2024-06-01T18:42:58.944Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58abee37-2d89-5b0b-6f07-e4bbf1247b37 Address:58abee37-2d89-5b0b-6f07-e4bbf1247b37} {Suffrage:Voter ID:server-76baec6a-31ac-fc47-0355-a08cf5345795 Address:76baec6a-31ac-fc47-0355-a08cf5345795} {Suffrage:Voter ID:server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 Address:a0c4b961-bf6c-7e0f-7bde-3bb1113b9709}]" 2024-06-01T18:42:58.944Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: entering follower state: follower="Node at 58abee37-2d89-5b0b-6f07-e4bbf1247b37 [Follower]" leader-address= leader-id= 2024-06-01T18:42:58.944Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58abee37-2d89-5b0b-6f07-e4bbf1247b37 Address:58abee37-2d89-5b0b-6f07-e4bbf1247b37} {Suffrage:Voter ID:server-76baec6a-31ac-fc47-0355-a08cf5345795 Address:76baec6a-31ac-fc47-0355-a08cf5345795} {Suffrage:Voter ID:server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 Address:a0c4b961-bf6c-7e0f-7bde-3bb1113b9709}]" 2024-06-01T18:42:58.944Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: entering follower state: follower="Node at 76baec6a-31ac-fc47-0355-a08cf5345795 [Follower]" leader-address= leader-id= 2024-06-01T18:42:58.944Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:42:58.944Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: entering follower state: follower="Node at a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 [Follower]" leader-address= leader-id= 2024-06-01T18:42:59.514Z [WARN] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:42:59.514Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: entering candidate state: node="Node at 58abee37-2d89-5b0b-6f07-e4bbf1247b37 [Candidate]" term=2 2024-06-01T18:42:59.514Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: election won: term=2 tally=2 2024-06-01T18:42:59.514Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: entering leader state: leader="Node at 58abee37-2d89-5b0b-6f07-e4bbf1247b37 [Leader]" 2024-06-01T18:42:59.514Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: added peer, starting replication: peer=server-76baec6a-31ac-fc47-0355-a08cf5345795 2024-06-01T18:42:59.514Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: added peer, starting replication: peer=server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 2024-06-01T18:42:59.514Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: pipelining replication: peer="{Voter server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 a0c4b961-bf6c-7e0f-7bde-3bb1113b9709}" 2024-06-01T18:42:59.514Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: pipelining replication: peer="{Voter server-76baec6a-31ac-fc47-0355-a08cf5345795 76baec6a-31ac-fc47-0355-a08cf5345795}" 2024-06-01T18:43:00.519Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:42:59.51469271 +0000 UTC m=+95.012882162 (1 nodes), 569.840446ms from start of poll, 570.088757ms from cluster start. Timeout at 2024-06-01 18:43:00.519879615 +0000 UTC m=+96.018069077, 1.005186915s after stability 2024-06-01T18:43:00.520Z [INFO] creating new snapshot: path=/tmp/raft337286669/snapshots/2-10013-1717267380520.tmp 2024-06-01T18:43:00.520Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: copied to local snapshot: bytes=71 2024-06-01T18:43:00.520Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: snapshot restore progress: id=2-10013-1717267380520 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:43:00.520Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: removing all old logs from log store 2024-06-01T18:43:00.520Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: compacting logs: from=1 to=2 2024-06-01T18:43:00.520Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: restored user snapshot: index=10013 2024-06-01T18:43:00.520Z [ERROR] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: failed to get log: index=2 error="log not found" 2024-06-01T18:43:00.520Z [ERROR] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: failed to get log: index=2 error="log not found" 2024-06-01T18:43:00.520Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: aborting pipeline replication: peer="{Voter server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 a0c4b961-bf6c-7e0f-7bde-3bb1113b9709}" 2024-06-01T18:43:00.520Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: aborting pipeline replication: peer="{Voter server-76baec6a-31ac-fc47-0355-a08cf5345795 76baec6a-31ac-fc47-0355-a08cf5345795}" 2024-06-01T18:43:00.527Z [ERROR] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: failed to get log: index=2 error="log not found" 2024-06-01T18:43:00.527Z [INFO] creating new snapshot: path=/tmp/raft349424814/snapshots/2-10013-1717267380527.tmp 2024-06-01T18:43:00.527Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:43:00.527Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: copied to local snapshot: bytes=71 2024-06-01T18:43:00.527Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: snapshot restore progress: id=2-10013-1717267380527 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:43:00.527Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: removing all old logs from log store 2024-06-01T18:43:00.527Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: compacting logs: from=1 to=2 2024-06-01T18:43:00.527Z [INFO] server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709: Installed remote snapshot 2024-06-01T18:43:00.527Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: pipelining replication: peer="{Voter server-a0c4b961-bf6c-7e0f-7bde-3bb1113b9709 a0c4b961-bf6c-7e0f-7bde-3bb1113b9709}" 2024-06-01T18:43:00.527Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:00.531Z [ERROR] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: failed to get log: index=2 error="log not found" 2024-06-01T18:43:00.531Z [INFO] creating new snapshot: path=/tmp/raft2248940794/snapshots/2-10013-1717267380531.tmp 2024-06-01T18:43:00.533Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2024-06-01T18:43:00.533Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: copied to local snapshot: bytes=71 2024-06-01T18:43:00.533Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: snapshot restore progress: id=2-10013-1717267380531 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2024-06-01T18:43:00.533Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: removing all old logs from log store 2024-06-01T18:43:00.533Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: compacting logs: from=1 to=2 2024-06-01T18:43:00.533Z [INFO] server-76baec6a-31ac-fc47-0355-a08cf5345795: Installed remote snapshot 2024-06-01T18:43:00.533Z [INFO] server-58abee37-2d89-5b0b-6f07-e4bbf1247b37: pipelining replication: peer="{Voter server-76baec6a-31ac-fc47-0355-a08cf5345795 76baec6a-31ac-fc47-0355-a08cf5345795}" 2024-06-01T18:43:01.533Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:00.527987438 +0000 UTC m=+96.026176900 (2 nodes), 1.262µs from start of poll, 1.583383495s from cluster start. Timeout at 2024-06-01 18:43:01.533050814 +0000 UTC m=+97.031240266, 1.005063366s after stability 2024-06-01T18:43:01.543Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: aborting pipeline replication: peer="{Voter server-81ae3033-d08b-8e47-70bb-1a087a25b53b 81ae3033-d08b-8e47-70bb-1a087a25b53b}" 2024-06-01T18:43:01.544Z [INFO] server-5a63c006-2b3d-54a1-6ab1-a108a27a1ec5: aborting pipeline replication: peer="{Voter server-a1d82919-2465-611e-ba90-75ca448c6611 a1d82919-2465-611e-ba90-75ca448c6611}" --- PASS: TestRaft_UserRestore (83.88s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.70s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.61s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.35s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.24s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.85s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.55s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.09s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.24s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.62s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.58s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.36s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.22s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.76s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.61s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.41s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.42s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.77s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.89s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.18s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.30s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.71s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.82s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.47s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.14s) === RUN TestRaft_SendSnapshotFollower 2024-06-01T18:43:01.545Z [INFO] server-db92e9d1-f4ac-5650-1078-f63f8f0717c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 Address:db92e9d1-f4ac-5650-1078-f63f8f0717c2} {Suffrage:Voter ID:server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc Address:8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc} {Suffrage:Voter ID:server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9 Address:1f0c3a26-af15-7abf-165c-de2b5baa4ab9}]" 2024-06-01T18:43:01.545Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 Address:db92e9d1-f4ac-5650-1078-f63f8f0717c2} {Suffrage:Voter ID:server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc Address:8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc} {Suffrage:Voter ID:server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9 Address:1f0c3a26-af15-7abf-165c-de2b5baa4ab9}]" 2024-06-01T18:43:01.545Z [INFO] server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 Address:db92e9d1-f4ac-5650-1078-f63f8f0717c2} {Suffrage:Voter ID:server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc Address:8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc} {Suffrage:Voter ID:server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9 Address:1f0c3a26-af15-7abf-165c-de2b5baa4ab9}]" 2024-06-01T18:43:01.545Z [INFO] server-db92e9d1-f4ac-5650-1078-f63f8f0717c2: entering follower state: follower="Node at db92e9d1-f4ac-5650-1078-f63f8f0717c2 [Follower]" leader-address= leader-id= 2024-06-01T18:43:01.545Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:01.545Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: entering follower state: follower="Node at 8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc [Follower]" leader-address= leader-id= 2024-06-01T18:43:01.545Z [INFO] server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9: entering follower state: follower="Node at 1f0c3a26-af15-7abf-165c-de2b5baa4ab9 [Follower]" leader-address= leader-id= 2024-06-01T18:43:01.604Z [WARN] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:01.604Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: entering candidate state: node="Node at 8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc [Candidate]" term=2 2024-06-01T18:43:01.605Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: election won: term=2 tally=2 2024-06-01T18:43:01.605Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: entering leader state: leader="Node at 8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc [Leader]" 2024-06-01T18:43:01.605Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: added peer, starting replication: peer=server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 2024-06-01T18:43:01.605Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: added peer, starting replication: peer=server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9 2024-06-01T18:43:01.605Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: pipelining replication: peer="{Voter server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 db92e9d1-f4ac-5650-1078-f63f8f0717c2}" 2024-06-01T18:43:01.605Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: pipelining replication: peer="{Voter server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9 1f0c3a26-af15-7abf-165c-de2b5baa4ab9}" 2024-06-01T18:43:01.710Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:01.605090898 +0000 UTC m=+97.103280350 (2 nodes), 59.758792ms from start of poll, 60.003198ms from cluster start. Timeout at 2024-06-01 18:43:01.710130423 +0000 UTC m=+97.208319875, 105.039525ms after stability 2024-06-01T18:43:01.710Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:01.815Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:01.710208428 +0000 UTC m=+97.208397880 (1 nodes), 1.924µs from start of poll, 165.120728ms from cluster start. Timeout at 2024-06-01 18:43:01.815574991 +0000 UTC m=+97.313764443, 105.366563ms after stability 2024-06-01T18:43:01.815Z [ERROR] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: failed to pipeline appendEntries: peer="{Voter server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 db92e9d1-f4ac-5650-1078-f63f8f0717c2}" error="append pipeline closed" 2024-06-01T18:43:01.815Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: aborting pipeline replication: peer="{Voter server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 db92e9d1-f4ac-5650-1078-f63f8f0717c2}" 2024-06-01T18:43:01.815Z [ERROR] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: failed to appendEntries to: peer="{Voter server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 db92e9d1-f4ac-5650-1078-f63f8f0717c2}" error="failed to connect to peer: db92e9d1-f4ac-5650-1078-f63f8f0717c2" raft_test.go:1627: [INFO] Finished apply without behind follower 2024-06-01T18:43:01.817Z [ERROR] server-db92e9d1-f4ac-5650-1078-f63f8f0717c2: failed to take snapshot: error="nothing new to snapshot" 2024-06-01T18:43:01.817Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: starting snapshot up to: index=102 2024-06-01T18:43:01.817Z [INFO] creating new snapshot: path=/tmp/raft477016255/snapshots/2-102-1717267381817.tmp 2024-06-01T18:43:01.819Z [ERROR] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: failed to heartbeat to: peer=db92e9d1-f4ac-5650-1078-f63f8f0717c2 backoff time=10ms error="failed to connect to peer: db92e9d1-f4ac-5650-1078-f63f8f0717c2" 2024-06-01T18:43:01.826Z [ERROR] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: failed to appendEntries to: peer="{Voter server-db92e9d1-f4ac-5650-1078-f63f8f0717c2 db92e9d1-f4ac-5650-1078-f63f8f0717c2}" error="failed to connect to peer: db92e9d1-f4ac-5650-1078-f63f8f0717c2" 2024-06-01T18:43:01.837Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: compacting logs: from=1 to=92 2024-06-01T18:43:01.837Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: snapshot complete up to: index=102 2024-06-01T18:43:01.837Z [INFO] server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9: starting snapshot up to: index=102 2024-06-01T18:43:01.837Z [INFO] creating new snapshot: path=/tmp/raft3106259847/snapshots/2-102-1717267381837.tmp 2024-06-01T18:43:01.837Z [ERROR] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: failed to get log: index=2 error="log not found" 2024-06-01T18:43:01.837Z [INFO] server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9: compacting logs: from=1 to=92 2024-06-01T18:43:01.837Z [INFO] server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9: snapshot complete up to: index=102 2024-06-01T18:43:01.837Z [INFO] creating new snapshot: path=/tmp/raft3265428136/snapshots/2-102-1717267381837.tmp 2024-06-01T18:43:01.838Z [INFO] server-db92e9d1-f4ac-5650-1078-f63f8f0717c2: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2024-06-01T18:43:01.838Z [INFO] server-db92e9d1-f4ac-5650-1078-f63f8f0717c2: copied to local snapshot: bytes=693 2024-06-01T18:43:01.838Z [INFO] server-db92e9d1-f4ac-5650-1078-f63f8f0717c2: snapshot restore progress: id=2-102-1717267381837 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-01T18:43:01.838Z [INFO] server-db92e9d1-f4ac-5650-1078-f63f8f0717c2: Installed remote snapshot 2024-06-01T18:43:01.838Z [INFO] server-8874d49f-dfdf-273a-6f1d-9dd0bbd53ffc: aborting pipeline replication: peer="{Voter server-1f0c3a26-af15-7abf-165c-de2b5baa4ab9 1f0c3a26-af15-7abf-165c-de2b5baa4ab9}" --- PASS: TestRaft_SendSnapshotFollower (0.33s) === RUN TestRaft_SendSnapshotAndLogsFollower 2024-06-01T18:43:01.874Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf494d8c-e1af-c757-a9ef-fece50e1a217 Address:cf494d8c-e1af-c757-a9ef-fece50e1a217} {Suffrage:Voter ID:server-2a22570f-81d3-b6c5-6972-f51cf80e4572 Address:2a22570f-81d3-b6c5-6972-f51cf80e4572} {Suffrage:Voter ID:server-a60041e1-149a-6479-acec-2f772d53e958 Address:a60041e1-149a-6479-acec-2f772d53e958}]" 2024-06-01T18:43:01.874Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf494d8c-e1af-c757-a9ef-fece50e1a217 Address:cf494d8c-e1af-c757-a9ef-fece50e1a217} {Suffrage:Voter ID:server-2a22570f-81d3-b6c5-6972-f51cf80e4572 Address:2a22570f-81d3-b6c5-6972-f51cf80e4572} {Suffrage:Voter ID:server-a60041e1-149a-6479-acec-2f772d53e958 Address:a60041e1-149a-6479-acec-2f772d53e958}]" 2024-06-01T18:43:01.874Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: entering follower state: follower="Node at cf494d8c-e1af-c757-a9ef-fece50e1a217 [Follower]" leader-address= leader-id= 2024-06-01T18:43:01.874Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: entering follower state: follower="Node at 2a22570f-81d3-b6c5-6972-f51cf80e4572 [Follower]" leader-address= leader-id= 2024-06-01T18:43:01.874Z [INFO] server-a60041e1-149a-6479-acec-2f772d53e958: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf494d8c-e1af-c757-a9ef-fece50e1a217 Address:cf494d8c-e1af-c757-a9ef-fece50e1a217} {Suffrage:Voter ID:server-2a22570f-81d3-b6c5-6972-f51cf80e4572 Address:2a22570f-81d3-b6c5-6972-f51cf80e4572} {Suffrage:Voter ID:server-a60041e1-149a-6479-acec-2f772d53e958 Address:a60041e1-149a-6479-acec-2f772d53e958}]" 2024-06-01T18:43:01.874Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:01.874Z [INFO] server-a60041e1-149a-6479-acec-2f772d53e958: entering follower state: follower="Node at a60041e1-149a-6479-acec-2f772d53e958 [Follower]" leader-address= leader-id= 2024-06-01T18:43:01.932Z [WARN] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:01.932Z [WARN] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:01.932Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: entering candidate state: node="Node at 2a22570f-81d3-b6c5-6972-f51cf80e4572 [Candidate]" term=2 2024-06-01T18:43:01.932Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: entering candidate state: node="Node at cf494d8c-e1af-c757-a9ef-fece50e1a217 [Candidate]" term=2 2024-06-01T18:43:01.932Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: duplicate requestVote for same term: term=2 2024-06-01T18:43:01.932Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: election won: term=2 tally=2 2024-06-01T18:43:01.932Z [INFO] server-a60041e1-149a-6479-acec-2f772d53e958: duplicate requestVote for same term: term=2 2024-06-01T18:43:01.932Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: entering leader state: leader="Node at 2a22570f-81d3-b6c5-6972-f51cf80e4572 [Leader]" 2024-06-01T18:43:01.932Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: added peer, starting replication: peer=server-cf494d8c-e1af-c757-a9ef-fece50e1a217 2024-06-01T18:43:01.932Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: added peer, starting replication: peer=server-a60041e1-149a-6479-acec-2f772d53e958 2024-06-01T18:43:01.932Z [WARN] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: rejecting vote request since we have a leader: from=cf494d8c-e1af-c757-a9ef-fece50e1a217 leader=2a22570f-81d3-b6c5-6972-f51cf80e4572 leader-id=server-2a22570f-81d3-b6c5-6972-f51cf80e4572 2024-06-01T18:43:01.932Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: entering follower state: follower="Node at cf494d8c-e1af-c757-a9ef-fece50e1a217 [Follower]" leader-address=2a22570f-81d3-b6c5-6972-f51cf80e4572 leader-id=server-2a22570f-81d3-b6c5-6972-f51cf80e4572 2024-06-01T18:43:01.932Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: pipelining replication: peer="{Voter server-cf494d8c-e1af-c757-a9ef-fece50e1a217 cf494d8c-e1af-c757-a9ef-fece50e1a217}" 2024-06-01T18:43:01.932Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: pipelining replication: peer="{Voter server-a60041e1-149a-6479-acec-2f772d53e958 a60041e1-149a-6479-acec-2f772d53e958}" 2024-06-01T18:43:02.038Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:01.932429539 +0000 UTC m=+97.430618991 (2 nodes), 57.939517ms from start of poll, 58.259983ms from cluster start. Timeout at 2024-06-01 18:43:02.038438417 +0000 UTC m=+97.536627869, 106.008878ms after stability 2024-06-01T18:43:02.038Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:02.144Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:02.038528666 +0000 UTC m=+97.536718118 (1 nodes), 1.553µs from start of poll, 164.35911ms from cluster start. Timeout at 2024-06-01 18:43:02.144461943 +0000 UTC m=+97.642651395, 105.933277ms after stability 2024-06-01T18:43:02.144Z [ERROR] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: failed to pipeline appendEntries: peer="{Voter server-cf494d8c-e1af-c757-a9ef-fece50e1a217 cf494d8c-e1af-c757-a9ef-fece50e1a217}" error="append pipeline closed" 2024-06-01T18:43:02.144Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: aborting pipeline replication: peer="{Voter server-cf494d8c-e1af-c757-a9ef-fece50e1a217 cf494d8c-e1af-c757-a9ef-fece50e1a217}" 2024-06-01T18:43:02.144Z [ERROR] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: failed to appendEntries to: peer="{Voter server-cf494d8c-e1af-c757-a9ef-fece50e1a217 cf494d8c-e1af-c757-a9ef-fece50e1a217}" error="failed to connect to peer: cf494d8c-e1af-c757-a9ef-fece50e1a217" raft_test.go:1669: [INFO] Finished apply without behind follower 2024-06-01T18:43:02.145Z [ERROR] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: failed to take snapshot: error="nothing new to snapshot" 2024-06-01T18:43:02.145Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: starting snapshot up to: index=102 2024-06-01T18:43:02.146Z [INFO] creating new snapshot: path=/tmp/raft636730121/snapshots/2-102-1717267382145.tmp 2024-06-01T18:43:02.147Z [ERROR] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: failed to heartbeat to: peer=cf494d8c-e1af-c757-a9ef-fece50e1a217 backoff time=10ms error="failed to connect to peer: cf494d8c-e1af-c757-a9ef-fece50e1a217" 2024-06-01T18:43:02.155Z [ERROR] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: failed to appendEntries to: peer="{Voter server-cf494d8c-e1af-c757-a9ef-fece50e1a217 cf494d8c-e1af-c757-a9ef-fece50e1a217}" error="failed to connect to peer: cf494d8c-e1af-c757-a9ef-fece50e1a217" 2024-06-01T18:43:02.162Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: compacting logs: from=1 to=92 2024-06-01T18:43:02.162Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: snapshot complete up to: index=102 2024-06-01T18:43:02.162Z [INFO] server-a60041e1-149a-6479-acec-2f772d53e958: starting snapshot up to: index=102 2024-06-01T18:43:02.162Z [INFO] creating new snapshot: path=/tmp/raft602745552/snapshots/2-102-1717267382162.tmp 2024-06-01T18:43:02.162Z [INFO] server-a60041e1-149a-6479-acec-2f772d53e958: compacting logs: from=1 to=92 2024-06-01T18:43:02.162Z [INFO] server-a60041e1-149a-6479-acec-2f772d53e958: snapshot complete up to: index=102 raft_test.go:1690: [INFO] Finished apply without behind follower 2024-06-01T18:43:02.165Z [ERROR] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: failed to get log: index=2 error="log not found" 2024-06-01T18:43:02.166Z [INFO] creating new snapshot: path=/tmp/raft603739069/snapshots/2-102-1717267382166.tmp 2024-06-01T18:43:02.166Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2024-06-01T18:43:02.166Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: copied to local snapshot: bytes=693 2024-06-01T18:43:02.166Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: snapshot restore progress: id=2-102-1717267382166 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2024-06-01T18:43:02.166Z [INFO] server-cf494d8c-e1af-c757-a9ef-fece50e1a217: Installed remote snapshot 2024-06-01T18:43:02.166Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: pipelining replication: peer="{Voter server-cf494d8c-e1af-c757-a9ef-fece50e1a217 cf494d8c-e1af-c757-a9ef-fece50e1a217}" 2024-06-01T18:43:02.171Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: aborting pipeline replication: peer="{Voter server-a60041e1-149a-6479-acec-2f772d53e958 a60041e1-149a-6479-acec-2f772d53e958}" 2024-06-01T18:43:02.172Z [INFO] server-2a22570f-81d3-b6c5-6972-f51cf80e4572: aborting pipeline replication: peer="{Voter server-cf494d8c-e1af-c757-a9ef-fece50e1a217 cf494d8c-e1af-c757-a9ef-fece50e1a217}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.37s) === RUN TestRaft_ReJoinFollower 2024-06-01T18:43:02.242Z [INFO] server-95545c40-be20-1b7b-99ed-af95031cbf83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95545c40-be20-1b7b-99ed-af95031cbf83 Address:95545c40-be20-1b7b-99ed-af95031cbf83} {Suffrage:Voter ID:server-5437a68b-6bde-c67c-8c9c-bd19c5155029 Address:5437a68b-6bde-c67c-8c9c-bd19c5155029} {Suffrage:Voter ID:server-809e565f-8fda-7462-1b72-3ace006a2545 Address:809e565f-8fda-7462-1b72-3ace006a2545}]" 2024-06-01T18:43:02.242Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95545c40-be20-1b7b-99ed-af95031cbf83 Address:95545c40-be20-1b7b-99ed-af95031cbf83} {Suffrage:Voter ID:server-5437a68b-6bde-c67c-8c9c-bd19c5155029 Address:5437a68b-6bde-c67c-8c9c-bd19c5155029} {Suffrage:Voter ID:server-809e565f-8fda-7462-1b72-3ace006a2545 Address:809e565f-8fda-7462-1b72-3ace006a2545}]" 2024-06-01T18:43:02.242Z [INFO] server-95545c40-be20-1b7b-99ed-af95031cbf83: entering follower state: follower="Node at 95545c40-be20-1b7b-99ed-af95031cbf83 [Follower]" leader-address= leader-id= 2024-06-01T18:43:02.243Z [INFO] server-809e565f-8fda-7462-1b72-3ace006a2545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95545c40-be20-1b7b-99ed-af95031cbf83 Address:95545c40-be20-1b7b-99ed-af95031cbf83} {Suffrage:Voter ID:server-5437a68b-6bde-c67c-8c9c-bd19c5155029 Address:5437a68b-6bde-c67c-8c9c-bd19c5155029} {Suffrage:Voter ID:server-809e565f-8fda-7462-1b72-3ace006a2545 Address:809e565f-8fda-7462-1b72-3ace006a2545}]" 2024-06-01T18:43:02.243Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:02.243Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: entering follower state: follower="Node at 5437a68b-6bde-c67c-8c9c-bd19c5155029 [Follower]" leader-address= leader-id= 2024-06-01T18:43:02.243Z [INFO] server-809e565f-8fda-7462-1b72-3ace006a2545: entering follower state: follower="Node at 809e565f-8fda-7462-1b72-3ace006a2545 [Follower]" leader-address= leader-id= 2024-06-01T18:43:02.294Z [WARN] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:02.294Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: entering candidate state: node="Node at 5437a68b-6bde-c67c-8c9c-bd19c5155029 [Candidate]" term=2 2024-06-01T18:43:02.294Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: election won: term=2 tally=2 2024-06-01T18:43:02.294Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: entering leader state: leader="Node at 5437a68b-6bde-c67c-8c9c-bd19c5155029 [Leader]" 2024-06-01T18:43:02.294Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: added peer, starting replication: peer=server-95545c40-be20-1b7b-99ed-af95031cbf83 2024-06-01T18:43:02.294Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: added peer, starting replication: peer=server-809e565f-8fda-7462-1b72-3ace006a2545 2024-06-01T18:43:02.294Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: pipelining replication: peer="{Voter server-809e565f-8fda-7462-1b72-3ace006a2545 809e565f-8fda-7462-1b72-3ace006a2545}" 2024-06-01T18:43:02.294Z [WARN] server-95545c40-be20-1b7b-99ed-af95031cbf83: rejecting vote request since our last term is greater: candidate=5437a68b-6bde-c67c-8c9c-bd19c5155029 last-term=2 last-candidate-term=1 2024-06-01T18:43:02.294Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: pipelining replication: peer="{Voter server-95545c40-be20-1b7b-99ed-af95031cbf83 95545c40-be20-1b7b-99ed-af95031cbf83}" 2024-06-01T18:43:02.400Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:02.294644151 +0000 UTC m=+97.792833604 (1 nodes), 51.533721ms from start of poll, 52.030367ms from cluster start. Timeout at 2024-06-01 18:43:02.400664422 +0000 UTC m=+97.898853874, 106.02027ms after stability 2024-06-01T18:43:02.406Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:02.512Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:02.406421158 +0000 UTC m=+97.904610600 (2 nodes), 1.713µs from start of poll, 163.807363ms from cluster start. Timeout at 2024-06-01 18:43:02.512226207 +0000 UTC m=+98.010415659, 105.805059ms after stability 2024-06-01T18:43:02.512Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: updating configuration: command=RemoveServer server-id=server-95545c40-be20-1b7b-99ed-af95031cbf83 server-addr= servers="[{Suffrage:Voter ID:server-5437a68b-6bde-c67c-8c9c-bd19c5155029 Address:5437a68b-6bde-c67c-8c9c-bd19c5155029} {Suffrage:Voter ID:server-809e565f-8fda-7462-1b72-3ace006a2545 Address:809e565f-8fda-7462-1b72-3ace006a2545}]" 2024-06-01T18:43:02.512Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: removed peer, stopping replication: peer=server-95545c40-be20-1b7b-99ed-af95031cbf83 last-index=3 2024-06-01T18:43:02.512Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: aborting pipeline replication: peer="{Voter server-95545c40-be20-1b7b-99ed-af95031cbf83 95545c40-be20-1b7b-99ed-af95031cbf83}" 2024-06-01T18:43:02.573Z [WARN] server-95545c40-be20-1b7b-99ed-af95031cbf83: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2024-06-01T18:43:02.617Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: updating configuration: command=AddVoter server-id=server-95545c40-be20-1b7b-99ed-af95031cbf83 server-addr=95545c40-be20-1b7b-99ed-af95031cbf83 servers="[{Suffrage:Voter ID:server-5437a68b-6bde-c67c-8c9c-bd19c5155029 Address:5437a68b-6bde-c67c-8c9c-bd19c5155029} {Suffrage:Voter ID:server-809e565f-8fda-7462-1b72-3ace006a2545 Address:809e565f-8fda-7462-1b72-3ace006a2545} {Suffrage:Voter ID:server-95545c40-be20-1b7b-99ed-af95031cbf83 Address:95545c40-be20-1b7b-99ed-af95031cbf83}]" 2024-06-01T18:43:02.617Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: added peer, starting replication: peer=server-95545c40-be20-1b7b-99ed-af95031cbf83 2024-06-01T18:43:02.617Z [WARN] server-95545c40-be20-1b7b-99ed-af95031cbf83: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2024-06-01T18:43:02.617Z [WARN] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: appendEntries rejected, sending older logs: peer="{Voter server-95545c40-be20-1b7b-99ed-af95031cbf83 95545c40-be20-1b7b-99ed-af95031cbf83}" next=4 2024-06-01T18:43:02.617Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: pipelining replication: peer="{Voter server-95545c40-be20-1b7b-99ed-af95031cbf83 95545c40-be20-1b7b-99ed-af95031cbf83}" 2024-06-01T18:43:02.617Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:02.723Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:02.617787083 +0000 UTC m=+98.115976525 (1 nodes), 752ns from start of poll, 375.173288ms from cluster start. Timeout at 2024-06-01 18:43:02.723668473 +0000 UTC m=+98.221857925, 105.8814ms after stability 2024-06-01T18:43:02.723Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: aborting pipeline replication: peer="{Voter server-95545c40-be20-1b7b-99ed-af95031cbf83 95545c40-be20-1b7b-99ed-af95031cbf83}" 2024-06-01T18:43:02.723Z [INFO] server-5437a68b-6bde-c67c-8c9c-bd19c5155029: aborting pipeline replication: peer="{Voter server-809e565f-8fda-7462-1b72-3ace006a2545 809e565f-8fda-7462-1b72-3ace006a2545}" --- PASS: TestRaft_ReJoinFollower (0.51s) === RUN TestRaft_LeaderLeaseExpire 2024-06-01T18:43:02.748Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 Address:fdaa0c63-d02a-b39f-f870-7f7bffd94ad0} {Suffrage:Voter ID:server-e1399ffe-c647-814f-a787-1dbcc8f975fd Address:e1399ffe-c647-814f-a787-1dbcc8f975fd}]" 2024-06-01T18:43:02.748Z [INFO] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 Address:fdaa0c63-d02a-b39f-f870-7f7bffd94ad0} {Suffrage:Voter ID:server-e1399ffe-c647-814f-a787-1dbcc8f975fd Address:e1399ffe-c647-814f-a787-1dbcc8f975fd}]" 2024-06-01T18:43:02.748Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:02.748Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering follower state: follower="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Follower]" leader-address= leader-id= 2024-06-01T18:43:02.748Z [INFO] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: entering follower state: follower="Node at e1399ffe-c647-814f-a787-1dbcc8f975fd [Follower]" leader-address= leader-id= 2024-06-01T18:43:02.825Z [WARN] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:02.825Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering candidate state: node="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Candidate]" term=2 2024-06-01T18:43:02.825Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: election won: term=2 tally=2 2024-06-01T18:43:02.825Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering leader state: leader="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Leader]" 2024-06-01T18:43:02.825Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: added peer, starting replication: peer=server-e1399ffe-c647-814f-a787-1dbcc8f975fd 2024-06-01T18:43:02.826Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: pipelining replication: peer="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" 2024-06-01T18:43:02.931Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:02.825952007 +0000 UTC m=+98.324141459 (1 nodes), 77.435759ms from start of poll, 77.643495ms from cluster start. Timeout at 2024-06-01 18:43:02.931814312 +0000 UTC m=+98.430003764, 105.862305ms after stability 2024-06-01T18:43:02.937Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:03.043Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:02.937639938 +0000 UTC m=+98.435829390 (1 nodes), 1.643µs from start of poll, 189.331426ms from cluster start. Timeout at 2024-06-01 18:43:03.043405773 +0000 UTC m=+98.541595225, 105.765835ms after stability raft_test.go:1799: [INFO] Disconnecting Node at e1399ffe-c647-814f-a787-1dbcc8f975fd [Follower] 2024-06-01T18:43:03.047Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to heartbeat to: peer=e1399ffe-c647-814f-a787-1dbcc8f975fd backoff time=10ms error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" 2024-06-01T18:43:03.052Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to pipeline appendEntries: peer="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="append pipeline closed" 2024-06-01T18:43:03.052Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: aborting pipeline replication: peer="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" 2024-06-01T18:43:03.062Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to appendEntries to: peer="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" 2024-06-01T18:43:03.067Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to heartbeat to: peer=e1399ffe-c647-814f-a787-1dbcc8f975fd backoff time=10ms error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" 2024-06-01T18:43:03.078Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to appendEntries to: peer="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" 2024-06-01T18:43:03.086Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to heartbeat to: peer=e1399ffe-c647-814f-a787-1dbcc8f975fd backoff time=10ms error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" 2024-06-01T18:43:03.089Z [WARN] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to contact: server-id=server-e1399ffe-c647-814f-a787-1dbcc8f975fd time=50.333418ms 2024-06-01T18:43:03.089Z [WARN] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to contact quorum of nodes, stepping down 2024-06-01T18:43:03.090Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering follower state: follower="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Follower]" leader-address= leader-id= 2024-06-01T18:43:03.090Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:03.095Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to appendEntries to: peer="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" 2024-06-01T18:43:03.149Z [WARN] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: heartbeat timeout reached, starting election: last-leader-addr=fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 last-leader-id=server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 2024-06-01T18:43:03.149Z [INFO] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: entering candidate state: node="Node at e1399ffe-c647-814f-a787-1dbcc8f975fd [Candidate]" term=3 2024-06-01T18:43:03.149Z [ERROR] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: failed to make requestVote RPC: target="{Voter server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 fdaa0c63-d02a-b39f-f870-7f7bffd94ad0}" error="failed to connect to peer: fdaa0c63-d02a-b39f-f870-7f7bffd94ad0" term=3 2024-06-01T18:43:03.164Z [WARN] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:03.164Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering candidate state: node="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Candidate]" term=3 2024-06-01T18:43:03.164Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to make requestVote RPC: target="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" term=3 2024-06-01T18:43:03.240Z [WARN] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: Election timeout reached, restarting election 2024-06-01T18:43:03.240Z [INFO] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: entering candidate state: node="Node at e1399ffe-c647-814f-a787-1dbcc8f975fd [Candidate]" term=4 2024-06-01T18:43:03.240Z [ERROR] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: failed to make requestVote RPC: target="{Voter server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 fdaa0c63-d02a-b39f-f870-7f7bffd94ad0}" error="failed to connect to peer: fdaa0c63-d02a-b39f-f870-7f7bffd94ad0" term=4 2024-06-01T18:43:03.241Z [WARN] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: Election timeout reached, restarting election 2024-06-01T18:43:03.241Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering candidate state: node="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Candidate]" term=4 2024-06-01T18:43:03.241Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to make requestVote RPC: target="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" term=4 2024-06-01T18:43:03.269Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:03.164397503 +0000 UTC m=+98.662586955 (0 nodes), 74.343312ms from start of poll, 416.088991ms from cluster start. Timeout at 2024-06-01 18:43:03.269457586 +0000 UTC m=+98.767647038, 105.060083ms after stability 2024-06-01T18:43:03.295Z [WARN] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: Election timeout reached, restarting election 2024-06-01T18:43:03.295Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering candidate state: node="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Candidate]" term=5 2024-06-01T18:43:03.295Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to make requestVote RPC: target="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" term=5 2024-06-01T18:43:03.304Z [WARN] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: Election timeout reached, restarting election 2024-06-01T18:43:03.304Z [INFO] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: entering candidate state: node="Node at e1399ffe-c647-814f-a787-1dbcc8f975fd [Candidate]" term=5 2024-06-01T18:43:03.304Z [ERROR] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: failed to make requestVote RPC: target="{Voter server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 fdaa0c63-d02a-b39f-f870-7f7bffd94ad0}" error="failed to connect to peer: fdaa0c63-d02a-b39f-f870-7f7bffd94ad0" term=5 2024-06-01T18:43:03.350Z [WARN] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: Election timeout reached, restarting election 2024-06-01T18:43:03.350Z [INFO] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: entering candidate state: node="Node at fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 [Candidate]" term=6 2024-06-01T18:43:03.350Z [ERROR] server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0: failed to make requestVote RPC: target="{Voter server-e1399ffe-c647-814f-a787-1dbcc8f975fd e1399ffe-c647-814f-a787-1dbcc8f975fd}" error="failed to connect to peer: e1399ffe-c647-814f-a787-1dbcc8f975fd" term=6 2024-06-01T18:43:03.369Z [WARN] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: Election timeout reached, restarting election 2024-06-01T18:43:03.369Z [INFO] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: entering candidate state: node="Node at e1399ffe-c647-814f-a787-1dbcc8f975fd [Candidate]" term=6 2024-06-01T18:43:03.369Z [ERROR] server-e1399ffe-c647-814f-a787-1dbcc8f975fd: failed to make requestVote RPC: target="{Voter server-fdaa0c63-d02a-b39f-f870-7f7bffd94ad0 fdaa0c63-d02a-b39f-f870-7f7bffd94ad0}" error="failed to connect to peer: fdaa0c63-d02a-b39f-f870-7f7bffd94ad0" term=6 --- PASS: TestRaft_LeaderLeaseExpire (0.67s) === RUN TestRaft_Barrier 2024-06-01T18:43:03.414Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1af2daf2-2fb1-7927-7303-dec8da8b2afa Address:1af2daf2-2fb1-7927-7303-dec8da8b2afa} {Suffrage:Voter ID:server-3eca1482-17ff-941c-ea0f-b7058dc1b345 Address:3eca1482-17ff-941c-ea0f-b7058dc1b345} {Suffrage:Voter ID:server-c123cc3c-0915-4cc1-1449-bf32e77f8902 Address:c123cc3c-0915-4cc1-1449-bf32e77f8902}]" 2024-06-01T18:43:03.414Z [INFO] server-3eca1482-17ff-941c-ea0f-b7058dc1b345: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1af2daf2-2fb1-7927-7303-dec8da8b2afa Address:1af2daf2-2fb1-7927-7303-dec8da8b2afa} {Suffrage:Voter ID:server-3eca1482-17ff-941c-ea0f-b7058dc1b345 Address:3eca1482-17ff-941c-ea0f-b7058dc1b345} {Suffrage:Voter ID:server-c123cc3c-0915-4cc1-1449-bf32e77f8902 Address:c123cc3c-0915-4cc1-1449-bf32e77f8902}]" 2024-06-01T18:43:03.414Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: entering follower state: follower="Node at 1af2daf2-2fb1-7927-7303-dec8da8b2afa [Follower]" leader-address= leader-id= 2024-06-01T18:43:03.414Z [INFO] server-c123cc3c-0915-4cc1-1449-bf32e77f8902: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1af2daf2-2fb1-7927-7303-dec8da8b2afa Address:1af2daf2-2fb1-7927-7303-dec8da8b2afa} {Suffrage:Voter ID:server-3eca1482-17ff-941c-ea0f-b7058dc1b345 Address:3eca1482-17ff-941c-ea0f-b7058dc1b345} {Suffrage:Voter ID:server-c123cc3c-0915-4cc1-1449-bf32e77f8902 Address:c123cc3c-0915-4cc1-1449-bf32e77f8902}]" 2024-06-01T18:43:03.414Z [INFO] server-3eca1482-17ff-941c-ea0f-b7058dc1b345: entering follower state: follower="Node at 3eca1482-17ff-941c-ea0f-b7058dc1b345 [Follower]" leader-address= leader-id= 2024-06-01T18:43:03.414Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:03.414Z [INFO] server-c123cc3c-0915-4cc1-1449-bf32e77f8902: entering follower state: follower="Node at c123cc3c-0915-4cc1-1449-bf32e77f8902 [Follower]" leader-address= leader-id= 2024-06-01T18:43:03.473Z [WARN] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:03.473Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: entering candidate state: node="Node at 1af2daf2-2fb1-7927-7303-dec8da8b2afa [Candidate]" term=2 2024-06-01T18:43:03.473Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: election won: term=2 tally=2 2024-06-01T18:43:03.473Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: entering leader state: leader="Node at 1af2daf2-2fb1-7927-7303-dec8da8b2afa [Leader]" 2024-06-01T18:43:03.473Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: added peer, starting replication: peer=server-3eca1482-17ff-941c-ea0f-b7058dc1b345 2024-06-01T18:43:03.473Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: added peer, starting replication: peer=server-c123cc3c-0915-4cc1-1449-bf32e77f8902 2024-06-01T18:43:03.473Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: pipelining replication: peer="{Voter server-c123cc3c-0915-4cc1-1449-bf32e77f8902 c123cc3c-0915-4cc1-1449-bf32e77f8902}" 2024-06-01T18:43:03.473Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: pipelining replication: peer="{Voter server-3eca1482-17ff-941c-ea0f-b7058dc1b345 3eca1482-17ff-941c-ea0f-b7058dc1b345}" 2024-06-01T18:43:03.579Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:03.473532026 +0000 UTC m=+98.971721478 (1 nodes), 58.705662ms from start of poll, 59.103493ms from cluster start. Timeout at 2024-06-01 18:43:03.579329121 +0000 UTC m=+99.077518573, 105.797095ms after stability 2024-06-01T18:43:03.592Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: aborting pipeline replication: peer="{Voter server-c123cc3c-0915-4cc1-1449-bf32e77f8902 c123cc3c-0915-4cc1-1449-bf32e77f8902}" 2024-06-01T18:43:03.592Z [INFO] server-1af2daf2-2fb1-7927-7303-dec8da8b2afa: aborting pipeline replication: peer="{Voter server-3eca1482-17ff-941c-ea0f-b7058dc1b345 3eca1482-17ff-941c-ea0f-b7058dc1b345}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader 2024-06-01T18:43:03.593Z [INFO] server-1973cea2-5436-d250-a686-3ae037c7d39f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1973cea2-5436-d250-a686-3ae037c7d39f Address:1973cea2-5436-d250-a686-3ae037c7d39f} {Suffrage:Voter ID:server-fbd50d20-174f-c676-2541-9994ff3b8ac3 Address:fbd50d20-174f-c676-2541-9994ff3b8ac3} {Suffrage:Voter ID:server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411 Address:e21d1ef1-f7d9-1b81-76e4-7597b0f30411}]" 2024-06-01T18:43:03.593Z [INFO] server-fbd50d20-174f-c676-2541-9994ff3b8ac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1973cea2-5436-d250-a686-3ae037c7d39f Address:1973cea2-5436-d250-a686-3ae037c7d39f} {Suffrage:Voter ID:server-fbd50d20-174f-c676-2541-9994ff3b8ac3 Address:fbd50d20-174f-c676-2541-9994ff3b8ac3} {Suffrage:Voter ID:server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411 Address:e21d1ef1-f7d9-1b81-76e4-7597b0f30411}]" 2024-06-01T18:43:03.593Z [INFO] server-1973cea2-5436-d250-a686-3ae037c7d39f: entering follower state: follower="Node at 1973cea2-5436-d250-a686-3ae037c7d39f [Follower]" leader-address= leader-id= 2024-06-01T18:43:03.593Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1973cea2-5436-d250-a686-3ae037c7d39f Address:1973cea2-5436-d250-a686-3ae037c7d39f} {Suffrage:Voter ID:server-fbd50d20-174f-c676-2541-9994ff3b8ac3 Address:fbd50d20-174f-c676-2541-9994ff3b8ac3} {Suffrage:Voter ID:server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411 Address:e21d1ef1-f7d9-1b81-76e4-7597b0f30411}]" 2024-06-01T18:43:03.593Z [INFO] server-fbd50d20-174f-c676-2541-9994ff3b8ac3: entering follower state: follower="Node at fbd50d20-174f-c676-2541-9994ff3b8ac3 [Follower]" leader-address= leader-id= 2024-06-01T18:43:03.593Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:03.593Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: entering follower state: follower="Node at e21d1ef1-f7d9-1b81-76e4-7597b0f30411 [Follower]" leader-address= leader-id= 2024-06-01T18:43:03.652Z [WARN] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:03.652Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: entering candidate state: node="Node at e21d1ef1-f7d9-1b81-76e4-7597b0f30411 [Candidate]" term=2 2024-06-01T18:43:03.652Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: election won: term=2 tally=2 2024-06-01T18:43:03.652Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: entering leader state: leader="Node at e21d1ef1-f7d9-1b81-76e4-7597b0f30411 [Leader]" 2024-06-01T18:43:03.652Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: added peer, starting replication: peer=server-1973cea2-5436-d250-a686-3ae037c7d39f 2024-06-01T18:43:03.652Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: added peer, starting replication: peer=server-fbd50d20-174f-c676-2541-9994ff3b8ac3 2024-06-01T18:43:03.652Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: pipelining replication: peer="{Voter server-fbd50d20-174f-c676-2541-9994ff3b8ac3 fbd50d20-174f-c676-2541-9994ff3b8ac3}" 2024-06-01T18:43:03.652Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: pipelining replication: peer="{Voter server-1973cea2-5436-d250-a686-3ae037c7d39f 1973cea2-5436-d250-a686-3ae037c7d39f}" 2024-06-01T18:43:03.757Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:03.652599516 +0000 UTC m=+99.150788968 (1 nodes), 59.152854ms from start of poll, 59.401417ms from cluster start. Timeout at 2024-06-01 18:43:03.757894395 +0000 UTC m=+99.256083847, 105.294879ms after stability 2024-06-01T18:43:03.758Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: aborting pipeline replication: peer="{Voter server-fbd50d20-174f-c676-2541-9994ff3b8ac3 fbd50d20-174f-c676-2541-9994ff3b8ac3}" 2024-06-01T18:43:03.758Z [INFO] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: aborting pipeline replication: peer="{Voter server-1973cea2-5436-d250-a686-3ae037c7d39f 1973cea2-5436-d250-a686-3ae037c7d39f}" 2024-06-01T18:43:04.258Z [ERROR] server-e21d1ef1-f7d9-1b81-76e4-7597b0f30411: failed to heartbeat to: peer=1973cea2-5436-d250-a686-3ae037c7d39f backoff time=10ms error="command timed out" --- PASS: TestRaft_VerifyLeader (0.67s) === RUN TestRaft_VerifyLeader_Single 2024-06-01T18:43:04.259Z [INFO] server-0558f26c-0c2b-9c97-4e9d-e467514a321a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0558f26c-0c2b-9c97-4e9d-e467514a321a Address:0558f26c-0c2b-9c97-4e9d-e467514a321a}]" 2024-06-01T18:43:04.259Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:04.259Z [INFO] server-0558f26c-0c2b-9c97-4e9d-e467514a321a: entering follower state: follower="Node at 0558f26c-0c2b-9c97-4e9d-e467514a321a [Follower]" leader-address= leader-id= 2024-06-01T18:43:04.335Z [WARN] server-0558f26c-0c2b-9c97-4e9d-e467514a321a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:04.335Z [INFO] server-0558f26c-0c2b-9c97-4e9d-e467514a321a: entering candidate state: node="Node at 0558f26c-0c2b-9c97-4e9d-e467514a321a [Candidate]" term=2 2024-06-01T18:43:04.335Z [INFO] server-0558f26c-0c2b-9c97-4e9d-e467514a321a: election won: term=2 tally=1 2024-06-01T18:43:04.335Z [INFO] server-0558f26c-0c2b-9c97-4e9d-e467514a321a: entering leader state: leader="Node at 0558f26c-0c2b-9c97-4e9d-e467514a321a [Leader]" 2024-06-01T18:43:04.441Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:04.335844887 +0000 UTC m=+99.834034339 (1 nodes), 76.487583ms from start of poll, 76.648723ms from cluster start. Timeout at 2024-06-01 18:43:04.441208654 +0000 UTC m=+99.939398097, 105.363758ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail 2024-06-01T18:43:04.443Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d809d893-2f51-effe-5322-e5bb00bec1e3 Address:d809d893-2f51-effe-5322-e5bb00bec1e3} {Suffrage:Voter ID:server-67f1d165-d79b-debb-64c8-e29be90aac33 Address:67f1d165-d79b-debb-64c8-e29be90aac33}]" 2024-06-01T18:43:04.443Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: entering follower state: follower="Node at d809d893-2f51-effe-5322-e5bb00bec1e3 [Follower]" leader-address= leader-id= 2024-06-01T18:43:04.444Z [INFO] server-67f1d165-d79b-debb-64c8-e29be90aac33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d809d893-2f51-effe-5322-e5bb00bec1e3 Address:d809d893-2f51-effe-5322-e5bb00bec1e3} {Suffrage:Voter ID:server-67f1d165-d79b-debb-64c8-e29be90aac33 Address:67f1d165-d79b-debb-64c8-e29be90aac33}]" 2024-06-01T18:43:04.444Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:04.444Z [INFO] server-67f1d165-d79b-debb-64c8-e29be90aac33: entering follower state: follower="Node at 67f1d165-d79b-debb-64c8-e29be90aac33 [Follower]" leader-address= leader-id= 2024-06-01T18:43:04.506Z [WARN] server-d809d893-2f51-effe-5322-e5bb00bec1e3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:04.506Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: entering candidate state: node="Node at d809d893-2f51-effe-5322-e5bb00bec1e3 [Candidate]" term=2 2024-06-01T18:43:04.506Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: election won: term=2 tally=2 2024-06-01T18:43:04.506Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: entering leader state: leader="Node at d809d893-2f51-effe-5322-e5bb00bec1e3 [Leader]" 2024-06-01T18:43:04.506Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: added peer, starting replication: peer=server-67f1d165-d79b-debb-64c8-e29be90aac33 2024-06-01T18:43:04.506Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: pipelining replication: peer="{Voter server-67f1d165-d79b-debb-64c8-e29be90aac33 67f1d165-d79b-debb-64c8-e29be90aac33}" 2024-06-01T18:43:04.611Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:04.506141509 +0000 UTC m=+100.004330961 (1 nodes), 62.004332ms from start of poll, 62.42186ms from cluster start. Timeout at 2024-06-01 18:43:04.611575247 +0000 UTC m=+100.109764699, 105.433738ms after stability 2024-06-01T18:43:04.611Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:04.717Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:04.611719946 +0000 UTC m=+100.109909408 (1 nodes), 2.584µs from start of poll, 168.000307ms from cluster start. Timeout at 2024-06-01 18:43:04.717357063 +0000 UTC m=+100.215546515, 105.637107ms after stability 2024-06-01T18:43:04.722Z [ERROR] server-d809d893-2f51-effe-5322-e5bb00bec1e3: peer has newer term, stopping replication: peer="{Voter server-67f1d165-d79b-debb-64c8-e29be90aac33 67f1d165-d79b-debb-64c8-e29be90aac33}" 2024-06-01T18:43:04.722Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: aborting pipeline replication: peer="{Voter server-67f1d165-d79b-debb-64c8-e29be90aac33 67f1d165-d79b-debb-64c8-e29be90aac33}" 2024-06-01T18:43:04.722Z [INFO] server-d809d893-2f51-effe-5322-e5bb00bec1e3: entering follower state: follower="Node at d809d893-2f51-effe-5322-e5bb00bec1e3 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect 2024-06-01T18:43:04.725Z [INFO] server-2bd6c4d5-017c-e461-06a6-00da522683b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd6c4d5-017c-e461-06a6-00da522683b2 Address:2bd6c4d5-017c-e461-06a6-00da522683b2} {Suffrage:Voter ID:server-067e89cd-db74-6666-e112-ffee7515953e Address:067e89cd-db74-6666-e112-ffee7515953e} {Suffrage:Voter ID:server-3772f7fa-be91-6737-183b-53767dc1b8f0 Address:3772f7fa-be91-6737-183b-53767dc1b8f0}]" 2024-06-01T18:43:04.725Z [INFO] server-067e89cd-db74-6666-e112-ffee7515953e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd6c4d5-017c-e461-06a6-00da522683b2 Address:2bd6c4d5-017c-e461-06a6-00da522683b2} {Suffrage:Voter ID:server-067e89cd-db74-6666-e112-ffee7515953e Address:067e89cd-db74-6666-e112-ffee7515953e} {Suffrage:Voter ID:server-3772f7fa-be91-6737-183b-53767dc1b8f0 Address:3772f7fa-be91-6737-183b-53767dc1b8f0}]" 2024-06-01T18:43:04.725Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2bd6c4d5-017c-e461-06a6-00da522683b2 Address:2bd6c4d5-017c-e461-06a6-00da522683b2} {Suffrage:Voter ID:server-067e89cd-db74-6666-e112-ffee7515953e Address:067e89cd-db74-6666-e112-ffee7515953e} {Suffrage:Voter ID:server-3772f7fa-be91-6737-183b-53767dc1b8f0 Address:3772f7fa-be91-6737-183b-53767dc1b8f0}]" 2024-06-01T18:43:04.725Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:04.725Z [INFO] server-2bd6c4d5-017c-e461-06a6-00da522683b2: entering follower state: follower="Node at 2bd6c4d5-017c-e461-06a6-00da522683b2 [Follower]" leader-address= leader-id= 2024-06-01T18:43:04.725Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: entering follower state: follower="Node at 3772f7fa-be91-6737-183b-53767dc1b8f0 [Follower]" leader-address= leader-id= 2024-06-01T18:43:04.725Z [INFO] server-067e89cd-db74-6666-e112-ffee7515953e: entering follower state: follower="Node at 067e89cd-db74-6666-e112-ffee7515953e [Follower]" leader-address= leader-id= 2024-06-01T18:43:04.801Z [WARN] server-3772f7fa-be91-6737-183b-53767dc1b8f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:04.801Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: entering candidate state: node="Node at 3772f7fa-be91-6737-183b-53767dc1b8f0 [Candidate]" term=2 2024-06-01T18:43:04.801Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: election won: term=2 tally=2 2024-06-01T18:43:04.801Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: entering leader state: leader="Node at 3772f7fa-be91-6737-183b-53767dc1b8f0 [Leader]" 2024-06-01T18:43:04.801Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: added peer, starting replication: peer=server-2bd6c4d5-017c-e461-06a6-00da522683b2 2024-06-01T18:43:04.801Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: added peer, starting replication: peer=server-067e89cd-db74-6666-e112-ffee7515953e 2024-06-01T18:43:04.801Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: pipelining replication: peer="{Voter server-067e89cd-db74-6666-e112-ffee7515953e 067e89cd-db74-6666-e112-ffee7515953e}" 2024-06-01T18:43:04.802Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: pipelining replication: peer="{Voter server-2bd6c4d5-017c-e461-06a6-00da522683b2 2bd6c4d5-017c-e461-06a6-00da522683b2}" 2024-06-01T18:43:04.907Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:04.8021565 +0000 UTC m=+100.300345952 (1 nodes), 76.514043ms from start of poll, 76.77021ms from cluster start. Timeout at 2024-06-01 18:43:04.907536989 +0000 UTC m=+100.405726441, 105.380489ms after stability 2024-06-01T18:43:04.913Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:05.018Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:04.913878223 +0000 UTC m=+100.412067675 (2 nodes), 2.355µs from start of poll, 188.491933ms from cluster start. Timeout at 2024-06-01 18:43:05.018886729 +0000 UTC m=+100.517076181, 105.008506ms after stability raft_test.go:1968: [INFO] Disconnecting Node at 2bd6c4d5-017c-e461-06a6-00da522683b2 [Follower] 2024-06-01T18:43:05.019Z [ERROR] server-3772f7fa-be91-6737-183b-53767dc1b8f0: failed to heartbeat to: peer=2bd6c4d5-017c-e461-06a6-00da522683b2 backoff time=10ms error="failed to connect to peer: 2bd6c4d5-017c-e461-06a6-00da522683b2" 2024-06-01T18:43:05.019Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: aborting pipeline replication: peer="{Voter server-067e89cd-db74-6666-e112-ffee7515953e 067e89cd-db74-6666-e112-ffee7515953e}" 2024-06-01T18:43:05.019Z [INFO] server-3772f7fa-be91-6737-183b-53767dc1b8f0: aborting pipeline replication: peer="{Voter server-2bd6c4d5-017c-e461-06a6-00da522683b2 2bd6c4d5-017c-e461-06a6-00da522683b2}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) === RUN TestRaft_NotifyCh 2024-06-01T18:43:05.019Z [INFO] server-a1a0ae7f-c250-1974-6c40-9052995d6135: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1a0ae7f-c250-1974-6c40-9052995d6135 Address:a1a0ae7f-c250-1974-6c40-9052995d6135}]" 2024-06-01T18:43:05.019Z [INFO] server-a1a0ae7f-c250-1974-6c40-9052995d6135: entering follower state: follower="Node at a1a0ae7f-c250-1974-6c40-9052995d6135 [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.097Z [WARN] server-a1a0ae7f-c250-1974-6c40-9052995d6135: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:05.097Z [INFO] server-a1a0ae7f-c250-1974-6c40-9052995d6135: entering candidate state: node="Node at a1a0ae7f-c250-1974-6c40-9052995d6135 [Candidate]" term=2 2024-06-01T18:43:05.097Z [INFO] server-a1a0ae7f-c250-1974-6c40-9052995d6135: election won: term=2 tally=1 2024-06-01T18:43:05.097Z [INFO] server-a1a0ae7f-c250-1974-6c40-9052995d6135: entering leader state: leader="Node at a1a0ae7f-c250-1974-6c40-9052995d6135 [Leader]" --- PASS: TestRaft_NotifyCh (0.08s) === RUN TestRaft_AppendEntry 2024-06-01T18:43:05.099Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e Address:45755dc7-b3f0-fba1-c2bf-ae8d606c233e} {Suffrage:Voter ID:server-f6716691-1789-c730-129d-a78f1d9a9c62 Address:f6716691-1789-c730-129d-a78f1d9a9c62} {Suffrage:Voter ID:server-860aaa3c-c561-9ee6-74b1-05092da7ef5e Address:860aaa3c-c561-9ee6-74b1-05092da7ef5e}]" 2024-06-01T18:43:05.099Z [INFO] server-f6716691-1789-c730-129d-a78f1d9a9c62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e Address:45755dc7-b3f0-fba1-c2bf-ae8d606c233e} {Suffrage:Voter ID:server-f6716691-1789-c730-129d-a78f1d9a9c62 Address:f6716691-1789-c730-129d-a78f1d9a9c62} {Suffrage:Voter ID:server-860aaa3c-c561-9ee6-74b1-05092da7ef5e Address:860aaa3c-c561-9ee6-74b1-05092da7ef5e}]" 2024-06-01T18:43:05.099Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: entering follower state: follower="Node at 45755dc7-b3f0-fba1-c2bf-ae8d606c233e [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.099Z [INFO] server-860aaa3c-c561-9ee6-74b1-05092da7ef5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e Address:45755dc7-b3f0-fba1-c2bf-ae8d606c233e} {Suffrage:Voter ID:server-f6716691-1789-c730-129d-a78f1d9a9c62 Address:f6716691-1789-c730-129d-a78f1d9a9c62} {Suffrage:Voter ID:server-860aaa3c-c561-9ee6-74b1-05092da7ef5e Address:860aaa3c-c561-9ee6-74b1-05092da7ef5e}]" 2024-06-01T18:43:05.099Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:05.099Z [INFO] server-860aaa3c-c561-9ee6-74b1-05092da7ef5e: entering follower state: follower="Node at 860aaa3c-c561-9ee6-74b1-05092da7ef5e [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.099Z [INFO] server-f6716691-1789-c730-129d-a78f1d9a9c62: entering follower state: follower="Node at f6716691-1789-c730-129d-a78f1d9a9c62 [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.151Z [WARN] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:05.151Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: entering candidate state: node="Node at 45755dc7-b3f0-fba1-c2bf-ae8d606c233e [Candidate]" term=2 2024-06-01T18:43:05.151Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: election won: term=2 tally=2 2024-06-01T18:43:05.151Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: entering leader state: leader="Node at 45755dc7-b3f0-fba1-c2bf-ae8d606c233e [Leader]" 2024-06-01T18:43:05.152Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: added peer, starting replication: peer=server-f6716691-1789-c730-129d-a78f1d9a9c62 2024-06-01T18:43:05.152Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: added peer, starting replication: peer=server-860aaa3c-c561-9ee6-74b1-05092da7ef5e 2024-06-01T18:43:05.152Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: pipelining replication: peer="{Voter server-860aaa3c-c561-9ee6-74b1-05092da7ef5e 860aaa3c-c561-9ee6-74b1-05092da7ef5e}" 2024-06-01T18:43:05.152Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: pipelining replication: peer="{Voter server-f6716691-1789-c730-129d-a78f1d9a9c62 f6716691-1789-c730-129d-a78f1d9a9c62}" 2024-06-01T18:43:05.257Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:05.152021393 +0000 UTC m=+100.650210845 (2 nodes), 52.173512ms from start of poll, 52.618751ms from cluster start. Timeout at 2024-06-01 18:43:05.257452476 +0000 UTC m=+100.755641928, 105.431083ms after stability 2024-06-01T18:43:05.257Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:05.363Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:05.257549487 +0000 UTC m=+100.755738939 (1 nodes), 1.603µs from start of poll, 158.146845ms from cluster start. Timeout at 2024-06-01 18:43:05.36346916 +0000 UTC m=+100.861658612, 105.919673ms after stability 2024-06-01T18:43:05.363Z [WARN] server-f6716691-1789-c730-129d-a78f1d9a9c62: clearing log suffix: from=1 to=2 2024-06-01T18:43:05.363Z [WARN] server-f6716691-1789-c730-129d-a78f1d9a9c62: skipping application of old log: index=1 2024-06-01T18:43:05.363Z [WARN] server-f6716691-1789-c730-129d-a78f1d9a9c62: skipping application of old log: index=1 2024-06-01T18:43:05.363Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: aborting pipeline replication: peer="{Voter server-860aaa3c-c561-9ee6-74b1-05092da7ef5e 860aaa3c-c561-9ee6-74b1-05092da7ef5e}" 2024-06-01T18:43:05.363Z [INFO] server-45755dc7-b3f0-fba1-c2bf-ae8d606c233e: aborting pipeline replication: peer="{Voter server-f6716691-1789-c730-129d-a78f1d9a9c62 f6716691-1789-c730-129d-a78f1d9a9c62}" --- PASS: TestRaft_AppendEntry (0.27s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2024-06-01T18:43:05.372Z [INFO] server-715c5a06-317b-5918-ba2c-96193840ae5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-715c5a06-317b-5918-ba2c-96193840ae5c Address:715c5a06-317b-5918-ba2c-96193840ae5c} {Suffrage:Voter ID:server-bb59f71f-d269-62fb-de51-97def5bea6ee Address:bb59f71f-d269-62fb-de51-97def5bea6ee} {Suffrage:Voter ID:server-f41a9476-7cda-ee20-b812-987872c1869f Address:f41a9476-7cda-ee20-b812-987872c1869f}]" 2024-06-01T18:43:05.372Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-715c5a06-317b-5918-ba2c-96193840ae5c Address:715c5a06-317b-5918-ba2c-96193840ae5c} {Suffrage:Voter ID:server-bb59f71f-d269-62fb-de51-97def5bea6ee Address:bb59f71f-d269-62fb-de51-97def5bea6ee} {Suffrage:Voter ID:server-f41a9476-7cda-ee20-b812-987872c1869f Address:f41a9476-7cda-ee20-b812-987872c1869f}]" 2024-06-01T18:43:05.372Z [INFO] server-f41a9476-7cda-ee20-b812-987872c1869f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-715c5a06-317b-5918-ba2c-96193840ae5c Address:715c5a06-317b-5918-ba2c-96193840ae5c} {Suffrage:Voter ID:server-bb59f71f-d269-62fb-de51-97def5bea6ee Address:bb59f71f-d269-62fb-de51-97def5bea6ee} {Suffrage:Voter ID:server-f41a9476-7cda-ee20-b812-987872c1869f Address:f41a9476-7cda-ee20-b812-987872c1869f}]" 2024-06-01T18:43:05.372Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:05.372Z [INFO] server-715c5a06-317b-5918-ba2c-96193840ae5c: entering follower state: follower="Node at 715c5a06-317b-5918-ba2c-96193840ae5c [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.372Z [INFO] server-f41a9476-7cda-ee20-b812-987872c1869f: entering follower state: follower="Node at f41a9476-7cda-ee20-b812-987872c1869f [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.372Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: entering follower state: follower="Node at bb59f71f-d269-62fb-de51-97def5bea6ee [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.432Z [WARN] server-bb59f71f-d269-62fb-de51-97def5bea6ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:05.432Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: entering candidate state: node="Node at bb59f71f-d269-62fb-de51-97def5bea6ee [Candidate]" term=2 2024-06-01T18:43:05.432Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: election won: term=2 tally=2 2024-06-01T18:43:05.432Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: entering leader state: leader="Node at bb59f71f-d269-62fb-de51-97def5bea6ee [Leader]" 2024-06-01T18:43:05.432Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: added peer, starting replication: peer=server-715c5a06-317b-5918-ba2c-96193840ae5c 2024-06-01T18:43:05.432Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: added peer, starting replication: peer=server-f41a9476-7cda-ee20-b812-987872c1869f 2024-06-01T18:43:05.432Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: pipelining replication: peer="{Voter server-f41a9476-7cda-ee20-b812-987872c1869f f41a9476-7cda-ee20-b812-987872c1869f}" 2024-06-01T18:43:05.432Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: pipelining replication: peer="{Voter server-715c5a06-317b-5918-ba2c-96193840ae5c 715c5a06-317b-5918-ba2c-96193840ae5c}" 2024-06-01T18:43:05.537Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:05.43279422 +0000 UTC m=+100.930983682 (2 nodes), 59.826168ms from start of poll, 60.238956ms from cluster start. Timeout at 2024-06-01 18:43:05.53783691 +0000 UTC m=+101.036026362, 105.04268ms after stability 2024-06-01T18:43:05.537Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:05.643Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:05.537942817 +0000 UTC m=+101.036132269 (1 nodes), 2.554µs from start of poll, 165.387543ms from cluster start. Timeout at 2024-06-01 18:43:05.64347625 +0000 UTC m=+101.141665702, 105.533433ms after stability 2024-06-01T18:43:05.643Z [WARN] server-f41a9476-7cda-ee20-b812-987872c1869f: rejecting vote request since we have a leader: from=715c5a06-317b-5918-ba2c-96193840ae5c leader=bb59f71f-d269-62fb-de51-97def5bea6ee leader-id=server-bb59f71f-d269-62fb-de51-97def5bea6ee 2024-06-01T18:43:05.643Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: aborting pipeline replication: peer="{Voter server-715c5a06-317b-5918-ba2c-96193840ae5c 715c5a06-317b-5918-ba2c-96193840ae5c}" 2024-06-01T18:43:05.643Z [INFO] server-bb59f71f-d269-62fb-de51-97def5bea6ee: aborting pipeline replication: peer="{Voter server-f41a9476-7cda-ee20-b812-987872c1869f f41a9476-7cda-ee20-b812-987872c1869f}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.33s) === RUN TestRaft_ProtocolVersion_RejectRPC 2024-06-01T18:43:05.723Z [INFO] server-3eb48994-5ba4-6788-c313-6264ab90a995: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3eb48994-5ba4-6788-c313-6264ab90a995 Address:3eb48994-5ba4-6788-c313-6264ab90a995} {Suffrage:Voter ID:server-4e732530-f7e6-41e3-affd-3b22da21b9a1 Address:4e732530-f7e6-41e3-affd-3b22da21b9a1} {Suffrage:Voter ID:server-122a4829-79c1-571d-7072-a245c2d4891c Address:122a4829-79c1-571d-7072-a245c2d4891c}]" 2024-06-01T18:43:05.723Z [INFO] server-3eb48994-5ba4-6788-c313-6264ab90a995: entering follower state: follower="Node at 3eb48994-5ba4-6788-c313-6264ab90a995 [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.723Z [INFO] server-4e732530-f7e6-41e3-affd-3b22da21b9a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3eb48994-5ba4-6788-c313-6264ab90a995 Address:3eb48994-5ba4-6788-c313-6264ab90a995} {Suffrage:Voter ID:server-4e732530-f7e6-41e3-affd-3b22da21b9a1 Address:4e732530-f7e6-41e3-affd-3b22da21b9a1} {Suffrage:Voter ID:server-122a4829-79c1-571d-7072-a245c2d4891c Address:122a4829-79c1-571d-7072-a245c2d4891c}]" 2024-06-01T18:43:05.723Z [INFO] server-4e732530-f7e6-41e3-affd-3b22da21b9a1: entering follower state: follower="Node at 4e732530-f7e6-41e3-affd-3b22da21b9a1 [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.723Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3eb48994-5ba4-6788-c313-6264ab90a995 Address:3eb48994-5ba4-6788-c313-6264ab90a995} {Suffrage:Voter ID:server-4e732530-f7e6-41e3-affd-3b22da21b9a1 Address:4e732530-f7e6-41e3-affd-3b22da21b9a1} {Suffrage:Voter ID:server-122a4829-79c1-571d-7072-a245c2d4891c Address:122a4829-79c1-571d-7072-a245c2d4891c}]" 2024-06-01T18:43:05.723Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:05.723Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: entering follower state: follower="Node at 122a4829-79c1-571d-7072-a245c2d4891c [Follower]" leader-address= leader-id= 2024-06-01T18:43:05.777Z [WARN] server-122a4829-79c1-571d-7072-a245c2d4891c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:05.777Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: entering candidate state: node="Node at 122a4829-79c1-571d-7072-a245c2d4891c [Candidate]" term=2 2024-06-01T18:43:05.777Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: election won: term=2 tally=2 2024-06-01T18:43:05.777Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: entering leader state: leader="Node at 122a4829-79c1-571d-7072-a245c2d4891c [Leader]" 2024-06-01T18:43:05.777Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: added peer, starting replication: peer=server-3eb48994-5ba4-6788-c313-6264ab90a995 2024-06-01T18:43:05.777Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: added peer, starting replication: peer=server-4e732530-f7e6-41e3-affd-3b22da21b9a1 2024-06-01T18:43:05.777Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: pipelining replication: peer="{Voter server-4e732530-f7e6-41e3-affd-3b22da21b9a1 4e732530-f7e6-41e3-affd-3b22da21b9a1}" 2024-06-01T18:43:05.778Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: pipelining replication: peer="{Voter server-3eb48994-5ba4-6788-c313-6264ab90a995 3eb48994-5ba4-6788-c313-6264ab90a995}" 2024-06-01T18:43:05.884Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:05.778074249 +0000 UTC m=+101.276263701 (2 nodes), 54.465038ms from start of poll, 54.718179ms from cluster start. Timeout at 2024-06-01 18:43:05.883975357 +0000 UTC m=+101.382164809, 105.901108ms after stability 2024-06-01T18:43:05.884Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:05.989Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:05.884099668 +0000 UTC m=+101.382289120 (1 nodes), 1.102µs from start of poll, 160.743598ms from cluster start. Timeout at 2024-06-01 18:43:05.989543355 +0000 UTC m=+101.487732807, 105.443687ms after stability 2024-06-01T18:43:05.989Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: aborting pipeline replication: peer="{Voter server-4e732530-f7e6-41e3-affd-3b22da21b9a1 4e732530-f7e6-41e3-affd-3b22da21b9a1}" 2024-06-01T18:43:05.989Z [INFO] server-122a4829-79c1-571d-7072-a245c2d4891c: aborting pipeline replication: peer="{Voter server-3eb48994-5ba4-6788-c313-6264ab90a995 3eb48994-5ba4-6788-c313-6264ab90a995}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.34s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2024-06-01T18:43:06.048Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1de4821a-408d-5c47-7b0b-b62b10d57c7a Address:1de4821a-408d-5c47-7b0b-b62b10d57c7a} {Suffrage:Voter ID:server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb Address:cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb} {Suffrage:Voter ID:server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4 Address:c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4}]" 2024-06-01T18:43:06.048Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: entering follower state: follower="Node at 1de4821a-408d-5c47-7b0b-b62b10d57c7a [Follower]" leader-address= leader-id= 2024-06-01T18:43:06.048Z [INFO] server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1de4821a-408d-5c47-7b0b-b62b10d57c7a Address:1de4821a-408d-5c47-7b0b-b62b10d57c7a} {Suffrage:Voter ID:server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb Address:cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb} {Suffrage:Voter ID:server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4 Address:c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4}]" 2024-06-01T18:43:06.048Z [INFO] server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1de4821a-408d-5c47-7b0b-b62b10d57c7a Address:1de4821a-408d-5c47-7b0b-b62b10d57c7a} {Suffrage:Voter ID:server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb Address:cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb} {Suffrage:Voter ID:server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4 Address:c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4}]" 2024-06-01T18:43:06.048Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.048Z [INFO] server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb: entering follower state: follower="Node at cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb [Follower]" leader-address= leader-id= 2024-06-01T18:43:06.048Z [INFO] server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4: entering follower state: follower="Node at c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4 [Follower]" leader-address= leader-id= 2024-06-01T18:43:06.135Z [WARN] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:06.135Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: entering candidate state: node="Node at 1de4821a-408d-5c47-7b0b-b62b10d57c7a [Candidate]" term=2 2024-06-01T18:43:06.135Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: election won: term=2 tally=2 2024-06-01T18:43:06.135Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: entering leader state: leader="Node at 1de4821a-408d-5c47-7b0b-b62b10d57c7a [Leader]" 2024-06-01T18:43:06.135Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: added peer, starting replication: peer=server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb 2024-06-01T18:43:06.135Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: added peer, starting replication: peer=server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4 2024-06-01T18:43:06.135Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: pipelining replication: peer="{Voter server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4 c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4}" 2024-06-01T18:43:06.136Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: pipelining replication: peer="{Voter server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb}" 2024-06-01T18:43:06.241Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:06.135969019 +0000 UTC m=+101.634158471 (1 nodes), 87.182531ms from start of poll, 87.612021ms from cluster start. Timeout at 2024-06-01 18:43:06.24103895 +0000 UTC m=+101.739228402, 105.069931ms after stability 2024-06-01T18:43:06.241Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.346Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:06.241136242 +0000 UTC m=+101.739325694 (1 nodes), 1.393µs from start of poll, 192.779244ms from cluster start. Timeout at 2024-06-01 18:43:06.346290199 +0000 UTC m=+101.844479651, 105.153957ms after stability 2024-06-01T18:43:06.346Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.452Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:06.346451129 +0000 UTC m=+101.844640571 (1 nodes), 591ns from start of poll, 298.094121ms from cluster start. Timeout at 2024-06-01 18:43:06.452277677 +0000 UTC m=+101.950467129, 105.826558ms after stability 2024-06-01T18:43:06.452Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.557Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:06.452426775 +0000 UTC m=+101.950616217 (1 nodes), 511ns from start of poll, 404.069767ms from cluster start. Timeout at 2024-06-01 18:43:06.557876132 +0000 UTC m=+102.056065584, 105.449367ms after stability 2024-06-01T18:43:06.558Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.664Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:06.558080011 +0000 UTC m=+102.056269463 (1 nodes), 521ns from start of poll, 509.723013ms from cluster start. Timeout at 2024-06-01 18:43:06.664048525 +0000 UTC m=+102.162237977, 105.968514ms after stability 2024-06-01T18:43:06.664Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.770Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:06.664257224 +0000 UTC m=+102.162446676 (1 nodes), 691ns from start of poll, 615.900226ms from cluster start. Timeout at 2024-06-01 18:43:06.770260512 +0000 UTC m=+102.268449964, 106.003288ms after stability 2024-06-01T18:43:06.875Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.981Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:06.875958621 +0000 UTC m=+102.374148104 (1 nodes), 1.233µs from start of poll, 827.601654ms from cluster start. Timeout at 2024-06-01 18:43:06.980961888 +0000 UTC m=+102.479151340, 105.003236ms after stability 2024-06-01T18:43:06.981Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: aborting pipeline replication: peer="{Voter server-c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4 c5bcd8f3-66cf-a23b-8d02-c5cd75c3ccd4}" 2024-06-01T18:43:06.981Z [INFO] server-1de4821a-408d-5c47-7b0b-b62b10d57c7a: aborting pipeline replication: peer="{Voter server-cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb cc340ffb-e9bc-2c0f-5bdd-92ffeddcd5bb}" --- PASS: TestRaft_LeaderID_Propagated (0.93s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2024-06-01T18:43:06.982Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d88280a1-27db-4983-3d36-8058a3fac567 Address:d88280a1-27db-4983-3d36-8058a3fac567} {Suffrage:Voter ID:server-72703dbd-6803-f326-118e-a93594b7fc15 Address:72703dbd-6803-f326-118e-a93594b7fc15} {Suffrage:Voter ID:server-8cd9958b-1649-98f5-dc74-09702d99fffb Address:8cd9958b-1649-98f5-dc74-09702d99fffb}]" 2024-06-01T18:43:06.982Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d88280a1-27db-4983-3d36-8058a3fac567 Address:d88280a1-27db-4983-3d36-8058a3fac567} {Suffrage:Voter ID:server-72703dbd-6803-f326-118e-a93594b7fc15 Address:72703dbd-6803-f326-118e-a93594b7fc15} {Suffrage:Voter ID:server-8cd9958b-1649-98f5-dc74-09702d99fffb Address:8cd9958b-1649-98f5-dc74-09702d99fffb}]" 2024-06-01T18:43:06.982Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: entering follower state: follower="Node at d88280a1-27db-4983-3d36-8058a3fac567 [Follower]" leader-address= leader-id= 2024-06-01T18:43:06.982Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: entering follower state: follower="Node at 72703dbd-6803-f326-118e-a93594b7fc15 [Follower]" leader-address= leader-id= 2024-06-01T18:43:06.987Z [INFO] server-8cd9958b-1649-98f5-dc74-09702d99fffb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d88280a1-27db-4983-3d36-8058a3fac567 Address:d88280a1-27db-4983-3d36-8058a3fac567} {Suffrage:Voter ID:server-72703dbd-6803-f326-118e-a93594b7fc15 Address:72703dbd-6803-f326-118e-a93594b7fc15} {Suffrage:Voter ID:server-8cd9958b-1649-98f5-dc74-09702d99fffb Address:8cd9958b-1649-98f5-dc74-09702d99fffb}]" 2024-06-01T18:43:06.987Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:06.987Z [INFO] server-8cd9958b-1649-98f5-dc74-09702d99fffb: entering follower state: follower="Node at 8cd9958b-1649-98f5-dc74-09702d99fffb [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.033Z [WARN] server-d88280a1-27db-4983-3d36-8058a3fac567: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:07.033Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: entering candidate state: node="Node at d88280a1-27db-4983-3d36-8058a3fac567 [Candidate]" term=2 2024-06-01T18:43:07.033Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: election won: term=2 tally=2 2024-06-01T18:43:07.033Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: entering leader state: leader="Node at d88280a1-27db-4983-3d36-8058a3fac567 [Leader]" 2024-06-01T18:43:07.033Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: added peer, starting replication: peer=server-72703dbd-6803-f326-118e-a93594b7fc15 2024-06-01T18:43:07.033Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: added peer, starting replication: peer=server-8cd9958b-1649-98f5-dc74-09702d99fffb 2024-06-01T18:43:07.033Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: pipelining replication: peer="{Voter server-8cd9958b-1649-98f5-dc74-09702d99fffb 8cd9958b-1649-98f5-dc74-09702d99fffb}" 2024-06-01T18:43:07.033Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: pipelining replication: peer="{Voter server-72703dbd-6803-f326-118e-a93594b7fc15 72703dbd-6803-f326-118e-a93594b7fc15}" 2024-06-01T18:43:07.139Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:07.033957662 +0000 UTC m=+102.532147114 (1 nodes), 46.766157ms from start of poll, 51.954786ms from cluster start. Timeout at 2024-06-01 18:43:07.139632448 +0000 UTC m=+102.637821900, 105.674786ms after stability 2024-06-01T18:43:07.139Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:07.244Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:07.13971367 +0000 UTC m=+102.637903122 (1 nodes), 2.244µs from start of poll, 157.710794ms from cluster start. Timeout at 2024-06-01 18:43:07.2449287 +0000 UTC m=+102.743118153, 105.215031ms after stability 2024-06-01T18:43:07.245Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: entering candidate state: node="Node at 72703dbd-6803-f326-118e-a93594b7fc15 [Candidate]" term=3 2024-06-01T18:43:07.245Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: election won: term=3 tally=2 2024-06-01T18:43:07.245Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:07.245Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: entering follower state: follower="Node at d88280a1-27db-4983-3d36-8058a3fac567 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.245Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: aborting pipeline replication: peer="{Voter server-8cd9958b-1649-98f5-dc74-09702d99fffb 8cd9958b-1649-98f5-dc74-09702d99fffb}" 2024-06-01T18:43:07.245Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: entering leader state: leader="Node at 72703dbd-6803-f326-118e-a93594b7fc15 [Leader]" 2024-06-01T18:43:07.245Z [INFO] server-d88280a1-27db-4983-3d36-8058a3fac567: aborting pipeline replication: peer="{Voter server-72703dbd-6803-f326-118e-a93594b7fc15 72703dbd-6803-f326-118e-a93594b7fc15}" 2024-06-01T18:43:07.245Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: added peer, starting replication: peer=server-d88280a1-27db-4983-3d36-8058a3fac567 2024-06-01T18:43:07.245Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: added peer, starting replication: peer=server-8cd9958b-1649-98f5-dc74-09702d99fffb 2024-06-01T18:43:07.245Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: pipelining replication: peer="{Voter server-8cd9958b-1649-98f5-dc74-09702d99fffb 8cd9958b-1649-98f5-dc74-09702d99fffb}" 2024-06-01T18:43:07.245Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: pipelining replication: peer="{Voter server-d88280a1-27db-4983-3d36-8058a3fac567 d88280a1-27db-4983-3d36-8058a3fac567}" 2024-06-01T18:43:07.350Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:07.245132059 +0000 UTC m=+102.743321511 (1 nodes), 27.412µs from start of poll, 263.129183ms from cluster start. Timeout at 2024-06-01 18:43:07.350410037 +0000 UTC m=+102.848599489, 105.277978ms after stability 2024-06-01T18:43:07.350Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: aborting pipeline replication: peer="{Voter server-8cd9958b-1649-98f5-dc74-09702d99fffb 8cd9958b-1649-98f5-dc74-09702d99fffb}" 2024-06-01T18:43:07.350Z [INFO] server-72703dbd-6803-f326-118e-a93594b7fc15: aborting pipeline replication: peer="{Voter server-d88280a1-27db-4983-3d36-8058a3fac567 d88280a1-27db-4983-3d36-8058a3fac567}" --- PASS: TestRaft_LeadershipTransfer (0.37s) === RUN TestRaft_LeadershipTransferWithOneNode 2024-06-01T18:43:07.352Z [INFO] server-8738afc3-21bc-d627-677d-1f22c9cec168: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8738afc3-21bc-d627-677d-1f22c9cec168 Address:8738afc3-21bc-d627-677d-1f22c9cec168}]" 2024-06-01T18:43:07.352Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:07.352Z [INFO] server-8738afc3-21bc-d627-677d-1f22c9cec168: entering follower state: follower="Node at 8738afc3-21bc-d627-677d-1f22c9cec168 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.405Z [WARN] server-8738afc3-21bc-d627-677d-1f22c9cec168: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:07.405Z [INFO] server-8738afc3-21bc-d627-677d-1f22c9cec168: entering candidate state: node="Node at 8738afc3-21bc-d627-677d-1f22c9cec168 [Candidate]" term=2 2024-06-01T18:43:07.405Z [INFO] server-8738afc3-21bc-d627-677d-1f22c9cec168: election won: term=2 tally=1 2024-06-01T18:43:07.405Z [INFO] server-8738afc3-21bc-d627-677d-1f22c9cec168: entering leader state: leader="Node at 8738afc3-21bc-d627-677d-1f22c9cec168 [Leader]" 2024-06-01T18:43:07.511Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:07.405864236 +0000 UTC m=+102.904053698 (1 nodes), 53.125554ms from start of poll, 53.280142ms from cluster start. Timeout at 2024-06-01 18:43:07.511858768 +0000 UTC m=+103.010048230, 105.994532ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.16s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2024-06-01T18:43:07.513Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35974fb1-b0b6-7976-453b-81767fd2f415 Address:35974fb1-b0b6-7976-453b-81767fd2f415} {Suffrage:Voter ID:server-2bc14736-a5d8-72af-64c5-302fab436c6c Address:2bc14736-a5d8-72af-64c5-302fab436c6c} {Suffrage:Voter ID:server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb Address:2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb} {Suffrage:Voter ID:server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c Address:1912ffcb-e0bc-1739-3b9c-dd726bc49e9c} {Suffrage:Voter ID:server-3e6a9967-1a90-9575-6deb-8028f6549168 Address:3e6a9967-1a90-9575-6deb-8028f6549168} {Suffrage:Voter ID:server-55c520df-924d-da96-6b99-7190fae66a08 Address:55c520df-924d-da96-6b99-7190fae66a08} {Suffrage:Voter ID:server-576d1544-ab22-ff13-589a-665aab9bc87b Address:576d1544-ab22-ff13-589a-665aab9bc87b}]" 2024-06-01T18:43:07.513Z [INFO] server-2bc14736-a5d8-72af-64c5-302fab436c6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35974fb1-b0b6-7976-453b-81767fd2f415 Address:35974fb1-b0b6-7976-453b-81767fd2f415} {Suffrage:Voter ID:server-2bc14736-a5d8-72af-64c5-302fab436c6c Address:2bc14736-a5d8-72af-64c5-302fab436c6c} {Suffrage:Voter ID:server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb Address:2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb} {Suffrage:Voter ID:server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c Address:1912ffcb-e0bc-1739-3b9c-dd726bc49e9c} {Suffrage:Voter ID:server-3e6a9967-1a90-9575-6deb-8028f6549168 Address:3e6a9967-1a90-9575-6deb-8028f6549168} {Suffrage:Voter ID:server-55c520df-924d-da96-6b99-7190fae66a08 Address:55c520df-924d-da96-6b99-7190fae66a08} {Suffrage:Voter ID:server-576d1544-ab22-ff13-589a-665aab9bc87b Address:576d1544-ab22-ff13-589a-665aab9bc87b}]" 2024-06-01T18:43:07.513Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: entering follower state: follower="Node at 35974fb1-b0b6-7976-453b-81767fd2f415 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.513Z [INFO] server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35974fb1-b0b6-7976-453b-81767fd2f415 Address:35974fb1-b0b6-7976-453b-81767fd2f415} {Suffrage:Voter ID:server-2bc14736-a5d8-72af-64c5-302fab436c6c Address:2bc14736-a5d8-72af-64c5-302fab436c6c} {Suffrage:Voter ID:server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb Address:2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb} {Suffrage:Voter ID:server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c Address:1912ffcb-e0bc-1739-3b9c-dd726bc49e9c} {Suffrage:Voter ID:server-3e6a9967-1a90-9575-6deb-8028f6549168 Address:3e6a9967-1a90-9575-6deb-8028f6549168} {Suffrage:Voter ID:server-55c520df-924d-da96-6b99-7190fae66a08 Address:55c520df-924d-da96-6b99-7190fae66a08} {Suffrage:Voter ID:server-576d1544-ab22-ff13-589a-665aab9bc87b Address:576d1544-ab22-ff13-589a-665aab9bc87b}]" 2024-06-01T18:43:07.513Z [INFO] server-2bc14736-a5d8-72af-64c5-302fab436c6c: entering follower state: follower="Node at 2bc14736-a5d8-72af-64c5-302fab436c6c [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.513Z [INFO] server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb: entering follower state: follower="Node at 2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.514Z [INFO] server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35974fb1-b0b6-7976-453b-81767fd2f415 Address:35974fb1-b0b6-7976-453b-81767fd2f415} {Suffrage:Voter ID:server-2bc14736-a5d8-72af-64c5-302fab436c6c Address:2bc14736-a5d8-72af-64c5-302fab436c6c} {Suffrage:Voter ID:server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb Address:2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb} {Suffrage:Voter ID:server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c Address:1912ffcb-e0bc-1739-3b9c-dd726bc49e9c} {Suffrage:Voter ID:server-3e6a9967-1a90-9575-6deb-8028f6549168 Address:3e6a9967-1a90-9575-6deb-8028f6549168} {Suffrage:Voter ID:server-55c520df-924d-da96-6b99-7190fae66a08 Address:55c520df-924d-da96-6b99-7190fae66a08} {Suffrage:Voter ID:server-576d1544-ab22-ff13-589a-665aab9bc87b Address:576d1544-ab22-ff13-589a-665aab9bc87b}]" 2024-06-01T18:43:07.514Z [INFO] server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c: entering follower state: follower="Node at 1912ffcb-e0bc-1739-3b9c-dd726bc49e9c [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.514Z [INFO] server-3e6a9967-1a90-9575-6deb-8028f6549168: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35974fb1-b0b6-7976-453b-81767fd2f415 Address:35974fb1-b0b6-7976-453b-81767fd2f415} {Suffrage:Voter ID:server-2bc14736-a5d8-72af-64c5-302fab436c6c Address:2bc14736-a5d8-72af-64c5-302fab436c6c} {Suffrage:Voter ID:server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb Address:2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb} {Suffrage:Voter ID:server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c Address:1912ffcb-e0bc-1739-3b9c-dd726bc49e9c} {Suffrage:Voter ID:server-3e6a9967-1a90-9575-6deb-8028f6549168 Address:3e6a9967-1a90-9575-6deb-8028f6549168} {Suffrage:Voter ID:server-55c520df-924d-da96-6b99-7190fae66a08 Address:55c520df-924d-da96-6b99-7190fae66a08} {Suffrage:Voter ID:server-576d1544-ab22-ff13-589a-665aab9bc87b Address:576d1544-ab22-ff13-589a-665aab9bc87b}]" 2024-06-01T18:43:07.514Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35974fb1-b0b6-7976-453b-81767fd2f415 Address:35974fb1-b0b6-7976-453b-81767fd2f415} {Suffrage:Voter ID:server-2bc14736-a5d8-72af-64c5-302fab436c6c Address:2bc14736-a5d8-72af-64c5-302fab436c6c} {Suffrage:Voter ID:server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb Address:2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb} {Suffrage:Voter ID:server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c Address:1912ffcb-e0bc-1739-3b9c-dd726bc49e9c} {Suffrage:Voter ID:server-3e6a9967-1a90-9575-6deb-8028f6549168 Address:3e6a9967-1a90-9575-6deb-8028f6549168} {Suffrage:Voter ID:server-55c520df-924d-da96-6b99-7190fae66a08 Address:55c520df-924d-da96-6b99-7190fae66a08} {Suffrage:Voter ID:server-576d1544-ab22-ff13-589a-665aab9bc87b Address:576d1544-ab22-ff13-589a-665aab9bc87b}]" 2024-06-01T18:43:07.514Z [INFO] server-3e6a9967-1a90-9575-6deb-8028f6549168: entering follower state: follower="Node at 3e6a9967-1a90-9575-6deb-8028f6549168 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.514Z [INFO] server-576d1544-ab22-ff13-589a-665aab9bc87b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35974fb1-b0b6-7976-453b-81767fd2f415 Address:35974fb1-b0b6-7976-453b-81767fd2f415} {Suffrage:Voter ID:server-2bc14736-a5d8-72af-64c5-302fab436c6c Address:2bc14736-a5d8-72af-64c5-302fab436c6c} {Suffrage:Voter ID:server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb Address:2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb} {Suffrage:Voter ID:server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c Address:1912ffcb-e0bc-1739-3b9c-dd726bc49e9c} {Suffrage:Voter ID:server-3e6a9967-1a90-9575-6deb-8028f6549168 Address:3e6a9967-1a90-9575-6deb-8028f6549168} {Suffrage:Voter ID:server-55c520df-924d-da96-6b99-7190fae66a08 Address:55c520df-924d-da96-6b99-7190fae66a08} {Suffrage:Voter ID:server-576d1544-ab22-ff13-589a-665aab9bc87b Address:576d1544-ab22-ff13-589a-665aab9bc87b}]" 2024-06-01T18:43:07.514Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:07.514Z [INFO] server-576d1544-ab22-ff13-589a-665aab9bc87b: entering follower state: follower="Node at 576d1544-ab22-ff13-589a-665aab9bc87b [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.514Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: entering follower state: follower="Node at 55c520df-924d-da96-6b99-7190fae66a08 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.567Z [WARN] server-55c520df-924d-da96-6b99-7190fae66a08: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:07.567Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: entering candidate state: node="Node at 55c520df-924d-da96-6b99-7190fae66a08 [Candidate]" term=2 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: election won: term=2 tally=4 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: entering leader state: leader="Node at 55c520df-924d-da96-6b99-7190fae66a08 [Leader]" 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: added peer, starting replication: peer=server-35974fb1-b0b6-7976-453b-81767fd2f415 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: added peer, starting replication: peer=server-2bc14736-a5d8-72af-64c5-302fab436c6c 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: added peer, starting replication: peer=server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: added peer, starting replication: peer=server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: added peer, starting replication: peer=server-3e6a9967-1a90-9575-6deb-8028f6549168 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: added peer, starting replication: peer=server-576d1544-ab22-ff13-589a-665aab9bc87b 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: pipelining replication: peer="{Voter server-576d1544-ab22-ff13-589a-665aab9bc87b 576d1544-ab22-ff13-589a-665aab9bc87b}" 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: pipelining replication: peer="{Voter server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb 2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb}" 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: pipelining replication: peer="{Voter server-35974fb1-b0b6-7976-453b-81767fd2f415 35974fb1-b0b6-7976-453b-81767fd2f415}" 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: pipelining replication: peer="{Voter server-2bc14736-a5d8-72af-64c5-302fab436c6c 2bc14736-a5d8-72af-64c5-302fab436c6c}" 2024-06-01T18:43:07.568Z [WARN] server-3e6a9967-1a90-9575-6deb-8028f6549168: rejecting vote request since our last term is greater: candidate=55c520df-924d-da96-6b99-7190fae66a08 last-term=2 last-candidate-term=1 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: pipelining replication: peer="{Voter server-3e6a9967-1a90-9575-6deb-8028f6549168 3e6a9967-1a90-9575-6deb-8028f6549168}" 2024-06-01T18:43:07.568Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: pipelining replication: peer="{Voter server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c 1912ffcb-e0bc-1739-3b9c-dd726bc49e9c}" 2024-06-01T18:43:07.674Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:07.568589704 +0000 UTC m=+103.066779156 (1 nodes), 53.91834ms from start of poll, 54.947145ms from cluster start. Timeout at 2024-06-01 18:43:07.674265703 +0000 UTC m=+103.172455155, 105.675999ms after stability 2024-06-01T18:43:07.674Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:07.780Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:07.674341214 +0000 UTC m=+103.172530656 (6 nodes), 1.433µs from start of poll, 160.698645ms from cluster start. Timeout at 2024-06-01 18:43:07.77998916 +0000 UTC m=+103.278178612, 105.647956ms after stability 2024-06-01T18:43:07.780Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:07.885Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:07.780109876 +0000 UTC m=+103.278299328 (1 nodes), 2.705µs from start of poll, 266.467317ms from cluster start. Timeout at 2024-06-01 18:43:07.885648668 +0000 UTC m=+103.383838120, 105.538792ms after stability 2024-06-01T18:43:07.885Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: entering candidate state: node="Node at 35974fb1-b0b6-7976-453b-81767fd2f415 [Candidate]" term=3 2024-06-01T18:43:07.885Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: election won: term=3 tally=4 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: entering leader state: leader="Node at 35974fb1-b0b6-7976-453b-81767fd2f415 [Leader]" 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: added peer, starting replication: peer=server-2bc14736-a5d8-72af-64c5-302fab436c6c 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: added peer, starting replication: peer=server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: added peer, starting replication: peer=server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c 2024-06-01T18:43:07.887Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: entering follower state: follower="Node at 55c520df-924d-da96-6b99-7190fae66a08 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: added peer, starting replication: peer=server-3e6a9967-1a90-9575-6deb-8028f6549168 2024-06-01T18:43:07.887Z [ERROR] server-55c520df-924d-da96-6b99-7190fae66a08: peer has newer term, stopping replication: peer="{Voter server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c 1912ffcb-e0bc-1739-3b9c-dd726bc49e9c}" 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: added peer, starting replication: peer=server-55c520df-924d-da96-6b99-7190fae66a08 2024-06-01T18:43:07.887Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: added peer, starting replication: peer=server-576d1544-ab22-ff13-589a-665aab9bc87b 2024-06-01T18:43:07.887Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: aborting pipeline replication: peer="{Voter server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb 2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb}" 2024-06-01T18:43:07.888Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: pipelining replication: peer="{Voter server-2bc14736-a5d8-72af-64c5-302fab436c6c 2bc14736-a5d8-72af-64c5-302fab436c6c}" 2024-06-01T18:43:07.888Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: pipelining replication: peer="{Voter server-576d1544-ab22-ff13-589a-665aab9bc87b 576d1544-ab22-ff13-589a-665aab9bc87b}" 2024-06-01T18:43:07.888Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: aborting pipeline replication: peer="{Voter server-2bc14736-a5d8-72af-64c5-302fab436c6c 2bc14736-a5d8-72af-64c5-302fab436c6c}" 2024-06-01T18:43:07.888Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: aborting pipeline replication: peer="{Voter server-576d1544-ab22-ff13-589a-665aab9bc87b 576d1544-ab22-ff13-589a-665aab9bc87b}" 2024-06-01T18:43:07.888Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: aborting pipeline replication: peer="{Voter server-35974fb1-b0b6-7976-453b-81767fd2f415 35974fb1-b0b6-7976-453b-81767fd2f415}" 2024-06-01T18:43:07.888Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: aborting pipeline replication: peer="{Voter server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c 1912ffcb-e0bc-1739-3b9c-dd726bc49e9c}" 2024-06-01T18:43:07.888Z [INFO] server-55c520df-924d-da96-6b99-7190fae66a08: aborting pipeline replication: peer="{Voter server-3e6a9967-1a90-9575-6deb-8028f6549168 3e6a9967-1a90-9575-6deb-8028f6549168}" 2024-06-01T18:43:07.888Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: pipelining replication: peer="{Voter server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c 1912ffcb-e0bc-1739-3b9c-dd726bc49e9c}" 2024-06-01T18:43:07.888Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: pipelining replication: peer="{Voter server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb 2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb}" 2024-06-01T18:43:07.888Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: pipelining replication: peer="{Voter server-3e6a9967-1a90-9575-6deb-8028f6549168 3e6a9967-1a90-9575-6deb-8028f6549168}" 2024-06-01T18:43:07.889Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: pipelining replication: peer="{Voter server-55c520df-924d-da96-6b99-7190fae66a08 55c520df-924d-da96-6b99-7190fae66a08}" 2024-06-01T18:43:07.993Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:07.888223812 +0000 UTC m=+103.386413264 (1 nodes), 1.410706ms from start of poll, 374.581253ms from cluster start. Timeout at 2024-06-01 18:43:07.993395573 +0000 UTC m=+103.491585045, 105.171781ms after stability 2024-06-01T18:43:07.993Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: aborting pipeline replication: peer="{Voter server-3e6a9967-1a90-9575-6deb-8028f6549168 3e6a9967-1a90-9575-6deb-8028f6549168}" 2024-06-01T18:43:07.993Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: aborting pipeline replication: peer="{Voter server-55c520df-924d-da96-6b99-7190fae66a08 55c520df-924d-da96-6b99-7190fae66a08}" 2024-06-01T18:43:07.993Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: aborting pipeline replication: peer="{Voter server-576d1544-ab22-ff13-589a-665aab9bc87b 576d1544-ab22-ff13-589a-665aab9bc87b}" 2024-06-01T18:43:07.993Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: aborting pipeline replication: peer="{Voter server-2bc14736-a5d8-72af-64c5-302fab436c6c 2bc14736-a5d8-72af-64c5-302fab436c6c}" 2024-06-01T18:43:07.993Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: aborting pipeline replication: peer="{Voter server-2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb 2854e1d6-fc91-3fa1-e0e1-d9ce37f0defb}" 2024-06-01T18:43:07.993Z [INFO] server-35974fb1-b0b6-7976-453b-81767fd2f415: aborting pipeline replication: peer="{Voter server-1912ffcb-e0bc-1739-3b9c-dd726bc49e9c 1912ffcb-e0bc-1739-3b9c-dd726bc49e9c}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) === RUN TestRaft_LeadershipTransferToInvalidID 2024-06-01T18:43:07.995Z [INFO] server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 Address:b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3} {Suffrage:Voter ID:server-11d39106-d92e-aae6-b784-6aea8b908962 Address:11d39106-d92e-aae6-b784-6aea8b908962} {Suffrage:Voter ID:server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e Address:d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e}]" 2024-06-01T18:43:07.995Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 Address:b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3} {Suffrage:Voter ID:server-11d39106-d92e-aae6-b784-6aea8b908962 Address:11d39106-d92e-aae6-b784-6aea8b908962} {Suffrage:Voter ID:server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e Address:d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e}]" 2024-06-01T18:43:07.995Z [INFO] server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 Address:b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3} {Suffrage:Voter ID:server-11d39106-d92e-aae6-b784-6aea8b908962 Address:11d39106-d92e-aae6-b784-6aea8b908962} {Suffrage:Voter ID:server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e Address:d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e}]" 2024-06-01T18:43:07.995Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:07.995Z [INFO] server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3: entering follower state: follower="Node at b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.995Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: entering follower state: follower="Node at 11d39106-d92e-aae6-b784-6aea8b908962 [Follower]" leader-address= leader-id= 2024-06-01T18:43:07.995Z [INFO] server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e: entering follower state: follower="Node at d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.059Z [WARN] server-11d39106-d92e-aae6-b784-6aea8b908962: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:08.059Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: entering candidate state: node="Node at 11d39106-d92e-aae6-b784-6aea8b908962 [Candidate]" term=2 2024-06-01T18:43:08.060Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: election won: term=2 tally=2 2024-06-01T18:43:08.060Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: entering leader state: leader="Node at 11d39106-d92e-aae6-b784-6aea8b908962 [Leader]" 2024-06-01T18:43:08.060Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: added peer, starting replication: peer=server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 2024-06-01T18:43:08.060Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: added peer, starting replication: peer=server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e 2024-06-01T18:43:08.061Z [WARN] server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:08.061Z [INFO] server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3: entering candidate state: node="Node at b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 [Candidate]" term=2 2024-06-01T18:43:08.061Z [INFO] server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3: duplicate requestVote for same term: term=2 2024-06-01T18:43:08.061Z [WARN] server-11d39106-d92e-aae6-b784-6aea8b908962: rejecting vote request since we have a leader: from=b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 leader=11d39106-d92e-aae6-b784-6aea8b908962 leader-id=server-11d39106-d92e-aae6-b784-6aea8b908962 2024-06-01T18:43:08.061Z [WARN] server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e: rejecting vote request since we have a leader: from=b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 leader=11d39106-d92e-aae6-b784-6aea8b908962 leader-id=server-11d39106-d92e-aae6-b784-6aea8b908962 2024-06-01T18:43:08.062Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: pipelining replication: peer="{Voter server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e}" 2024-06-01T18:43:08.062Z [INFO] server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3: entering follower state: follower="Node at b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 [Follower]" leader-address=11d39106-d92e-aae6-b784-6aea8b908962 leader-id=server-11d39106-d92e-aae6-b784-6aea8b908962 2024-06-01T18:43:08.062Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: pipelining replication: peer="{Voter server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3}" 2024-06-01T18:43:08.167Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:08.06263253 +0000 UTC m=+103.560821982 (1 nodes), 66.992739ms from start of poll, 67.347339ms from cluster start. Timeout at 2024-06-01 18:43:08.167737265 +0000 UTC m=+103.665926717, 105.104735ms after stability 2024-06-01T18:43:08.167Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: aborting pipeline replication: peer="{Voter server-d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e d5c4e3aa-2f00-2f7f-0d3a-01dce8e2651e}" 2024-06-01T18:43:08.168Z [INFO] server-11d39106-d92e-aae6-b784-6aea8b908962: aborting pipeline replication: peer="{Voter server-b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3 b6fe824e-8c94-09ab-50dd-ad4e4b95bfb3}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2024-06-01T18:43:08.169Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67 Address:9f9eb7da-ccd2-83c7-5376-4c7f45698e67} {Suffrage:Voter ID:server-dd72a0cd-fa01-0d03-221e-b530076f4c44 Address:dd72a0cd-fa01-0d03-221e-b530076f4c44} {Suffrage:Voter ID:server-0fe3b220-2eac-71ef-e3bc-931df3997115 Address:0fe3b220-2eac-71ef-e3bc-931df3997115}]" 2024-06-01T18:43:08.169Z [INFO] server-dd72a0cd-fa01-0d03-221e-b530076f4c44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67 Address:9f9eb7da-ccd2-83c7-5376-4c7f45698e67} {Suffrage:Voter ID:server-dd72a0cd-fa01-0d03-221e-b530076f4c44 Address:dd72a0cd-fa01-0d03-221e-b530076f4c44} {Suffrage:Voter ID:server-0fe3b220-2eac-71ef-e3bc-931df3997115 Address:0fe3b220-2eac-71ef-e3bc-931df3997115}]" 2024-06-01T18:43:08.169Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: entering follower state: follower="Node at 9f9eb7da-ccd2-83c7-5376-4c7f45698e67 [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.169Z [INFO] server-0fe3b220-2eac-71ef-e3bc-931df3997115: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67 Address:9f9eb7da-ccd2-83c7-5376-4c7f45698e67} {Suffrage:Voter ID:server-dd72a0cd-fa01-0d03-221e-b530076f4c44 Address:dd72a0cd-fa01-0d03-221e-b530076f4c44} {Suffrage:Voter ID:server-0fe3b220-2eac-71ef-e3bc-931df3997115 Address:0fe3b220-2eac-71ef-e3bc-931df3997115}]" 2024-06-01T18:43:08.169Z [INFO] server-dd72a0cd-fa01-0d03-221e-b530076f4c44: entering follower state: follower="Node at dd72a0cd-fa01-0d03-221e-b530076f4c44 [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.169Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:08.169Z [INFO] server-0fe3b220-2eac-71ef-e3bc-931df3997115: entering follower state: follower="Node at 0fe3b220-2eac-71ef-e3bc-931df3997115 [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.223Z [WARN] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:08.223Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: entering candidate state: node="Node at 9f9eb7da-ccd2-83c7-5376-4c7f45698e67 [Candidate]" term=2 2024-06-01T18:43:08.223Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: election won: term=2 tally=2 2024-06-01T18:43:08.223Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: entering leader state: leader="Node at 9f9eb7da-ccd2-83c7-5376-4c7f45698e67 [Leader]" 2024-06-01T18:43:08.223Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: added peer, starting replication: peer=server-dd72a0cd-fa01-0d03-221e-b530076f4c44 2024-06-01T18:43:08.223Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: added peer, starting replication: peer=server-0fe3b220-2eac-71ef-e3bc-931df3997115 2024-06-01T18:43:08.223Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: pipelining replication: peer="{Voter server-dd72a0cd-fa01-0d03-221e-b530076f4c44 dd72a0cd-fa01-0d03-221e-b530076f4c44}" 2024-06-01T18:43:08.223Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: pipelining replication: peer="{Voter server-0fe3b220-2eac-71ef-e3bc-931df3997115 0fe3b220-2eac-71ef-e3bc-931df3997115}" 2024-06-01T18:43:08.329Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:08.223246246 +0000 UTC m=+103.721435698 (2 nodes), 54.005412ms from start of poll, 54.294391ms from cluster start. Timeout at 2024-06-01 18:43:08.329108963 +0000 UTC m=+103.827298415, 105.862717ms after stability 2024-06-01T18:43:08.329Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:08.435Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:08.32919332 +0000 UTC m=+103.827382772 (1 nodes), 1.804µs from start of poll, 160.241465ms from cluster start. Timeout at 2024-06-01 18:43:08.435340686 +0000 UTC m=+103.933530138, 106.147366ms after stability 2024-06-01T18:43:08.439Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: aborting pipeline replication: peer="{Voter server-0fe3b220-2eac-71ef-e3bc-931df3997115 0fe3b220-2eac-71ef-e3bc-931df3997115}" 2024-06-01T18:43:08.439Z [INFO] server-9f9eb7da-ccd2-83c7-5376-4c7f45698e67: aborting pipeline replication: peer="{Voter server-dd72a0cd-fa01-0d03-221e-b530076f4c44 dd72a0cd-fa01-0d03-221e-b530076f4c44}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer 2024-06-01T18:43:08.441Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa09eda3-be2f-e629-389d-1430b7d546d3 Address:aa09eda3-be2f-e629-389d-1430b7d546d3} {Suffrage:Voter ID:server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 Address:2ccbe915-3b57-7ae9-0402-4ee287bb79c6} {Suffrage:Voter ID:server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c Address:942ff52f-e7f3-6130-4a23-baa9ac4ee44c}]" 2024-06-01T18:43:08.441Z [INFO] server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa09eda3-be2f-e629-389d-1430b7d546d3 Address:aa09eda3-be2f-e629-389d-1430b7d546d3} {Suffrage:Voter ID:server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 Address:2ccbe915-3b57-7ae9-0402-4ee287bb79c6} {Suffrage:Voter ID:server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c Address:942ff52f-e7f3-6130-4a23-baa9ac4ee44c}]" 2024-06-01T18:43:08.441Z [INFO] server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6: entering follower state: follower="Node at 2ccbe915-3b57-7ae9-0402-4ee287bb79c6 [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.441Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa09eda3-be2f-e629-389d-1430b7d546d3 Address:aa09eda3-be2f-e629-389d-1430b7d546d3} {Suffrage:Voter ID:server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 Address:2ccbe915-3b57-7ae9-0402-4ee287bb79c6} {Suffrage:Voter ID:server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c Address:942ff52f-e7f3-6130-4a23-baa9ac4ee44c}]" 2024-06-01T18:43:08.441Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: entering follower state: follower="Node at aa09eda3-be2f-e629-389d-1430b7d546d3 [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.441Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: entering follower state: follower="Node at 942ff52f-e7f3-6130-4a23-baa9ac4ee44c [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.441Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:08.501Z [WARN] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:08.501Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: entering candidate state: node="Node at 942ff52f-e7f3-6130-4a23-baa9ac4ee44c [Candidate]" term=2 2024-06-01T18:43:08.501Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: election won: term=2 tally=2 2024-06-01T18:43:08.501Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: entering leader state: leader="Node at 942ff52f-e7f3-6130-4a23-baa9ac4ee44c [Leader]" 2024-06-01T18:43:08.501Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: added peer, starting replication: peer=server-aa09eda3-be2f-e629-389d-1430b7d546d3 2024-06-01T18:43:08.501Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: added peer, starting replication: peer=server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 2024-06-01T18:43:08.502Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: pipelining replication: peer="{Voter server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 2ccbe915-3b57-7ae9-0402-4ee287bb79c6}" 2024-06-01T18:43:08.502Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: pipelining replication: peer="{Voter server-aa09eda3-be2f-e629-389d-1430b7d546d3 aa09eda3-be2f-e629-389d-1430b7d546d3}" 2024-06-01T18:43:08.607Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:08.501980358 +0000 UTC m=+104.000169810 (1 nodes), 60.639763ms from start of poll, 60.961011ms from cluster start. Timeout at 2024-06-01 18:43:08.607442829 +0000 UTC m=+104.105632272, 105.462462ms after stability 2024-06-01T18:43:08.607Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:08.712Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:08.607510896 +0000 UTC m=+104.105700348 (2 nodes), 2.214µs from start of poll, 166.491549ms from cluster start. Timeout at 2024-06-01 18:43:08.712598679 +0000 UTC m=+104.210788131, 105.087783ms after stability 2024-06-01T18:43:08.756Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: entering candidate state: node="Node at aa09eda3-be2f-e629-389d-1430b7d546d3 [Candidate]" term=3 2024-06-01T18:43:08.756Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: entering follower state: follower="Node at 942ff52f-e7f3-6130-4a23-baa9ac4ee44c [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.756Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: aborting pipeline replication: peer="{Voter server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 2ccbe915-3b57-7ae9-0402-4ee287bb79c6}" 2024-06-01T18:43:08.756Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:08.757Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: election won: term=3 tally=2 2024-06-01T18:43:08.757Z [INFO] server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c: aborting pipeline replication: peer="{Voter server-aa09eda3-be2f-e629-389d-1430b7d546d3 aa09eda3-be2f-e629-389d-1430b7d546d3}" 2024-06-01T18:43:08.757Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: entering leader state: leader="Node at aa09eda3-be2f-e629-389d-1430b7d546d3 [Leader]" 2024-06-01T18:43:08.757Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: added peer, starting replication: peer=server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 2024-06-01T18:43:08.757Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: added peer, starting replication: peer=server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c 2024-06-01T18:43:08.757Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: pipelining replication: peer="{Voter server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c 942ff52f-e7f3-6130-4a23-baa9ac4ee44c}" 2024-06-01T18:43:08.757Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: pipelining replication: peer="{Voter server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 2ccbe915-3b57-7ae9-0402-4ee287bb79c6}" 2024-06-01T18:43:08.862Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:08.757139263 +0000 UTC m=+104.255328715 (1 nodes), 136.233µs from start of poll, 316.119916ms from cluster start. Timeout at 2024-06-01 18:43:08.862559926 +0000 UTC m=+104.360749378, 105.420663ms after stability 2024-06-01T18:43:08.862Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: aborting pipeline replication: peer="{Voter server-2ccbe915-3b57-7ae9-0402-4ee287bb79c6 2ccbe915-3b57-7ae9-0402-4ee287bb79c6}" 2024-06-01T18:43:08.862Z [INFO] server-aa09eda3-be2f-e629-389d-1430b7d546d3: aborting pipeline replication: peer="{Voter server-942ff52f-e7f3-6130-4a23-baa9ac4ee44c 942ff52f-e7f3-6130-4a23-baa9ac4ee44c}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) === RUN TestRaft_LeadershipTransferToItself 2024-06-01T18:43:08.863Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a0a178c-90ae-3a25-58f3-51b980cbe864 Address:3a0a178c-90ae-3a25-58f3-51b980cbe864} {Suffrage:Voter ID:server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2 Address:e16662b5-0401-4e15-4ddc-dbd536ea8cf2} {Suffrage:Voter ID:server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc Address:d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc}]" 2024-06-01T18:43:08.863Z [INFO] server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a0a178c-90ae-3a25-58f3-51b980cbe864 Address:3a0a178c-90ae-3a25-58f3-51b980cbe864} {Suffrage:Voter ID:server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2 Address:e16662b5-0401-4e15-4ddc-dbd536ea8cf2} {Suffrage:Voter ID:server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc Address:d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc}]" 2024-06-01T18:43:08.863Z [INFO] server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a0a178c-90ae-3a25-58f3-51b980cbe864 Address:3a0a178c-90ae-3a25-58f3-51b980cbe864} {Suffrage:Voter ID:server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2 Address:e16662b5-0401-4e15-4ddc-dbd536ea8cf2} {Suffrage:Voter ID:server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc Address:d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc}]" 2024-06-01T18:43:08.863Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:08.863Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: entering follower state: follower="Node at 3a0a178c-90ae-3a25-58f3-51b980cbe864 [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.863Z [INFO] server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2: entering follower state: follower="Node at e16662b5-0401-4e15-4ddc-dbd536ea8cf2 [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.864Z [INFO] server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc: entering follower state: follower="Node at d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc [Follower]" leader-address= leader-id= 2024-06-01T18:43:08.920Z [WARN] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:08.921Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: entering candidate state: node="Node at 3a0a178c-90ae-3a25-58f3-51b980cbe864 [Candidate]" term=2 2024-06-01T18:43:08.921Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: election won: term=2 tally=2 2024-06-01T18:43:08.921Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: entering leader state: leader="Node at 3a0a178c-90ae-3a25-58f3-51b980cbe864 [Leader]" 2024-06-01T18:43:08.921Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: added peer, starting replication: peer=server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2 2024-06-01T18:43:08.921Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: added peer, starting replication: peer=server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc 2024-06-01T18:43:08.921Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: pipelining replication: peer="{Voter server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc}" 2024-06-01T18:43:08.921Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: pipelining replication: peer="{Voter server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2 e16662b5-0401-4e15-4ddc-dbd536ea8cf2}" 2024-06-01T18:43:09.027Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:08.921387265 +0000 UTC m=+104.419576717 (1 nodes), 57.484779ms from start of poll, 57.750313ms from cluster start. Timeout at 2024-06-01 18:43:09.027338236 +0000 UTC m=+104.525527688, 105.950971ms after stability 2024-06-01T18:43:09.027Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: cannot transfer leadership to itself 2024-06-01T18:43:09.027Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: aborting pipeline replication: peer="{Voter server-d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc d1fb9cfe-ab6f-1c50-d521-1b91fb770cdc}" 2024-06-01T18:43:09.027Z [INFO] server-3a0a178c-90ae-3a25-58f3-51b980cbe864: aborting pipeline replication: peer="{Voter server-e16662b5-0401-4e15-4ddc-dbd536ea8cf2 e16662b5-0401-4e15-4ddc-dbd536ea8cf2}" --- PASS: TestRaft_LeadershipTransferToItself (0.16s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2024-06-01T18:43:09.031Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46e272ff-a851-377d-ea5b-650da4a16b8e Address:46e272ff-a851-377d-ea5b-650da4a16b8e} {Suffrage:Voter ID:server-0af76920-1526-9c71-ea64-a36fff56feb9 Address:0af76920-1526-9c71-ea64-a36fff56feb9} {Suffrage:Voter ID:server-48d9cf17-304e-7814-bb68-3f0aa37f3123 Address:48d9cf17-304e-7814-bb68-3f0aa37f3123}]" 2024-06-01T18:43:09.031Z [INFO] server-0af76920-1526-9c71-ea64-a36fff56feb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46e272ff-a851-377d-ea5b-650da4a16b8e Address:46e272ff-a851-377d-ea5b-650da4a16b8e} {Suffrage:Voter ID:server-0af76920-1526-9c71-ea64-a36fff56feb9 Address:0af76920-1526-9c71-ea64-a36fff56feb9} {Suffrage:Voter ID:server-48d9cf17-304e-7814-bb68-3f0aa37f3123 Address:48d9cf17-304e-7814-bb68-3f0aa37f3123}]" 2024-06-01T18:43:09.031Z [INFO] server-48d9cf17-304e-7814-bb68-3f0aa37f3123: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46e272ff-a851-377d-ea5b-650da4a16b8e Address:46e272ff-a851-377d-ea5b-650da4a16b8e} {Suffrage:Voter ID:server-0af76920-1526-9c71-ea64-a36fff56feb9 Address:0af76920-1526-9c71-ea64-a36fff56feb9} {Suffrage:Voter ID:server-48d9cf17-304e-7814-bb68-3f0aa37f3123 Address:48d9cf17-304e-7814-bb68-3f0aa37f3123}]" 2024-06-01T18:43:09.031Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:09.031Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: entering follower state: follower="Node at 46e272ff-a851-377d-ea5b-650da4a16b8e [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.031Z [INFO] server-48d9cf17-304e-7814-bb68-3f0aa37f3123: entering follower state: follower="Node at 48d9cf17-304e-7814-bb68-3f0aa37f3123 [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.031Z [INFO] server-0af76920-1526-9c71-ea64-a36fff56feb9: entering follower state: follower="Node at 0af76920-1526-9c71-ea64-a36fff56feb9 [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.090Z [WARN] server-46e272ff-a851-377d-ea5b-650da4a16b8e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:09.090Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: entering candidate state: node="Node at 46e272ff-a851-377d-ea5b-650da4a16b8e [Candidate]" term=2 2024-06-01T18:43:09.090Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: election won: term=2 tally=2 2024-06-01T18:43:09.090Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: entering leader state: leader="Node at 46e272ff-a851-377d-ea5b-650da4a16b8e [Leader]" 2024-06-01T18:43:09.090Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: added peer, starting replication: peer=server-0af76920-1526-9c71-ea64-a36fff56feb9 2024-06-01T18:43:09.090Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: added peer, starting replication: peer=server-48d9cf17-304e-7814-bb68-3f0aa37f3123 2024-06-01T18:43:09.090Z [WARN] server-0af76920-1526-9c71-ea64-a36fff56feb9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:09.090Z [INFO] server-0af76920-1526-9c71-ea64-a36fff56feb9: entering candidate state: node="Node at 0af76920-1526-9c71-ea64-a36fff56feb9 [Candidate]" term=2 2024-06-01T18:43:09.090Z [WARN] server-46e272ff-a851-377d-ea5b-650da4a16b8e: rejecting vote request since we have a leader: from=0af76920-1526-9c71-ea64-a36fff56feb9 leader=46e272ff-a851-377d-ea5b-650da4a16b8e leader-id=server-46e272ff-a851-377d-ea5b-650da4a16b8e 2024-06-01T18:43:09.090Z [INFO] server-0af76920-1526-9c71-ea64-a36fff56feb9: entering follower state: follower="Node at 0af76920-1526-9c71-ea64-a36fff56feb9 [Follower]" leader-address=46e272ff-a851-377d-ea5b-650da4a16b8e leader-id=server-46e272ff-a851-377d-ea5b-650da4a16b8e 2024-06-01T18:43:09.090Z [INFO] server-0af76920-1526-9c71-ea64-a36fff56feb9: duplicate requestVote for same term: term=2 2024-06-01T18:43:09.090Z [WARN] server-48d9cf17-304e-7814-bb68-3f0aa37f3123: rejecting vote request since we have a leader: from=0af76920-1526-9c71-ea64-a36fff56feb9 leader=46e272ff-a851-377d-ea5b-650da4a16b8e leader-id=server-46e272ff-a851-377d-ea5b-650da4a16b8e 2024-06-01T18:43:09.090Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: pipelining replication: peer="{Voter server-48d9cf17-304e-7814-bb68-3f0aa37f3123 48d9cf17-304e-7814-bb68-3f0aa37f3123}" 2024-06-01T18:43:09.090Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: pipelining replication: peer="{Voter server-0af76920-1526-9c71-ea64-a36fff56feb9 0af76920-1526-9c71-ea64-a36fff56feb9}" 2024-06-01T18:43:09.195Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:09.090424683 +0000 UTC m=+104.588614136 (1 nodes), 58.894564ms from start of poll, 59.235199ms from cluster start. Timeout at 2024-06-01 18:43:09.195875914 +0000 UTC m=+104.694065376, 105.45124ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2024-06-01T18:43:09.196Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: aborting pipeline replication: peer="{Voter server-48d9cf17-304e-7814-bb68-3f0aa37f3123 48d9cf17-304e-7814-bb68-3f0aa37f3123}" 2024-06-01T18:43:09.196Z [INFO] server-46e272ff-a851-377d-ea5b-650da4a16b8e: aborting pipeline replication: peer="{Voter server-0af76920-1526-9c71-ea64-a36fff56feb9 0af76920-1526-9c71-ea64-a36fff56feb9}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2024-06-01T18:43:09.197Z [INFO] server-5f2388e1-5b79-6409-7386-524098fc615e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f2388e1-5b79-6409-7386-524098fc615e Address:5f2388e1-5b79-6409-7386-524098fc615e} {Suffrage:Voter ID:server-644c72b3-54e8-3e4d-7165-6314237c1729 Address:644c72b3-54e8-3e4d-7165-6314237c1729} {Suffrage:Voter ID:server-191a08fd-de0f-0840-a729-7096ccfe11a3 Address:191a08fd-de0f-0840-a729-7096ccfe11a3}]" 2024-06-01T18:43:09.197Z [INFO] server-644c72b3-54e8-3e4d-7165-6314237c1729: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f2388e1-5b79-6409-7386-524098fc615e Address:5f2388e1-5b79-6409-7386-524098fc615e} {Suffrage:Voter ID:server-644c72b3-54e8-3e4d-7165-6314237c1729 Address:644c72b3-54e8-3e4d-7165-6314237c1729} {Suffrage:Voter ID:server-191a08fd-de0f-0840-a729-7096ccfe11a3 Address:191a08fd-de0f-0840-a729-7096ccfe11a3}]" 2024-06-01T18:43:09.197Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f2388e1-5b79-6409-7386-524098fc615e Address:5f2388e1-5b79-6409-7386-524098fc615e} {Suffrage:Voter ID:server-644c72b3-54e8-3e4d-7165-6314237c1729 Address:644c72b3-54e8-3e4d-7165-6314237c1729} {Suffrage:Voter ID:server-191a08fd-de0f-0840-a729-7096ccfe11a3 Address:191a08fd-de0f-0840-a729-7096ccfe11a3}]" 2024-06-01T18:43:09.197Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:09.197Z [INFO] server-5f2388e1-5b79-6409-7386-524098fc615e: entering follower state: follower="Node at 5f2388e1-5b79-6409-7386-524098fc615e [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.197Z [INFO] server-644c72b3-54e8-3e4d-7165-6314237c1729: entering follower state: follower="Node at 644c72b3-54e8-3e4d-7165-6314237c1729 [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.197Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: entering follower state: follower="Node at 191a08fd-de0f-0840-a729-7096ccfe11a3 [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.249Z [WARN] server-191a08fd-de0f-0840-a729-7096ccfe11a3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:09.249Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: entering candidate state: node="Node at 191a08fd-de0f-0840-a729-7096ccfe11a3 [Candidate]" term=2 2024-06-01T18:43:09.249Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: election won: term=2 tally=2 2024-06-01T18:43:09.249Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: entering leader state: leader="Node at 191a08fd-de0f-0840-a729-7096ccfe11a3 [Leader]" 2024-06-01T18:43:09.249Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: added peer, starting replication: peer=server-5f2388e1-5b79-6409-7386-524098fc615e 2024-06-01T18:43:09.249Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: added peer, starting replication: peer=server-644c72b3-54e8-3e4d-7165-6314237c1729 2024-06-01T18:43:09.249Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: pipelining replication: peer="{Voter server-644c72b3-54e8-3e4d-7165-6314237c1729 644c72b3-54e8-3e4d-7165-6314237c1729}" 2024-06-01T18:43:09.249Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: pipelining replication: peer="{Voter server-5f2388e1-5b79-6409-7386-524098fc615e 5f2388e1-5b79-6409-7386-524098fc615e}" 2024-06-01T18:43:09.354Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:09.249293733 +0000 UTC m=+104.747483185 (1 nodes), 51.75824ms from start of poll, 52.034364ms from cluster start. Timeout at 2024-06-01 18:43:09.354733973 +0000 UTC m=+104.852923425, 105.44024ms after stability 2024-06-01T18:43:09.354Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:09.460Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:09.35482926 +0000 UTC m=+104.853018712 (2 nodes), 1.443µs from start of poll, 157.569891ms from cluster start. Timeout at 2024-06-01 18:43:09.460077162 +0000 UTC m=+104.958266614, 105.247902ms after stability 2024-06-01T18:43:09.803Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: aborting pipeline replication: peer="{Voter server-644c72b3-54e8-3e4d-7165-6314237c1729 644c72b3-54e8-3e4d-7165-6314237c1729}" 2024-06-01T18:43:09.803Z [INFO] server-191a08fd-de0f-0840-a729-7096ccfe11a3: aborting pipeline replication: peer="{Voter server-5f2388e1-5b79-6409-7386-524098fc615e 5f2388e1-5b79-6409-7386-524098fc615e}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.61s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2024-06-01T18:43:09.807Z [INFO] server-508aa369-54c9-646d-fa10-b55ccfefaf63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-508aa369-54c9-646d-fa10-b55ccfefaf63 Address:508aa369-54c9-646d-fa10-b55ccfefaf63} {Suffrage:Voter ID:server-f0ebb2cc-7a02-9d53-b813-4eac682012e5 Address:f0ebb2cc-7a02-9d53-b813-4eac682012e5}]" 2024-06-01T18:43:09.807Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-508aa369-54c9-646d-fa10-b55ccfefaf63 Address:508aa369-54c9-646d-fa10-b55ccfefaf63} {Suffrage:Voter ID:server-f0ebb2cc-7a02-9d53-b813-4eac682012e5 Address:f0ebb2cc-7a02-9d53-b813-4eac682012e5}]" 2024-06-01T18:43:09.807Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:09.807Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: entering follower state: follower="Node at f0ebb2cc-7a02-9d53-b813-4eac682012e5 [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.807Z [INFO] server-508aa369-54c9-646d-fa10-b55ccfefaf63: entering follower state: follower="Node at 508aa369-54c9-646d-fa10-b55ccfefaf63 [Follower]" leader-address= leader-id= 2024-06-01T18:43:09.892Z [WARN] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:09.892Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: entering candidate state: node="Node at f0ebb2cc-7a02-9d53-b813-4eac682012e5 [Candidate]" term=2 2024-06-01T18:43:09.892Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: election won: term=2 tally=2 2024-06-01T18:43:09.892Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: entering leader state: leader="Node at f0ebb2cc-7a02-9d53-b813-4eac682012e5 [Leader]" 2024-06-01T18:43:09.892Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: added peer, starting replication: peer=server-508aa369-54c9-646d-fa10-b55ccfefaf63 2024-06-01T18:43:09.893Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: pipelining replication: peer="{Voter server-508aa369-54c9-646d-fa10-b55ccfefaf63 508aa369-54c9-646d-fa10-b55ccfefaf63}" 2024-06-01T18:43:09.998Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:09.893110696 +0000 UTC m=+105.391300148 (1 nodes), 85.511853ms from start of poll, 85.843169ms from cluster start. Timeout at 2024-06-01 18:43:09.998629492 +0000 UTC m=+105.496818944, 105.518796ms after stability 2024-06-01T18:43:09.998Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:10.104Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:09.998711645 +0000 UTC m=+105.496901097 (1 nodes), 781ns from start of poll, 191.444118ms from cluster start. Timeout at 2024-06-01 18:43:10.104730562 +0000 UTC m=+105.602920014, 106.018917ms after stability 2024-06-01T18:43:10.104Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: updating configuration: command=DemoteVoter server-id=server-508aa369-54c9-646d-fa10-b55ccfefaf63 server-addr= servers="[{Suffrage:Nonvoter ID:server-508aa369-54c9-646d-fa10-b55ccfefaf63 Address:508aa369-54c9-646d-fa10-b55ccfefaf63} {Suffrage:Voter ID:server-f0ebb2cc-7a02-9d53-b813-4eac682012e5 Address:f0ebb2cc-7a02-9d53-b813-4eac682012e5}]" 2024-06-01T18:43:10.104Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:10.210Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:10.104946524 +0000 UTC m=+105.603135976 (1 nodes), 581ns from start of poll, 297.678997ms from cluster start. Timeout at 2024-06-01 18:43:10.21018531 +0000 UTC m=+105.708374762, 105.238786ms after stability 2024-06-01T18:43:10.210Z [INFO] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: aborting pipeline replication: peer="{Voter server-508aa369-54c9-646d-fa10-b55ccfefaf63 508aa369-54c9-646d-fa10-b55ccfefaf63}" 2024-06-01T18:43:10.710Z [ERROR] server-f0ebb2cc-7a02-9d53-b813-4eac682012e5: failed to heartbeat to: peer=508aa369-54c9-646d-fa10-b55ccfefaf63 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.91s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2024-06-01T18:43:10.714Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9 Address:3f2924e5-92fb-1735-fbaf-14be9dfb85b9} {Suffrage:Voter ID:server-398f58bc-221a-8fee-14ad-48b62cfc1432 Address:398f58bc-221a-8fee-14ad-48b62cfc1432}]" 2024-06-01T18:43:10.714Z [INFO] server-398f58bc-221a-8fee-14ad-48b62cfc1432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9 Address:3f2924e5-92fb-1735-fbaf-14be9dfb85b9} {Suffrage:Voter ID:server-398f58bc-221a-8fee-14ad-48b62cfc1432 Address:398f58bc-221a-8fee-14ad-48b62cfc1432}]" 2024-06-01T18:43:10.714Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:10.714Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: entering follower state: follower="Node at 3f2924e5-92fb-1735-fbaf-14be9dfb85b9 [Follower]" leader-address= leader-id= 2024-06-01T18:43:10.714Z [INFO] server-398f58bc-221a-8fee-14ad-48b62cfc1432: entering follower state: follower="Node at 398f58bc-221a-8fee-14ad-48b62cfc1432 [Follower]" leader-address= leader-id= 2024-06-01T18:43:10.768Z [WARN] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:10.768Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: entering candidate state: node="Node at 3f2924e5-92fb-1735-fbaf-14be9dfb85b9 [Candidate]" term=2 2024-06-01T18:43:10.768Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: election won: term=2 tally=2 2024-06-01T18:43:10.768Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: entering leader state: leader="Node at 3f2924e5-92fb-1735-fbaf-14be9dfb85b9 [Leader]" 2024-06-01T18:43:10.768Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: added peer, starting replication: peer=server-398f58bc-221a-8fee-14ad-48b62cfc1432 2024-06-01T18:43:10.768Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: pipelining replication: peer="{Voter server-398f58bc-221a-8fee-14ad-48b62cfc1432 398f58bc-221a-8fee-14ad-48b62cfc1432}" 2024-06-01T18:43:10.874Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:10.768863384 +0000 UTC m=+106.267052836 (1 nodes), 54.008118ms from start of poll, 54.236493ms from cluster start. Timeout at 2024-06-01 18:43:10.874288546 +0000 UTC m=+106.372477988, 105.425152ms after stability 2024-06-01T18:43:10.874Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:10.982Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:10.874424378 +0000 UTC m=+106.372613820 (1 nodes), 2.984µs from start of poll, 159.797477ms from cluster start. Timeout at 2024-06-01 18:43:10.982424474 +0000 UTC m=+106.480613926, 108.000106ms after stability 2024-06-01T18:43:10.993Z [INFO] server-398f58bc-221a-8fee-14ad-48b62cfc1432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9 Address:3f2924e5-92fb-1735-fbaf-14be9dfb85b9} {Suffrage:Voter ID:server-398f58bc-221a-8fee-14ad-48b62cfc1432 Address:398f58bc-221a-8fee-14ad-48b62cfc1432}]" 2024-06-01T18:43:10.994Z [INFO] server-3f2924e5-92fb-1735-fbaf-14be9dfb85b9: aborting pipeline replication: peer="{Voter server-398f58bc-221a-8fee-14ad-48b62cfc1432 398f58bc-221a-8fee-14ad-48b62cfc1432}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) === RUN TestRaft_LogStoreIsMonotonic 2024-06-01T18:43:10.994Z [INFO] server-3c7c0015-18fc-0799-118e-d521f6f4e3cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c7c0015-18fc-0799-118e-d521f6f4e3cb Address:3c7c0015-18fc-0799-118e-d521f6f4e3cb}]" 2024-06-01T18:43:10.994Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:10.994Z [INFO] server-3c7c0015-18fc-0799-118e-d521f6f4e3cb: entering follower state: follower="Node at 3c7c0015-18fc-0799-118e-d521f6f4e3cb [Follower]" leader-address= leader-id= 2024-06-01T18:43:11.047Z [WARN] server-3c7c0015-18fc-0799-118e-d521f6f4e3cb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:11.047Z [INFO] server-3c7c0015-18fc-0799-118e-d521f6f4e3cb: entering candidate state: node="Node at 3c7c0015-18fc-0799-118e-d521f6f4e3cb [Candidate]" term=2 2024-06-01T18:43:11.047Z [INFO] server-3c7c0015-18fc-0799-118e-d521f6f4e3cb: election won: term=2 tally=1 2024-06-01T18:43:11.047Z [INFO] server-3c7c0015-18fc-0799-118e-d521f6f4e3cb: entering leader state: leader="Node at 3c7c0015-18fc-0799-118e-d521f6f4e3cb [Leader]" 2024-06-01T18:43:11.152Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:11.047509141 +0000 UTC m=+106.545698593 (1 nodes), 52.853247ms from start of poll, 52.969635ms from cluster start. Timeout at 2024-06-01 18:43:11.152797117 +0000 UTC m=+106.650986569, 105.287976ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.18s) === RUN TestRaft_CacheLogWithStoreError 2024-06-01T18:43:11.172Z [INFO] server-8c40286f-6748-6f35-3947-6dbc01a61cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c40286f-6748-6f35-3947-6dbc01a61cb2 Address:8c40286f-6748-6f35-3947-6dbc01a61cb2} {Suffrage:Voter ID:server-548f25fd-b1f7-6a5a-0019-15a4119cb73d Address:548f25fd-b1f7-6a5a-0019-15a4119cb73d}]" 2024-06-01T18:43:11.173Z [INFO] server-8c40286f-6748-6f35-3947-6dbc01a61cb2: entering follower state: follower="Node at 8c40286f-6748-6f35-3947-6dbc01a61cb2 [Follower]" leader-address= leader-id= 2024-06-01T18:43:11.173Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c40286f-6748-6f35-3947-6dbc01a61cb2 Address:8c40286f-6748-6f35-3947-6dbc01a61cb2} {Suffrage:Voter ID:server-548f25fd-b1f7-6a5a-0019-15a4119cb73d Address:548f25fd-b1f7-6a5a-0019-15a4119cb73d}]" 2024-06-01T18:43:11.173Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:11.173Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: entering follower state: follower="Node at 548f25fd-b1f7-6a5a-0019-15a4119cb73d [Follower]" leader-address= leader-id= 2024-06-01T18:43:11.236Z [WARN] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:11.236Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: entering candidate state: node="Node at 548f25fd-b1f7-6a5a-0019-15a4119cb73d [Candidate]" term=2 2024-06-01T18:43:11.236Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: election won: term=2 tally=2 2024-06-01T18:43:11.236Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: entering leader state: leader="Node at 548f25fd-b1f7-6a5a-0019-15a4119cb73d [Leader]" 2024-06-01T18:43:11.236Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: added peer, starting replication: peer=server-8c40286f-6748-6f35-3947-6dbc01a61cb2 2024-06-01T18:43:11.236Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: pipelining replication: peer="{Voter server-8c40286f-6748-6f35-3947-6dbc01a61cb2 8c40286f-6748-6f35-3947-6dbc01a61cb2}" 2024-06-01T18:43:11.342Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:11.236859793 +0000 UTC m=+106.735049245 (1 nodes), 63.169573ms from start of poll, 64.124951ms from cluster start. Timeout at 2024-06-01 18:43:11.34233611 +0000 UTC m=+106.840525562, 105.476317ms after stability 2024-06-01T18:43:11.342Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:11.448Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:11.342428672 +0000 UTC m=+106.840618124 (1 nodes), 1.463µs from start of poll, 169.69383ms from cluster start. Timeout at 2024-06-01 18:43:11.44864134 +0000 UTC m=+106.946830802, 106.212678ms after stability 2024-06-01T18:43:11.449Z [ERROR] server-8c40286f-6748-6f35-3947-6dbc01a61cb2: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2024-06-01T18:43:11.449Z [WARN] server-8c40286f-6748-6f35-3947-6dbc01a61cb2: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2024-06-01T18:43:11.449Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: aborting pipeline replication: peer="{Voter server-8c40286f-6748-6f35-3947-6dbc01a61cb2 8c40286f-6748-6f35-3947-6dbc01a61cb2}" 2024-06-01T18:43:11.457Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: pipelining replication: peer="{Voter server-8c40286f-6748-6f35-3947-6dbc01a61cb2 8c40286f-6748-6f35-3947-6dbc01a61cb2}" 2024-06-01T18:43:11.457Z [INFO] server-8c40286f-6748-6f35-3947-6dbc01a61cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c40286f-6748-6f35-3947-6dbc01a61cb2 Address:8c40286f-6748-6f35-3947-6dbc01a61cb2} {Suffrage:Voter ID:server-548f25fd-b1f7-6a5a-0019-15a4119cb73d Address:548f25fd-b1f7-6a5a-0019-15a4119cb73d}]" 2024-06-01T18:43:11.458Z [INFO] server-548f25fd-b1f7-6a5a-0019-15a4119cb73d: aborting pipeline replication: peer="{Voter server-8c40286f-6748-6f35-3947-6dbc01a61cb2 8c40286f-6748-6f35-3947-6dbc01a61cb2}" --- PASS: TestRaft_CacheLogWithStoreError (0.29s) === RUN TestRaft_ReloadConfig 2024-06-01T18:43:11.459Z [INFO] server-ea39d722-3f72-e0a7-15c8-9ca7f598769e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea39d722-3f72-e0a7-15c8-9ca7f598769e Address:ea39d722-3f72-e0a7-15c8-9ca7f598769e}]" --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2024-06-01T18:43:11.460Z [INFO] server-690e2842-8b7f-f928-421a-2b131a7d045c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-690e2842-8b7f-f928-421a-2b131a7d045c Address:690e2842-8b7f-f928-421a-2b131a7d045c}]" --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2024-06-01T18:43:11.460Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2024-06-01T18:43:11.461Z [INFO] server-7ff73e1f-5a62-ac78-c837-270791892c02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ff73e1f-5a62-ac78-c837-270791892c02 Address:7ff73e1f-5a62-ac78-c837-270791892c02} {Suffrage:Voter ID:server-8d249e01-b003-df1e-9eec-cf8513437574 Address:8d249e01-b003-df1e-9eec-cf8513437574} {Suffrage:Voter ID:server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 Address:a0c3e9bd-e351-a609-00c5-89a9c93eea09}]" 2024-06-01T18:43:11.461Z [INFO] server-7ff73e1f-5a62-ac78-c837-270791892c02: entering follower state: follower="Node at 7ff73e1f-5a62-ac78-c837-270791892c02 [Follower]" leader-address= leader-id= 2024-06-01T18:43:11.461Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ff73e1f-5a62-ac78-c837-270791892c02 Address:7ff73e1f-5a62-ac78-c837-270791892c02} {Suffrage:Voter ID:server-8d249e01-b003-df1e-9eec-cf8513437574 Address:8d249e01-b003-df1e-9eec-cf8513437574} {Suffrage:Voter ID:server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 Address:a0c3e9bd-e351-a609-00c5-89a9c93eea09}]" 2024-06-01T18:43:11.461Z [INFO] server-a0c3e9bd-e351-a609-00c5-89a9c93eea09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ff73e1f-5a62-ac78-c837-270791892c02 Address:7ff73e1f-5a62-ac78-c837-270791892c02} {Suffrage:Voter ID:server-8d249e01-b003-df1e-9eec-cf8513437574 Address:8d249e01-b003-df1e-9eec-cf8513437574} {Suffrage:Voter ID:server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 Address:a0c3e9bd-e351-a609-00c5-89a9c93eea09}]" 2024-06-01T18:43:11.461Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: entering follower state: follower="Node at 8d249e01-b003-df1e-9eec-cf8513437574 [Follower]" leader-address= leader-id= 2024-06-01T18:43:11.461Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:11.461Z [INFO] server-a0c3e9bd-e351-a609-00c5-89a9c93eea09: entering follower state: follower="Node at a0c3e9bd-e351-a609-00c5-89a9c93eea09 [Follower]" leader-address= leader-id= 2024-06-01T18:43:11.522Z [WARN] server-8d249e01-b003-df1e-9eec-cf8513437574: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:11.522Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: entering candidate state: node="Node at 8d249e01-b003-df1e-9eec-cf8513437574 [Candidate]" term=2 2024-06-01T18:43:11.522Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: election won: term=2 tally=2 2024-06-01T18:43:11.522Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: entering leader state: leader="Node at 8d249e01-b003-df1e-9eec-cf8513437574 [Leader]" 2024-06-01T18:43:11.522Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: added peer, starting replication: peer=server-7ff73e1f-5a62-ac78-c837-270791892c02 2024-06-01T18:43:11.522Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: added peer, starting replication: peer=server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 2024-06-01T18:43:11.522Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: pipelining replication: peer="{Voter server-7ff73e1f-5a62-ac78-c837-270791892c02 7ff73e1f-5a62-ac78-c837-270791892c02}" 2024-06-01T18:43:11.522Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: pipelining replication: peer="{Voter server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 a0c3e9bd-e351-a609-00c5-89a9c93eea09}" 2024-06-01T18:43:11.628Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:11.522769251 +0000 UTC m=+107.020958703 (1 nodes), 61.048683ms from start of poll, 61.414354ms from cluster start. Timeout at 2024-06-01 18:43:11.628522133 +0000 UTC m=+107.126711585, 105.752882ms after stability 2024-06-01T18:43:11.634Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:11.739Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:11.634377623 +0000 UTC m=+107.132567075 (2 nodes), 852ns from start of poll, 173.022726ms from cluster start. Timeout at 2024-06-01 18:43:11.739909824 +0000 UTC m=+107.238099276, 105.532201ms after stability 2024-06-01T18:43:11.739Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: updating configuration: command=RemoveServer server-id=server-7ff73e1f-5a62-ac78-c837-270791892c02 server-addr= servers="[{Suffrage:Voter ID:server-8d249e01-b003-df1e-9eec-cf8513437574 Address:8d249e01-b003-df1e-9eec-cf8513437574} {Suffrage:Voter ID:server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 Address:a0c3e9bd-e351-a609-00c5-89a9c93eea09}]" 2024-06-01T18:43:11.740Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: removed peer, stopping replication: peer=server-7ff73e1f-5a62-ac78-c837-270791892c02 last-index=3 2024-06-01T18:43:11.740Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: aborting pipeline replication: peer="{Voter server-7ff73e1f-5a62-ac78-c837-270791892c02 7ff73e1f-5a62-ac78-c837-270791892c02}" 2024-06-01T18:43:11.850Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to pipeline appendEntries: peer="{Voter server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 a0c3e9bd-e351-a609-00c5-89a9c93eea09}" error="append pipeline closed" 2024-06-01T18:43:11.850Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: aborting pipeline replication: peer="{Voter server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 a0c3e9bd-e351-a609-00c5-89a9c93eea09}" 2024-06-01T18:43:11.851Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to heartbeat to: peer=a0c3e9bd-e351-a609-00c5-89a9c93eea09 backoff time=10ms error="failed to connect to peer: a0c3e9bd-e351-a609-00c5-89a9c93eea09" 2024-06-01T18:43:11.851Z [WARN] server-7ff73e1f-5a62-ac78-c837-270791892c02: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2024-06-01T18:43:11.857Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to appendEntries to: peer="{Voter server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 a0c3e9bd-e351-a609-00c5-89a9c93eea09}" error="failed to connect to peer: a0c3e9bd-e351-a609-00c5-89a9c93eea09" 2024-06-01T18:43:11.870Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to heartbeat to: peer=a0c3e9bd-e351-a609-00c5-89a9c93eea09 backoff time=10ms error="failed to connect to peer: a0c3e9bd-e351-a609-00c5-89a9c93eea09" 2024-06-01T18:43:11.875Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to appendEntries to: peer="{Voter server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 a0c3e9bd-e351-a609-00c5-89a9c93eea09}" error="failed to connect to peer: a0c3e9bd-e351-a609-00c5-89a9c93eea09" 2024-06-01T18:43:11.888Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to heartbeat to: peer=a0c3e9bd-e351-a609-00c5-89a9c93eea09 backoff time=10ms error="failed to connect to peer: a0c3e9bd-e351-a609-00c5-89a9c93eea09" 2024-06-01T18:43:11.894Z [WARN] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to contact: server-id=server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 time=50.394251ms 2024-06-01T18:43:11.894Z [WARN] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to contact quorum of nodes, stepping down 2024-06-01T18:43:11.894Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: entering follower state: follower="Node at 8d249e01-b003-df1e-9eec-cf8513437574 [Follower]" leader-address= leader-id= 2024-06-01T18:43:11.895Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to appendEntries to: peer="{Voter server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 a0c3e9bd-e351-a609-00c5-89a9c93eea09}" error="failed to connect to peer: a0c3e9bd-e351-a609-00c5-89a9c93eea09" 2024-06-01T18:43:11.948Z [WARN] server-8d249e01-b003-df1e-9eec-cf8513437574: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:11.948Z [INFO] server-8d249e01-b003-df1e-9eec-cf8513437574: entering candidate state: node="Node at 8d249e01-b003-df1e-9eec-cf8513437574 [Candidate]" term=3 2024-06-01T18:43:11.948Z [ERROR] server-8d249e01-b003-df1e-9eec-cf8513437574: failed to make requestVote RPC: target="{Voter server-a0c3e9bd-e351-a609-00c5-89a9c93eea09 a0c3e9bd-e351-a609-00c5-89a9c93eea09}" error="failed to connect to peer: a0c3e9bd-e351-a609-00c5-89a9c93eea09" term=3 2024-06-01T18:43:11.954Z [WARN] server-a0c3e9bd-e351-a609-00c5-89a9c93eea09: heartbeat timeout reached, starting election: last-leader-addr=8d249e01-b003-df1e-9eec-cf8513437574 last-leader-id=server-8d249e01-b003-df1e-9eec-cf8513437574 2024-06-01T18:43:11.954Z [INFO] server-a0c3e9bd-e351-a609-00c5-89a9c93eea09: entering candidate state: node="Node at a0c3e9bd-e351-a609-00c5-89a9c93eea09 [Candidate]" term=3 2024-06-01T18:43:11.954Z [ERROR] server-a0c3e9bd-e351-a609-00c5-89a9c93eea09: failed to make requestVote RPC: target="{Voter server-8d249e01-b003-df1e-9eec-cf8513437574 8d249e01-b003-df1e-9eec-cf8513437574}" error="failed to connect to peer: 8d249e01-b003-df1e-9eec-cf8513437574" term=3 2024-06-01T18:43:11.955Z [WARN] server-a0c3e9bd-e351-a609-00c5-89a9c93eea09: rejecting vote request since node is not in configuration: from=7ff73e1f-5a62-ac78-c837-270791892c02 --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.58s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2024-06-01T18:43:12.038Z [INFO] server-58c3822f-0e87-a939-b8bb-690119573f5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58c3822f-0e87-a939-b8bb-690119573f5d Address:58c3822f-0e87-a939-b8bb-690119573f5d} {Suffrage:Voter ID:server-297f658d-0890-7f3c-6104-5491a5b5ea3b Address:297f658d-0890-7f3c-6104-5491a5b5ea3b} {Suffrage:Voter ID:server-51c6c5ea-3556-62ba-d57b-63ed7d414003 Address:51c6c5ea-3556-62ba-d57b-63ed7d414003}]" 2024-06-01T18:43:12.039Z [INFO] server-297f658d-0890-7f3c-6104-5491a5b5ea3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58c3822f-0e87-a939-b8bb-690119573f5d Address:58c3822f-0e87-a939-b8bb-690119573f5d} {Suffrage:Voter ID:server-297f658d-0890-7f3c-6104-5491a5b5ea3b Address:297f658d-0890-7f3c-6104-5491a5b5ea3b} {Suffrage:Voter ID:server-51c6c5ea-3556-62ba-d57b-63ed7d414003 Address:51c6c5ea-3556-62ba-d57b-63ed7d414003}]" 2024-06-01T18:43:12.039Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58c3822f-0e87-a939-b8bb-690119573f5d Address:58c3822f-0e87-a939-b8bb-690119573f5d} {Suffrage:Voter ID:server-297f658d-0890-7f3c-6104-5491a5b5ea3b Address:297f658d-0890-7f3c-6104-5491a5b5ea3b} {Suffrage:Voter ID:server-51c6c5ea-3556-62ba-d57b-63ed7d414003 Address:51c6c5ea-3556-62ba-d57b-63ed7d414003}]" 2024-06-01T18:43:12.039Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:12.039Z [INFO] server-58c3822f-0e87-a939-b8bb-690119573f5d: entering follower state: follower="Node at 58c3822f-0e87-a939-b8bb-690119573f5d [Follower]" leader-address= leader-id= 2024-06-01T18:43:12.039Z [INFO] server-297f658d-0890-7f3c-6104-5491a5b5ea3b: entering follower state: follower="Node at 297f658d-0890-7f3c-6104-5491a5b5ea3b [Follower]" leader-address= leader-id= 2024-06-01T18:43:12.039Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: entering follower state: follower="Node at 51c6c5ea-3556-62ba-d57b-63ed7d414003 [Follower]" leader-address= leader-id= 2024-06-01T18:43:12.090Z [WARN] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:12.090Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: entering candidate state: node="Node at 51c6c5ea-3556-62ba-d57b-63ed7d414003 [Candidate]" term=2 2024-06-01T18:43:12.090Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: election won: term=2 tally=2 2024-06-01T18:43:12.090Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: entering leader state: leader="Node at 51c6c5ea-3556-62ba-d57b-63ed7d414003 [Leader]" 2024-06-01T18:43:12.090Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: added peer, starting replication: peer=server-58c3822f-0e87-a939-b8bb-690119573f5d 2024-06-01T18:43:12.090Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: added peer, starting replication: peer=server-297f658d-0890-7f3c-6104-5491a5b5ea3b 2024-06-01T18:43:12.090Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: pipelining replication: peer="{Voter server-297f658d-0890-7f3c-6104-5491a5b5ea3b 297f658d-0890-7f3c-6104-5491a5b5ea3b}" 2024-06-01T18:43:12.090Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: pipelining replication: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:12.090Z [WARN] server-58c3822f-0e87-a939-b8bb-690119573f5d: rejecting vote request since our last term is greater: candidate=51c6c5ea-3556-62ba-d57b-63ed7d414003 last-term=2 last-candidate-term=1 2024-06-01T18:43:12.198Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:12.090501811 +0000 UTC m=+107.588691263 (1 nodes), 51.386266ms from start of poll, 51.701904ms from cluster start. Timeout at 2024-06-01 18:43:12.198506414 +0000 UTC m=+107.696695866, 108.004603ms after stability 2024-06-01T18:43:12.203Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:12.309Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:12.203745047 +0000 UTC m=+107.701934499 (2 nodes), 2.545µs from start of poll, 164.94514ms from cluster start. Timeout at 2024-06-01 18:43:12.309265044 +0000 UTC m=+107.807454497, 105.519998ms after stability 2024-06-01T18:43:12.311Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to pipeline appendEntries: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" error="append pipeline closed" 2024-06-01T18:43:12.311Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: aborting pipeline replication: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:12.311Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to heartbeat to: peer=58c3822f-0e87-a939-b8bb-690119573f5d backoff time=10ms error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.320Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to appendEntries to: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.331Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to heartbeat to: peer=58c3822f-0e87-a939-b8bb-690119573f5d backoff time=10ms error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.336Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to appendEntries to: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.348Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to heartbeat to: peer=58c3822f-0e87-a939-b8bb-690119573f5d backoff time=10ms error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.355Z [WARN] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to contact: server-id=server-58c3822f-0e87-a939-b8bb-690119573f5d time=50.607327ms 2024-06-01T18:43:12.356Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to appendEntries to: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.368Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to heartbeat to: peer=58c3822f-0e87-a939-b8bb-690119573f5d backoff time=20ms error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.378Z [WARN] server-58c3822f-0e87-a939-b8bb-690119573f5d: heartbeat timeout reached, starting election: last-leader-addr=51c6c5ea-3556-62ba-d57b-63ed7d414003 last-leader-id=server-51c6c5ea-3556-62ba-d57b-63ed7d414003 2024-06-01T18:43:12.378Z [INFO] server-58c3822f-0e87-a939-b8bb-690119573f5d: entering candidate state: node="Node at 58c3822f-0e87-a939-b8bb-690119573f5d [Candidate]" term=3 2024-06-01T18:43:12.378Z [ERROR] server-58c3822f-0e87-a939-b8bb-690119573f5d: failed to make requestVote RPC: target="{Voter server-51c6c5ea-3556-62ba-d57b-63ed7d414003 51c6c5ea-3556-62ba-d57b-63ed7d414003}" error="failed to connect to peer: 51c6c5ea-3556-62ba-d57b-63ed7d414003" term=3 2024-06-01T18:43:12.378Z [ERROR] server-58c3822f-0e87-a939-b8bb-690119573f5d: failed to make requestVote RPC: target="{Voter server-297f658d-0890-7f3c-6104-5491a5b5ea3b 297f658d-0890-7f3c-6104-5491a5b5ea3b}" error="failed to connect to peer: 297f658d-0890-7f3c-6104-5491a5b5ea3b" term=3 2024-06-01T18:43:12.385Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to appendEntries to: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.396Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to heartbeat to: peer=58c3822f-0e87-a939-b8bb-690119573f5d backoff time=25ms error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.399Z [WARN] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to contact: server-id=server-58c3822f-0e87-a939-b8bb-690119573f5d time=94.846289ms 2024-06-01T18:43:12.414Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: updating configuration: command=RemoveServer server-id=server-58c3822f-0e87-a939-b8bb-690119573f5d server-addr= servers="[{Suffrage:Voter ID:server-297f658d-0890-7f3c-6104-5491a5b5ea3b Address:297f658d-0890-7f3c-6104-5491a5b5ea3b} {Suffrage:Voter ID:server-51c6c5ea-3556-62ba-d57b-63ed7d414003 Address:51c6c5ea-3556-62ba-d57b-63ed7d414003}]" 2024-06-01T18:43:12.414Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: removed peer, stopping replication: peer=server-58c3822f-0e87-a939-b8bb-690119573f5d last-index=3 2024-06-01T18:43:12.414Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: updating configuration: command=AddNonvoter server-id=server-58c3822f-0e87-a939-b8bb-690119573f5d server-addr=58c3822f-0e87-a939-b8bb-690119573f5d servers="[{Suffrage:Voter ID:server-297f658d-0890-7f3c-6104-5491a5b5ea3b Address:297f658d-0890-7f3c-6104-5491a5b5ea3b} {Suffrage:Voter ID:server-51c6c5ea-3556-62ba-d57b-63ed7d414003 Address:51c6c5ea-3556-62ba-d57b-63ed7d414003} {Suffrage:Nonvoter ID:server-58c3822f-0e87-a939-b8bb-690119573f5d Address:58c3822f-0e87-a939-b8bb-690119573f5d}]" 2024-06-01T18:43:12.414Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: added peer, starting replication: peer=server-58c3822f-0e87-a939-b8bb-690119573f5d 2024-06-01T18:43:12.414Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to appendEntries to: peer="{Nonvoter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" error="failed to connect to peer: 58c3822f-0e87-a939-b8bb-690119573f5d" 2024-06-01T18:43:12.432Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: peer has newer term, stopping replication: peer="{Nonvoter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:12.432Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: entering follower state: follower="Node at 51c6c5ea-3556-62ba-d57b-63ed7d414003 [Follower]" leader-address= leader-id= 2024-06-01T18:43:12.432Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: aborting pipeline replication: peer="{Voter server-297f658d-0890-7f3c-6104-5491a5b5ea3b 297f658d-0890-7f3c-6104-5491a5b5ea3b}" 2024-06-01T18:43:12.432Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: peer has newer term, stopping replication: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:12.478Z [WARN] server-58c3822f-0e87-a939-b8bb-690119573f5d: Election timeout reached, restarting election 2024-06-01T18:43:12.478Z [INFO] server-58c3822f-0e87-a939-b8bb-690119573f5d: entering candidate state: node="Node at 58c3822f-0e87-a939-b8bb-690119573f5d [Candidate]" term=23 2024-06-01T18:43:12.478Z [WARN] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: rejecting vote request since node is not a voter: from=58c3822f-0e87-a939-b8bb-690119573f5d 2024-06-01T18:43:12.478Z [WARN] server-297f658d-0890-7f3c-6104-5491a5b5ea3b: rejecting vote request since we have a leader: from=58c3822f-0e87-a939-b8bb-690119573f5d leader=51c6c5ea-3556-62ba-d57b-63ed7d414003 leader-id=server-51c6c5ea-3556-62ba-d57b-63ed7d414003 2024-06-01T18:43:12.521Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:12.531Z [WARN] server-297f658d-0890-7f3c-6104-5491a5b5ea3b: heartbeat timeout reached, starting election: last-leader-addr=51c6c5ea-3556-62ba-d57b-63ed7d414003 last-leader-id=server-51c6c5ea-3556-62ba-d57b-63ed7d414003 2024-06-01T18:43:12.531Z [INFO] server-297f658d-0890-7f3c-6104-5491a5b5ea3b: entering candidate state: node="Node at 297f658d-0890-7f3c-6104-5491a5b5ea3b [Candidate]" term=3 2024-06-01T18:43:12.531Z [WARN] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:12.531Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: entering candidate state: node="Node at 51c6c5ea-3556-62ba-d57b-63ed7d414003 [Candidate]" term=24 2024-06-01T18:43:12.531Z [INFO] server-297f658d-0890-7f3c-6104-5491a5b5ea3b: entering follower state: follower="Node at 297f658d-0890-7f3c-6104-5491a5b5ea3b [Follower]" leader-address= leader-id= 2024-06-01T18:43:12.531Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: election won: term=24 tally=2 2024-06-01T18:43:12.531Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: entering leader state: leader="Node at 51c6c5ea-3556-62ba-d57b-63ed7d414003 [Leader]" 2024-06-01T18:43:12.531Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: added peer, starting replication: peer=server-297f658d-0890-7f3c-6104-5491a5b5ea3b 2024-06-01T18:43:12.531Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: added peer, starting replication: peer=server-58c3822f-0e87-a939-b8bb-690119573f5d 2024-06-01T18:43:12.531Z [WARN] server-58c3822f-0e87-a939-b8bb-690119573f5d: failed to get previous log: previous-index=4 last-index=2 error="log not found" 2024-06-01T18:43:12.531Z [INFO] server-58c3822f-0e87-a939-b8bb-690119573f5d: entering follower state: follower="Node at 58c3822f-0e87-a939-b8bb-690119573f5d [Follower]" leader-address=51c6c5ea-3556-62ba-d57b-63ed7d414003 leader-id=server-51c6c5ea-3556-62ba-d57b-63ed7d414003 2024-06-01T18:43:12.532Z [WARN] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: appendEntries rejected, sending older logs: peer="{Nonvoter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" next=3 2024-06-01T18:43:12.532Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: pipelining replication: peer="{Nonvoter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:12.532Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: pipelining replication: peer="{Voter server-297f658d-0890-7f3c-6104-5491a5b5ea3b 297f658d-0890-7f3c-6104-5491a5b5ea3b}" 2024-06-01T18:43:12.637Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:12.532106781 +0000 UTC m=+108.030296233 (1 nodes), 10.857442ms from start of poll, 493.306874ms from cluster start. Timeout at 2024-06-01 18:43:12.637845416 +0000 UTC m=+108.136034868, 105.738635ms after stability 2024-06-01T18:43:12.659Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: updating configuration: command=RemoveServer server-id=server-58c3822f-0e87-a939-b8bb-690119573f5d server-addr= servers="[{Suffrage:Voter ID:server-297f658d-0890-7f3c-6104-5491a5b5ea3b Address:297f658d-0890-7f3c-6104-5491a5b5ea3b} {Suffrage:Voter ID:server-51c6c5ea-3556-62ba-d57b-63ed7d414003 Address:51c6c5ea-3556-62ba-d57b-63ed7d414003}]" 2024-06-01T18:43:12.659Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: removed peer, stopping replication: peer=server-58c3822f-0e87-a939-b8bb-690119573f5d last-index=106 2024-06-01T18:43:12.659Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: aborting pipeline replication: peer="{Nonvoter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:12.660Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: updating configuration: command=AddVoter server-id=server-58c3822f-0e87-a939-b8bb-690119573f5d server-addr=58c3822f-0e87-a939-b8bb-690119573f5d servers="[{Suffrage:Voter ID:server-297f658d-0890-7f3c-6104-5491a5b5ea3b Address:297f658d-0890-7f3c-6104-5491a5b5ea3b} {Suffrage:Voter ID:server-51c6c5ea-3556-62ba-d57b-63ed7d414003 Address:51c6c5ea-3556-62ba-d57b-63ed7d414003} {Suffrage:Voter ID:server-58c3822f-0e87-a939-b8bb-690119573f5d Address:58c3822f-0e87-a939-b8bb-690119573f5d}]" 2024-06-01T18:43:12.660Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: added peer, starting replication: peer=server-58c3822f-0e87-a939-b8bb-690119573f5d 2024-06-01T18:43:12.660Z [WARN] server-58c3822f-0e87-a939-b8bb-690119573f5d: failed to get previous log: previous-index=107 last-index=106 error="log not found" 2024-06-01T18:43:12.660Z [WARN] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: appendEntries rejected, sending older logs: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" next=107 2024-06-01T18:43:12.660Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: pipelining replication: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:13.738Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:13.843Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:13.738633192 +0000 UTC m=+109.236822645 (1 nodes), 2.425µs from start of poll, 1.699833286s from cluster start. Timeout at 2024-06-01 18:43:13.84366895 +0000 UTC m=+109.341858412, 105.035767ms after stability 2024-06-01T18:43:13.843Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: aborting pipeline replication: peer="{Voter server-58c3822f-0e87-a939-b8bb-690119573f5d 58c3822f-0e87-a939-b8bb-690119573f5d}" 2024-06-01T18:43:13.843Z [INFO] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: aborting pipeline replication: peer="{Voter server-297f658d-0890-7f3c-6104-5491a5b5ea3b 297f658d-0890-7f3c-6104-5491a5b5ea3b}" 2024-06-01T18:43:14.344Z [ERROR] server-51c6c5ea-3556-62ba-d57b-63ed7d414003: failed to heartbeat to: peer=58c3822f-0e87-a939-b8bb-690119573f5d backoff time=10ms error="command timed out" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (2.32s) === RUN TestRaft_FollowerRemovalNoElection 2024-06-01T18:43:14.354Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccd5795e-39f6-bf27-4176-6831daa471e3 Address:ccd5795e-39f6-bf27-4176-6831daa471e3} {Suffrage:Voter ID:server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 Address:8a46b337-01ad-b81c-b7c2-d6e876141ad0} {Suffrage:Voter ID:server-34b3bcc7-20c3-d414-60fc-6262e69f7085 Address:34b3bcc7-20c3-d414-60fc-6262e69f7085}]" 2024-06-01T18:43:14.354Z [INFO] server-8a46b337-01ad-b81c-b7c2-d6e876141ad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccd5795e-39f6-bf27-4176-6831daa471e3 Address:ccd5795e-39f6-bf27-4176-6831daa471e3} {Suffrage:Voter ID:server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 Address:8a46b337-01ad-b81c-b7c2-d6e876141ad0} {Suffrage:Voter ID:server-34b3bcc7-20c3-d414-60fc-6262e69f7085 Address:34b3bcc7-20c3-d414-60fc-6262e69f7085}]" 2024-06-01T18:43:14.354Z [INFO] server-34b3bcc7-20c3-d414-60fc-6262e69f7085: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccd5795e-39f6-bf27-4176-6831daa471e3 Address:ccd5795e-39f6-bf27-4176-6831daa471e3} {Suffrage:Voter ID:server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 Address:8a46b337-01ad-b81c-b7c2-d6e876141ad0} {Suffrage:Voter ID:server-34b3bcc7-20c3-d414-60fc-6262e69f7085 Address:34b3bcc7-20c3-d414-60fc-6262e69f7085}]" 2024-06-01T18:43:14.354Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:14.354Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: entering follower state: follower="Node at ccd5795e-39f6-bf27-4176-6831daa471e3 [Follower]" leader-address= leader-id= 2024-06-01T18:43:14.354Z [INFO] server-8a46b337-01ad-b81c-b7c2-d6e876141ad0: entering follower state: follower="Node at 8a46b337-01ad-b81c-b7c2-d6e876141ad0 [Follower]" leader-address= leader-id= 2024-06-01T18:43:14.354Z [INFO] server-34b3bcc7-20c3-d414-60fc-6262e69f7085: entering follower state: follower="Node at 34b3bcc7-20c3-d414-60fc-6262e69f7085 [Follower]" leader-address= leader-id= 2024-06-01T18:43:14.506Z [WARN] server-ccd5795e-39f6-bf27-4176-6831daa471e3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:14.506Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: entering candidate state: node="Node at ccd5795e-39f6-bf27-4176-6831daa471e3 [Candidate]" term=2 2024-06-01T18:43:14.506Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: election won: term=2 tally=2 2024-06-01T18:43:14.506Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: entering leader state: leader="Node at ccd5795e-39f6-bf27-4176-6831daa471e3 [Leader]" 2024-06-01T18:43:14.506Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: added peer, starting replication: peer=server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 2024-06-01T18:43:14.506Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: added peer, starting replication: peer=server-34b3bcc7-20c3-d414-60fc-6262e69f7085 2024-06-01T18:43:14.506Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: pipelining replication: peer="{Voter server-34b3bcc7-20c3-d414-60fc-6262e69f7085 34b3bcc7-20c3-d414-60fc-6262e69f7085}" 2024-06-01T18:43:14.506Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: pipelining replication: peer="{Voter server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 8a46b337-01ad-b81c-b7c2-d6e876141ad0}" 2024-06-01T18:43:14.712Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:14.506338727 +0000 UTC m=+110.004528190 (1 nodes), 151.699403ms from start of poll, 151.965848ms from cluster start. Timeout at 2024-06-01 18:43:14.711950129 +0000 UTC m=+110.210139591, 205.611401ms after stability 2024-06-01T18:43:14.712Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:14.917Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:14.712080541 +0000 UTC m=+110.210270004 (1 nodes), 2.605µs from start of poll, 357.707662ms from cluster start. Timeout at 2024-06-01 18:43:14.91714803 +0000 UTC m=+110.415337492, 205.067488ms after stability 2024-06-01T18:43:14.923Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:15.128Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:14.923217309 +0000 UTC m=+110.421406761 (2 nodes), 2.685µs from start of poll, 568.844419ms from cluster start. Timeout at 2024-06-01 18:43:15.128411503 +0000 UTC m=+110.626600956, 205.194195ms after stability 2024-06-01T18:43:15.128Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:15.334Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:15.128572853 +0000 UTC m=+110.626762305 (1 nodes), 3.066µs from start of poll, 774.199963ms from cluster start. Timeout at 2024-06-01 18:43:15.33454164 +0000 UTC m=+110.832731092, 205.968787ms after stability raft_test.go:2961: [INFO] restarting Node at 8a46b337-01ad-b81c-b7c2-d6e876141ad0 [Follower] 2024-06-01T18:43:15.334Z [INFO] server-8a46b337-01ad-b81c-b7c2-d6e876141ad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccd5795e-39f6-bf27-4176-6831daa471e3 Address:ccd5795e-39f6-bf27-4176-6831daa471e3} {Suffrage:Voter ID:server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 Address:8a46b337-01ad-b81c-b7c2-d6e876141ad0} {Suffrage:Voter ID:server-34b3bcc7-20c3-d414-60fc-6262e69f7085 Address:34b3bcc7-20c3-d414-60fc-6262e69f7085}]" 2024-06-01T18:43:15.334Z [INFO] server-8a46b337-01ad-b81c-b7c2-d6e876141ad0: entering follower state: follower="Node at 8a46b337-01ad-b81c-b7c2-d6e876141ad0 [Follower]" leader-address= leader-id= 2024-06-01T18:43:15.586Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: aborting pipeline replication: peer="{Voter server-34b3bcc7-20c3-d414-60fc-6262e69f7085 34b3bcc7-20c3-d414-60fc-6262e69f7085}" 2024-06-01T18:43:15.973Z [ERROR] server-ccd5795e-39f6-bf27-4176-6831daa471e3: failed to pipeline appendEntries: peer="{Voter server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 8a46b337-01ad-b81c-b7c2-d6e876141ad0}" error="command enqueue timeout" 2024-06-01T18:43:15.973Z [INFO] server-ccd5795e-39f6-bf27-4176-6831daa471e3: aborting pipeline replication: peer="{Voter server-8a46b337-01ad-b81c-b7c2-d6e876141ad0 8a46b337-01ad-b81c-b7c2-d6e876141ad0}" 2024-06-01T18:43:16.104Z [ERROR] server-ccd5795e-39f6-bf27-4176-6831daa471e3: failed to heartbeat to: peer=8a46b337-01ad-b81c-b7c2-d6e876141ad0 backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.77s) === RUN TestRaft_VoteWithNoIDNoAddr 2024-06-01T18:43:16.126Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9975ebf-0902-c201-aacb-cbd7d5344acc Address:f9975ebf-0902-c201-aacb-cbd7d5344acc} {Suffrage:Voter ID:server-b46c2132-3230-0ab5-cce6-013b0a403c79 Address:b46c2132-3230-0ab5-cce6-013b0a403c79} {Suffrage:Voter ID:server-445eda31-57be-390d-d60b-b0e37ad82ff4 Address:445eda31-57be-390d-d60b-b0e37ad82ff4}]" 2024-06-01T18:43:16.126Z [INFO] server-b46c2132-3230-0ab5-cce6-013b0a403c79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9975ebf-0902-c201-aacb-cbd7d5344acc Address:f9975ebf-0902-c201-aacb-cbd7d5344acc} {Suffrage:Voter ID:server-b46c2132-3230-0ab5-cce6-013b0a403c79 Address:b46c2132-3230-0ab5-cce6-013b0a403c79} {Suffrage:Voter ID:server-445eda31-57be-390d-d60b-b0e37ad82ff4 Address:445eda31-57be-390d-d60b-b0e37ad82ff4}]" 2024-06-01T18:43:16.126Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: entering follower state: follower="Node at f9975ebf-0902-c201-aacb-cbd7d5344acc [Follower]" leader-address= leader-id= 2024-06-01T18:43:16.126Z [INFO] server-445eda31-57be-390d-d60b-b0e37ad82ff4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9975ebf-0902-c201-aacb-cbd7d5344acc Address:f9975ebf-0902-c201-aacb-cbd7d5344acc} {Suffrage:Voter ID:server-b46c2132-3230-0ab5-cce6-013b0a403c79 Address:b46c2132-3230-0ab5-cce6-013b0a403c79} {Suffrage:Voter ID:server-445eda31-57be-390d-d60b-b0e37ad82ff4 Address:445eda31-57be-390d-d60b-b0e37ad82ff4}]" 2024-06-01T18:43:16.126Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:16.126Z [INFO] server-b46c2132-3230-0ab5-cce6-013b0a403c79: entering follower state: follower="Node at b46c2132-3230-0ab5-cce6-013b0a403c79 [Follower]" leader-address= leader-id= 2024-06-01T18:43:16.126Z [INFO] server-445eda31-57be-390d-d60b-b0e37ad82ff4: entering follower state: follower="Node at 445eda31-57be-390d-d60b-b0e37ad82ff4 [Follower]" leader-address= leader-id= 2024-06-01T18:43:16.199Z [WARN] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2024-06-01T18:43:16.199Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: entering candidate state: node="Node at f9975ebf-0902-c201-aacb-cbd7d5344acc [Candidate]" term=2 2024-06-01T18:43:16.199Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: election won: term=2 tally=2 2024-06-01T18:43:16.199Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: entering leader state: leader="Node at f9975ebf-0902-c201-aacb-cbd7d5344acc [Leader]" 2024-06-01T18:43:16.199Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: added peer, starting replication: peer=server-b46c2132-3230-0ab5-cce6-013b0a403c79 2024-06-01T18:43:16.199Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: added peer, starting replication: peer=server-445eda31-57be-390d-d60b-b0e37ad82ff4 2024-06-01T18:43:16.199Z [WARN] server-b46c2132-3230-0ab5-cce6-013b0a403c79: rejecting vote request since our last term is greater: candidate=f9975ebf-0902-c201-aacb-cbd7d5344acc last-term=2 last-candidate-term=1 2024-06-01T18:43:16.199Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: pipelining replication: peer="{Voter server-b46c2132-3230-0ab5-cce6-013b0a403c79 b46c2132-3230-0ab5-cce6-013b0a403c79}" 2024-06-01T18:43:16.199Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: pipelining replication: peer="{Voter server-445eda31-57be-390d-d60b-b0e37ad82ff4 445eda31-57be-390d-d60b-b0e37ad82ff4}" 2024-06-01T18:43:16.305Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:16.199620888 +0000 UTC m=+111.697810340 (1 nodes), 72.677049ms from start of poll, 73.031168ms from cluster start. Timeout at 2024-06-01 18:43:16.305688286 +0000 UTC m=+111.803877738, 106.067398ms after stability 2024-06-01T18:43:16.305Z [INFO] cluster: starting stability test: raft-state=Leader 2024-06-01T18:43:16.411Z [INFO] cluster: stable state for Leader reached at 2024-06-01 18:43:16.305797438 +0000 UTC m=+111.803986891 (1 nodes), 2.295µs from start of poll, 179.207719ms from cluster start. Timeout at 2024-06-01 18:43:16.411394801 +0000 UTC m=+111.909584253, 105.597362ms after stability 2024-06-01T18:43:16.416Z [INFO] cluster: starting stability test: raft-state=Follower 2024-06-01T18:43:16.521Z [INFO] cluster: stable state for Follower reached at 2024-06-01 18:43:16.416729311 +0000 UTC m=+111.914918774 (2 nodes), 2.726µs from start of poll, 290.139602ms from cluster start. Timeout at 2024-06-01 18:43:16.521934724 +0000 UTC m=+112.020124176, 105.205402ms after stability 2024-06-01T18:43:16.527Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to heartbeat to: peer=445eda31-57be-390d-d60b-b0e37ad82ff4 backoff time=10ms error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.528Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to pipeline appendEntries: peer="{Voter server-445eda31-57be-390d-d60b-b0e37ad82ff4 445eda31-57be-390d-d60b-b0e37ad82ff4}" error="append pipeline closed" 2024-06-01T18:43:16.528Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: aborting pipeline replication: peer="{Voter server-445eda31-57be-390d-d60b-b0e37ad82ff4 445eda31-57be-390d-d60b-b0e37ad82ff4}" 2024-06-01T18:43:16.531Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to pipeline appendEntries: peer="{Voter server-b46c2132-3230-0ab5-cce6-013b0a403c79 b46c2132-3230-0ab5-cce6-013b0a403c79}" error="append pipeline closed" 2024-06-01T18:43:16.531Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: aborting pipeline replication: peer="{Voter server-b46c2132-3230-0ab5-cce6-013b0a403c79 b46c2132-3230-0ab5-cce6-013b0a403c79}" 2024-06-01T18:43:16.531Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to heartbeat to: peer=b46c2132-3230-0ab5-cce6-013b0a403c79 backoff time=10ms error="failed to connect to peer: b46c2132-3230-0ab5-cce6-013b0a403c79" 2024-06-01T18:43:16.534Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to appendEntries to: peer="{Voter server-445eda31-57be-390d-d60b-b0e37ad82ff4 445eda31-57be-390d-d60b-b0e37ad82ff4}" error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.538Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to appendEntries to: peer="{Voter server-b46c2132-3230-0ab5-cce6-013b0a403c79 b46c2132-3230-0ab5-cce6-013b0a403c79}" error="failed to connect to peer: b46c2132-3230-0ab5-cce6-013b0a403c79" 2024-06-01T18:43:16.547Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to heartbeat to: peer=445eda31-57be-390d-d60b-b0e37ad82ff4 backoff time=10ms error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.550Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to heartbeat to: peer=b46c2132-3230-0ab5-cce6-013b0a403c79 backoff time=10ms error="failed to connect to peer: b46c2132-3230-0ab5-cce6-013b0a403c79" 2024-06-01T18:43:16.553Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to appendEntries to: peer="{Voter server-445eda31-57be-390d-d60b-b0e37ad82ff4 445eda31-57be-390d-d60b-b0e37ad82ff4}" error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.556Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to appendEntries to: peer="{Voter server-b46c2132-3230-0ab5-cce6-013b0a403c79 b46c2132-3230-0ab5-cce6-013b0a403c79}" error="failed to connect to peer: b46c2132-3230-0ab5-cce6-013b0a403c79" 2024-06-01T18:43:16.565Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to heartbeat to: peer=445eda31-57be-390d-d60b-b0e37ad82ff4 backoff time=10ms error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.568Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to heartbeat to: peer=b46c2132-3230-0ab5-cce6-013b0a403c79 backoff time=10ms error="failed to connect to peer: b46c2132-3230-0ab5-cce6-013b0a403c79" 2024-06-01T18:43:16.569Z [WARN] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to contact: server-id=server-445eda31-57be-390d-d60b-b0e37ad82ff4 time=50.089953ms 2024-06-01T18:43:16.573Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to appendEntries to: peer="{Voter server-445eda31-57be-390d-d60b-b0e37ad82ff4 445eda31-57be-390d-d60b-b0e37ad82ff4}" error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.577Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to appendEntries to: peer="{Voter server-b46c2132-3230-0ab5-cce6-013b0a403c79 b46c2132-3230-0ab5-cce6-013b0a403c79}" error="failed to connect to peer: b46c2132-3230-0ab5-cce6-013b0a403c79" 2024-06-01T18:43:16.580Z [WARN] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to contact: server-id=server-b46c2132-3230-0ab5-cce6-013b0a403c79 time=58.499659ms 2024-06-01T18:43:16.580Z [WARN] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to contact: server-id=server-445eda31-57be-390d-d60b-b0e37ad82ff4 time=60.79967ms 2024-06-01T18:43:16.580Z [WARN] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to contact quorum of nodes, stepping down 2024-06-01T18:43:16.580Z [INFO] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: entering follower state: follower="Node at f9975ebf-0902-c201-aacb-cbd7d5344acc [Follower]" leader-address= leader-id= 2024-06-01T18:43:16.585Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to heartbeat to: peer=445eda31-57be-390d-d60b-b0e37ad82ff4 backoff time=20ms error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.600Z [ERROR] server-f9975ebf-0902-c201-aacb-cbd7d5344acc: failed to appendEntries to: peer="{Voter server-445eda31-57be-390d-d60b-b0e37ad82ff4 445eda31-57be-390d-d60b-b0e37ad82ff4}" error="failed to connect to peer: 445eda31-57be-390d-d60b-b0e37ad82ff4" 2024-06-01T18:43:16.627Z [WARN] server-b46c2132-3230-0ab5-cce6-013b0a403c79: heartbeat timeout reached, starting election: last-leader-addr=f9975ebf-0902-c201-aacb-cbd7d5344acc last-leader-id=server-f9975ebf-0902-c201-aacb-cbd7d5344acc 2024-06-01T18:43:16.627Z [INFO] server-b46c2132-3230-0ab5-cce6-013b0a403c79: entering candidate state: node="Node at b46c2132-3230-0ab5-cce6-013b0a403c79 [Candidate]" term=3 2024-06-01T18:43:16.627Z [INFO] server-b46c2132-3230-0ab5-cce6-013b0a403c79: entering follower state: follower="Node at b46c2132-3230-0ab5-cce6-013b0a403c79 [Follower]" leader-address= leader-id= 2024-06-01T18:43:16.627Z [ERROR] server-b46c2132-3230-0ab5-cce6-013b0a403c79: failed to make requestVote RPC: target="{Voter server-f9975ebf-0902-c201-aacb-cbd7d5344acc f9975ebf-0902-c201-aacb-cbd7d5344acc}" error="failed to connect to peer: f9975ebf-0902-c201-aacb-cbd7d5344acc" term=3 --- PASS: TestRaft_VoteWithNoIDNoAddr (0.55s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2024-06-01T18:43:16.679Z [INFO] 127.0.0.1:40541 :: starting node: addr=127.0.0.1:40541 2024-06-01T18:43:16.679Z [INFO] 127.0.0.1:40541 :: initial configuration: index=0 servers=[] 2024-06-01T18:43:16.679Z [INFO] 127.0.0.1:40541 :: entering follower state: follower="Node at 127.0.0.1:40541 [Follower]" leader-address= leader-id= 2024-06-01T18:43:16.775Z [WARN] 127.0.0.1:40541 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2024-06-01T18:43:16.830Z [INFO] 127.0.0.1:43233 :: starting node: addr=127.0.0.1:43233 2024-06-01T18:43:16.830Z [INFO] 127.0.0.1:43233 :: initial configuration: index=0 servers=[] 2024-06-01T18:43:16.830Z [INFO] 127.0.0.1:43233 :: entering follower state: follower="Node at 127.0.0.1:43233 [Follower]" leader-address= leader-id= 2024-06-01T18:43:16.905Z [WARN] 127.0.0.1:43233 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2024-06-01T18:43:17.060Z [INFO] 127.0.0.1:36139 :: starting node: addr=127.0.0.1:36139 2024-06-01T18:43:17.061Z [INFO] 127.0.0.1:36139 :: initial configuration: index=0 servers=[] 2024-06-01T18:43:17.061Z [INFO] 127.0.0.1:36139 :: entering follower state: follower="Node at 127.0.0.1:36139 [Follower]" leader-address= leader-id= 2024-06-01T18:43:17.132Z [WARN] 127.0.0.1:36139 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.53s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.23s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2024-06-01T18:43:17.236Z [INFO] 127.0.0.1:33645 :: starting node: addr=127.0.0.1:33645 2024-06-01T18:43:17.236Z [INFO] 127.0.0.1:33645 :: initial configuration: index=0 servers=[] 2024-06-01T18:43:17.236Z [INFO] 127.0.0.1:33645 :: entering follower state: follower="Node at 127.0.0.1:33645 [Follower]" leader-address= leader-id= 2024-06-01T18:43:17.236Z [WARN] 127.0.0.1:33645 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.17s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS ok github.com/hashicorp/raft 112.895s make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/3061969 and its subdirectories I: Current time: Sat Jun 1 06:43:24 -12 2024 I: pbuilder-time-stamp: 1717267404 Sat Jun 1 18:43:26 UTC 2024 I: 1st build successful. Starting 2nd build on remote node infom02-amd64.debian.net. Sat Jun 1 18:43:26 UTC 2024 I: Preparing to do remote build '2' on infom02-amd64.debian.net. Sat Jun 1 18:49:30 UTC 2024 I: Deleting $TMPDIR on infom02-amd64.debian.net. Sat Jun 1 18:49:31 UTC 2024 I: golang-github-hashicorp-raft_1.5.0-1_amd64.changes: Format: 1.8 Date: Thu, 27 Jul 2023 15:11:32 +0000 Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.5.0-1 Distribution: unstable Urgency: medium Maintainer: Debian Go Packaging Team Changed-By: Daniel Swarbrick Description: golang-github-hashicorp-raft-dev - implementation of the Raft consensus protocol Changes: golang-github-hashicorp-raft (1.5.0-1) unstable; urgency=medium . * Team upload * New upstream release * Add flaky TestRaft_HasExistingState to skipped tests Checksums-Sha1: 661dc27cce54b2fa917be64df5d901ec5e69e04c 106468 golang-github-hashicorp-raft-dev_1.5.0-1_all.deb c4b8c3439a267c00920e3ea831b81f905aad1b55 8255 golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo Checksums-Sha256: a59f670467267462b33ea85847a52a174cc5c54327d3d38cf4352b0bddb2887e 106468 golang-github-hashicorp-raft-dev_1.5.0-1_all.deb 2f7d456edd0df61c06f470a898a3f95560dbb581ae5c5f6d2b7723caed298a15 8255 golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo Files: 6162adc0e7336a9201ced64e81c3ff6e 106468 golang optional golang-github-hashicorp-raft-dev_1.5.0-1_all.deb 12705b39ca67340277226f050910f5d9 8255 golang optional golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo Sat Jun 1 18:49:32 UTC 2024 I: diffoscope 268 will be used to compare the two builds: Running as unit: rb-diffoscope-amd64_37-14321.service # Profiling output for: /usr/bin/diffoscope --timeout 7200 --html /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/golang-github-hashicorp-raft_1.5.0-1.diffoscope.html --text /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/golang-github-hashicorp-raft_1.5.0-1.diffoscope.txt --json /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/golang-github-hashicorp-raft_1.5.0-1.diffoscope.json --profile=- /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/b1/golang-github-hashicorp-raft_1.5.0-1_amd64.changes /srv/reproducible-results/rbuild-debian/r-b-build.Bo8rBwgM/b2/golang-github-hashicorp-raft_1.5.0-1_amd64.changes ## command (total time: 0.000s) 0.000s 1 call cmp (internal) ## has_same_content_as (total time: 0.000s) 0.000s 1 call abc.DotChangesFile ## main (total time: 0.360s) 0.360s 2 calls outputs 0.000s 1 call cleanup ## recognizes (total time: 0.016s) 0.016s 12 calls diffoscope.comparators.binary.FilesystemFile ## specialize (total time: 0.000s) 0.000s 1 call specialize Finished with result: success Main processes terminated with: code=exited/status=0 Service runtime: 689ms CPU time consumed: 688ms Sat Jun 1 18:49:34 UTC 2024 I: diffoscope 268 found no differences in the changes files, and a .buildinfo file also exists. Sat Jun 1 18:49:34 UTC 2024 I: golang-github-hashicorp-raft from unstable built successfully and reproducibly on amd64. Sat Jun 1 18:49:35 UTC 2024 I: Submitting .buildinfo files to external archives: Sat Jun 1 18:49:35 UTC 2024 I: Submitting 12K b1/golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo.asc Sat Jun 1 18:49:36 UTC 2024 I: Submitting 12K b2/golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo.asc Sat Jun 1 18:49:37 UTC 2024 I: Done submitting .buildinfo files to http://buildinfo.debian.net/api/submit. Sat Jun 1 18:49:37 UTC 2024 I: Done submitting .buildinfo files. Sat Jun 1 18:49:37 UTC 2024 I: Removing signed golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo.asc files: removed './b1/golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo.asc' removed './b2/golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo.asc'